Conan mentioned that he probably worked for the Japanese Secret Police, but who are they exactly? Under what jurisdiction of Japanese law enforcement are they under? Are they like MI5 or something?
Furuya Rei is a member of 「黒の組織」(kuro no soshiki = the black organization). I don't know what chapter or episode you are referring to, so I cannot verify the original Japanese for "Japanese secret police" used there, but:
Yes, in real life there was a Japanese secret police along the same lines of Nazi Germany's Gestapo and the Soviet Union's KGB. It likewise carried out human rights abuses and crimes against humanity, such as experimentation on live prisoners of war and other individuals, and biological warfare (see also Japan’s Gestapo: Murder, Mayhem And Torture In Wartime Asia by Mark Felton).
It was called the Kempeitai. There was also a navy branch called the Tokubetsu Keisatsutai. In addition, the Tokubetsu Koutou Keisatsu conducted domestic criminal investigation of civilians suspected of being anti-war or dangerous and counter-espionage functions, and punished perceived offenders with torture.
However, all three were abolished by the Allied Powers in 1945, so it is not possible that Furuya Rei was a member, since he is only 29 years old in the Meitantei Conan series. Japan does not have a post-WWII secret service. It is likely that what is being translated as "Japanese secret police" is either 1) the Kouanchousachou a.k.a. the Public Security Intelligence Agency established under the Ministry of Justice in 1952, which "is tasked with internal security and espionage against threats to Japanese national security" and "has ties to several foreign intelligence as security agencies, including the CIA, FBI, MI6, and Mossad, with several PSIA agents having been invited to train with the CIA under its Intelligence Analysis Course," or 2) one of the types of special police in Japan.
Conan mention the name of the police just before we saw Akai for the first time after his death. Conan and Bourbon are investigating on a attempted murder. The target was Bourbon's client. The FBI is here, because the victim is also Jodie's friend. And Bourbon mocks the FBI, want them to leave Japan (in the French version he said "When did you leave my Japan ?" or something like that).
Scarlett arc, I think. And later, I think there is information that the Police is called, in French, PSB. When I googled that that give the Tokyo Metropolitan : Tokyo Metropolitan Police Public Security Bureau