We see a flashback to when Kagami lived in the US in episode 1 of season 2 (26Q).
Judging from all the palm trees, I'd hazard a guess that he was probably in California. California also seems like the most likely option on a demographic basis (i.e. where are Japanese temporary immigrants most likely to end up?).
Is it ever explicitly stated where he lived? And if it is California, is it ever stated whether he's in SoCal or NorCal (or somewhere else)?