In the manga Oibore in the fallen district (slums) ends up revealing who he is when asked why he helped Kenshin by telling the people there its his daughters husband who was there when she died. But how did he know? Kenshin was a top secret assassin and his identity was kept secret to most of the public. How did he even know they got married (it happened in the mountains away from anyone) or Tomoe died while kenshin was there (the only people who would know, kenshin killed and maybe the higher up imperialists)?
It was stated he abandoned the family after his wife died so he wouldn't have had any contact with her or Enishi. Enishi doesn't even meet him until after his fight with Kenshin, even then it is implied he doesn't really know who he is.