First of all, the Grail is not omniscient nor omnipotent, it's just a source of magic power. As such, in order for it to fulfill a wish, you have to also give it the method for fullfilling said wish. Since Kiritsugu's only method of "saving" was by sacrificing the lesser ones, that's what the grail would do, by culling humanity little by little. He realized that through the vision that the grail showed him, in the example with the boat, the Grail called it the correct decision to sacrifice 200 to save 300, but when the 300 separated into a 200 and 100 groups, it also called it correct to save 200 by sacrificing 100. By that point, he'd have sacrificed 300 to save 200, but the Grail still called it the correct decision.
After that, the Grail called him worthy of inheriting All the Evils in the World, Angra Mainyuu. I'm not sure if he realized that the Grail was polluted (something that happened in the Third Holy Grail War, when the Einzberns summoned Angra Mainyuu, expecting to win the war with an OP Servant, only for it to be too weak, lose early in the war, and pollute the Grail once it got sent its way), but at the very least, he learned that the Grail couldn't grant his wish the way he wanted to, and that it would destroy humanity if allowed to.
EDIT: And Iri and Illya appearing at the end was just part of the vision shown by the Grail, it was offering him to spend the rest of his life with them in a dream while it fulifilled his "wish". And that's where he makes his decision "Six billion people... or my wife and daughter". True to himself, he chooses to save humanity in exchange for his family.