All of these are closely related and many games would fall into more than one category. However, which word you choose places the emphasis on different aspects of the game. In any case, when it isn't important to be precise, these terms are often used interchangeably.
- Eroge (エロゲ) is a Japanese shortening of "erotic game". These can also be called H-games. Native Japanese speakers sometimes use the term to describe games without any sexual content (e.g. Clannad), but in English this isn't really a correct use of the term. Eroge can be any game with sex scenes (also called H-scenes). This includes some games that are not traditionally included as visual novels or dating sims. For example, Kamidori Alchemy Meister is an example of a game without a great deal of visual novel content but which still qualifies as an eroge.
- Visual Novel (ビジュアルノベル), often shortened to VN, is a general type of game with a lot of dialogue and minimal gameplay (usually the gameplay is reduced to just making choices at a few plot points to determine what route one enters). It may or may not involve any romance or sexual encounters. For example, Danganronpa could qualify as a visual novel, but probably not for any of the other categories here. A more canonical example would be Higurashi no Naku Koro ni, though purists will sometimes insist on calling it a "Sound Novel" as this is the official description. More on this in the related terms section. Calling something a visual novel emphasizes the "novel" aspect and suggests that there is at least some semblance of a story. It would not usually be used to describe eroge which are solely sex scenes.
- Visual novel can also be used in a more technical way to describe games where the text is overlayed over the background as opposed to being presented in dialogue boxes. This distinction is more common among Japanese speakers than English speakers. In English usually people will abbreviate this as NVL, and games where the dialogue is in a box at the bottom of the screen are called ADV. Fate/Stay Night is an example of this style.
- Dating Sims are a different category of game from Visual Novels. These games do feature gameplay, but the object of the gameplay is to get into a romantic relationship with a character from the game. The most familiar example is probably the Tokimeki Memorial series, though there are many other highly successful examples such as Love Plus. Unlike visual novels, the gameplay here can be fairly complex.
- A galge (ギャルゲーム) or bishoujo game (美少女ゲーム, lit. "beautiful girl game") can be used to describe any game where a major part of the game is getting into and being in a relationship with one or more of the female characters (usually not at the same time). These are marketed at heterosexual males. It includes a lot of games that would be classified as dating simulators and a lot that would be classified as visual novels. Most games that would fit into this category would also fit into one of the other categories, though there are examples like Gal*Gun which probably can only fit as galge.
There are a lot of other closely related terms that could be used to describe these kinds of games. I'll describe these briefly here:
A Sound Novel is a type of visual novel. The term is a trademark by Chunsoft and emphasizes the sound aspects as opposed to visual aspects. Sound novels are actually older than visual novels but because of the trademark the term "visual novel" is the one that's stuck in the long term. For older games the distinction between the two is somewhat notable, but for most modern games there's basically no difference between the meaning of the two terms.
A Kinetic Novel is a type of visual novel without any gameplay at all. That includes things like making decisions for the protagonist. Because of the lack of any player input, the story is entirely preset and the player just reads through it without any input. These are quite close to ordinary novels with added graphics, sound, and typically more focus on dialogue. Planetarian would be an example of such a game. Kinetic novels are usually shorter than other visual novels, though there are exceptions to this like Higurashi.
An Otome Game (乙女ゲーム) is a game marketed to females. It is roughly synonymous with the term Bishounen game (美少年ゲーム), which is of course directly analagous to bishoujo game except now the characters are males rather than females. A well-known example is Hakuoki.
A nakige (泣きゲー, literally "crying game") is a type of visual novel where the story is designed to have an emotional impact on the player. Most games by studio Key are nakige (e.g. Clannad). A related but distinct term is utsuge (鬱ゲー, literally "depressing game"). Unlike nakige, which usually have happy endings (though they may be only after significant hardships), utsuge are usually not as positive. The distinction is difficult to explain, but after playing a couple of games in each genre you'll definitely know. Some notable examples of utsuge would be Saya's Song or recent Key work Rewrite.
A nukige (抜きゲー, literally "masturbation game") is a type of eroge which is almost solely focused on sexual content. As the name suggests, they're intended as pornographic material. It would be unusual to describe such games as visual novels. These are the large majority of all eroge produced because they're easy to produce and still tend to sell well.
There are other terms but these are the ones that everyone should know when it comes to categorizing these sorts of games.