Currently, nearly 50% of Japanese adults are not sexually active. This has been reported in international news such as the Washington Post, the Telegraph, the Guardian, the Daily Mail, and Japan's own English-language news source, the Japan Times. Anime characters are a reflection of the real situation. Not all of the adults who are not currently sexually active are virgins (for example, some married couples stop having sex after birthing the number of children they want), but many are.
Japan is a dont-ask,-don't-tell society: what you want to do in the privacy of your own bedroom (watch hentai, use body pillow dolls, any sort of fetishes, etc.) is not considered any big deal, and mentioning that you felt up a stranger's breast in a hostess club can be done in front of your colleagues without anyone raising an eyebrow, so it is not a "prudish" society at all. However, the Japanese overwhelmingly want you to keep your sexuality and sexual activities to yourself: do whatever you like and that will generally be respected, but PDA (public display of affection) or coming out and sharing about it to others is not socially respected (i.e. don't bother other people with it). Many young Japanese believe, likely as an influence from the Western nations, that it makes sense to test-drive a potential spouse before marrying (sleep together and/or live together), but it is simply the case that not as many Japanese are even dating and getting married.
Another cultural factor that may contribute, which you can see in quite a lot of manga and anime, is that a major format of Japanese romance is to like someone who you are not friends with for a long time, and finally "confess" your feelings in a sudden love letter, on St. Valentine's Day, or on graduation day, in which the recipient must abruptly decide whether or not he/she has romantic interest in the other person -- who might not have been on the recipient's radar whatsoever. This can result in either being turned down on the spot ("I don't even know you"), a willingness to try to go on a couple dates ("Maybe I could get interested in you"), or the first-choice being the recipient is overjoyed ("I secretly pined away for you for years too!"). A lot of romantic feelings and sexual desire are never confessed, but some that are get rejected due to the format in which the potential couple does not get to know each other through friendship or casual dating before a major DTR (defining the relationship) event takes place, or reciprocal feelings are confessed right after the graduation ceremony and the respective parties part ways to go to different schools for high school or university, so the mutual interest doesn't lead anywhere.