As the comments say, this is a somewhat subjective question. I'll give my own reason, which I think is a big part of the reason this is common, but perhaps not the only reason.
Japanese society is more regimented than one might believe just from watching anime. By middle school age, and even to some degree in primary school, children are expected to interact with their peers in a relatively professional way. This is especially true of relationships between students of opposite sexes. Such requirements aren't held for family or close friends, but for everyone else one is expected to be polite and not share their true feelings or say things too directly. This concept of dividing people into an in-group and an out-group (内外, uchi-soto) is somewhat hard for non-Japanese people to understand, but is central to interactions between people in Japanese culture. In fact, you can find plenty of examples of this and of the related concepts of honne and tatemae in anime if you look, but if you aren't looking for them specifically it's easy to miss.
It's difficult for a person in the out-group to enter the in-group. While not impossible, generally this takes time and effort. The groups aren't a simple dichotomy either; they change depending on the situation. It's especially difficult for close friendships between members of the opposite sex to form. One way for this to happen is for the two to enter into a relationship, but this is obviously not ideal if you want to make a romantic comedy where the characters aren't already in relationships. Another way is for one person to act closer to the other than they really are, as a way of testing the boundaries, but that person is being somewhat rude in doing so and it can backfire.
The most realistic way for a character to have a close friend is for them to be long-time friends. Specifically, if their friendship goes back all the way to childhood, when they didn't have to follow these strict rules, a friendship could develop without much issue. Having a childhood friend character is one way to get a character who is already part of the protagonist's in-group. From a writer's standpoint, this is an attractive proposition, since it gives someone that the protagonist can have relatively serious conversations with, but who can also be a potential romantic interest. In (reverse) harem shows, it's especially attractive to diversify the female (male) cast of romantic interests.
That said, I think we have been seeing something of a decline in recent years in this archetype, at least among harem shows. Probably writers have realized that it's overused and that a generic osananajimi character without any unique personality traits isn't a very interesting character. Opposite-sex childhood friends are also far less common in real life than in anime, making the prevalence of this somewhat unrealistic. 10 years ago, nearly every harem/romcom show had a character like this, but the proportion seems to have dropped to less than half of them today. Partly, this can be attributed to other archetypes increasing in popularity (notably, the little sister/imouto character, who can be even closer to the protagonist than a childhood friend). When they do show up in shows today, it's often as a dual archetype character, e.g. a tsundere osananajimi.
