It is not true that no one ever crossed the Calm Belts.
First of all, from https://onepiece.fandom.com/wiki/Calm_Belt, we know that Marines used Seastone-lined ships to repel Sea Kings and cross the Calm Belts. Besides numbers, strong admirals, this is certainly one of their strong points, as they can travel fast through the world. Impel Down is in the Calm Belts, so travels there are pretty frequent.
Amazon Lily is in the middle of a Calm Belt, and Rayleigh traveled there by swimming! So powerful individuals can travel there if they want to. As for normal pirates, the Calm Belts is full of Sea Kings and there is no wind to travel by sails, so special boats are needed. This eliminates 99% of pirates.
Finally, there is no proof of this in the manga so far, but fishmen, at least strong enough ones like Arlong or Jinbei, can probably travel easily within the Calm Belts. Also, the Revolutionary Army is scattered through the world, the faction of each sea under the order of one Commander. The commanders meet at the beginning of Wano Country Arc and they seem to have casually traveled through the world. Granted, the revolutionaries are experts in infiltraton so they may have stealthily crossed lands but crossing Calm Belts seems within their reach.