Answer found on reddit. I take no credit in writing this. I just found out that both the question and the answer in this link answers this question.
Throughout DBZ there is a running gag of Vegeta saying "I am the Prince of all Saiyans" despite us only seeing a handful in series and movies (Goku, Bardock, Borgos, Broly, Fasha, King Vegeta, Nappa, Paragus, Raditz, Tarble, Tora, Turles, Vegeta, Bulla, Gohan, Goten, Pan, Trunks). Along with the notion that the Saiyan race is pretty much extinct except for the main characters. But this is wrong and not plausible given what the Saiyans' role was under Frieza.
So the Saiyan-Tuffle war ends in the Age of 730. At this point in their history the Saiyans are just brutes with no technology until the Arcosians arrive and trade technology in exchange for them conquering planets. Shortly after seeing how good Saiyans are at fighting/killing they are annexed under Frieza and his Planet Trade Organization in the Age of 731 and become part of his military. And let’s assume it takes about a year for the planet to be fully developed as seen in the flashbacks and in the movie Bardock- The Father of Goku. So the planet is all up and running in the Age of 732.
So Frieza has the Saiyans doing his dirty work conquering planets across the universe for himself or for sale. His need/demand/want is so great that the Saiyans send their babies to planets with weak inhabitants to conquer. This continues until Frieza destroys the planet in the Age of 737. So for 5 years the Saiyans are sending their young to the ends of the universe for Frieza. Now the only question is how many are being sent daily. In the movie Bardock- The Father of Goku you can see a long hallway where the space pods are where the babies are sent out. From the scene you can see ~14 pods from the two sides and we can safely assume the hallway has more. And in the movie Broly- The Legendary Super Saiyan you get a better look at the nursery and you see ~8 beds with babies and again we can assume there are more.
So the Saiyans had no problems sending their babies to these planets because any babies that were decently strong would stay on the planet to be raised as elites. The weak are sent away with no exception e.g. Tarble was sent away for being too weak despite being from Royal blood. In episode 124 "Z Warriors Prepare" King Vegeta says to Vegeta "The weakest of our people are sent to planets far away, where they face no threat from our enemies" while standing in front of a window pointing out the spacepods (carrying babies) leaving the planet. In the scene you can see ~60 spacepods leaving. And let's assume this happens at least once a week and that Planet Vegeta's year is similar to Earth since the whole series refers to time the same way the Earth does.
So considering all this you have:
60 Saiyans sent out weekly every year over 5 years
60x52x5 = 15600 Saiyans in the Universe
And what happens after Frieza destroys Planet Vegeta? He lies about what happened to cover his tracks telling people that a comet destroyed the planet. He doesn't bother hunting down and killing the remaining Saiyans because he killed the strongest of them on the planet with the exception of Vegeta, Nappa, and Raditz. The weakest ones are of no threat to him and scattered across the universe (except Goku). And everyone pretty much accepts this, even Raditz is under the impression that is how things happened stating in the second episode "3 years ago, our planet was destroyed by meteorites". And of the Saiyans that were sent some were female, since all Saiyans love to fight and women did help conquer planets as well (Fasha a female Saiyan was on Bardock's team). The Saiyan race is alive and well, just separated by a distance.
I think there are definitely more Saiyans around but I imagined after Frieza destroyed the planet, teams were sent to find remnants of the race. Like when they defeated Bardock's group but looking for lost Saiyan children instead. Some were found, some were not (like Goku).
There are definitely more out there but most are probably too distant to come across. It does leave things open ended for more Saiyan encounters.
I would also think that there are a lot of hybrids out there. I have a feeling saiyans weren't taking a moral high ground about rape.