In the first season of Free!, the three main swimmers of Iwatobi Swim Club could only manage to add one new swimmer, Rei. (Gou, while a member, was not a swimmer.) In the second season, even though it was an active and ongoing goal, they never managed to add anyone else. While the issue was not actually central to the plot, it bothers me that these likeable guys could not find anyone else.
I can think of two possibilities. The first is production-based -- the folks doing the anime chose not to complicate the plot with any more characters. To counter that, though, the Samezuka Acadamy, with already at least a dozen on their team, added two more significant ones during the 2nd season. The other possibility is that there is an underlying reason within the story that the Iwatobi High School Swim Club is supposed to die. It will do so after this school year, due to the graduation of Haru and Makoto. With only three members (two swimmers), the team would be auto-disbanded. Why can they not get any new teammates?