According to here:
There is a sacred fountain on Mount Myōboku which produces a special oil which allows people to more easily feel the natural energy around them, but if the person is not perfectly still, becoming one with nature, he or she starts becoming a toad and eventually turns into stone. This oil can only exist in the atmosphere of the mountain, and will evaporate if taken outside. There are many statues of the frog ancestors and of people who tried to learn senjutsu, but failed and became toad statues. The statues are treated with much respect.
People live at Mount Myōboku, and some can use senjutsu, so it stands to reason that there would have been a number who failed.
To address the other part of your question, Minato (the Fourth Hokage) is also known to have been able to summon toads, which shows that it isn't just Naruto and Jiraiya who can summon them.