The reason for this is because Super Saiyan 3 form was to fully maximize the ki of the user.
The Dragonball Wikia states :
Where the third stage of Super Saiyan's purpose was 100% utilization
of physical stamina, the purpose of the Super Saiyan 3 transformation
is to increase the utilization of ki, and as a result, the
transformation rapidly consumes the energy of the user. This notably
leads to extended levels of fatigue, even long after powering down.
However, with the enormous power, there are some very serious setbacks
stemming from the rapid use of ki energy. When the then-deceased Goku
showed Babidi and Majin Buu the form, he cut his remaining temporal
revival energy in half, and in the anime when he shows his power to
the excited Trunks and Goten, his energy was completely dissipated,
forcing him to return to Other World.
As far as Super Saiyan 4 the wikia states :
Unlike the Super Saiyan 3's strain due to the utilization in ki, this
form does not require as much energy consumption as Super Saiyan 3's,
allowing the user to stay in the state at a much longer duration. It
also appears that the form makes the individual larger in height and
Essentially Super Saiyan 3 maximizes the ki output while Super Saiyan 4 is a more powerful transformation which does not drain the user as much as the 3rd form does.
Also Super Saiyan 4 is not actually a canon form because it was not developed by Akira Toriyama.