Back in his youth Saiba was the best cook in Totsuki, but after he had a 50 vs 1 shokugeki match it's like he stopped having fun cooking, why?
Joichiro suffered from burn-out and had to leave the school to ease the pressure on his shoulders.
Joichiro is widely considered to be the best Totsuki student of his generation by his peers. As you mentioned, he managed to win an improbable 50 vs 1 cooking match -- armed with nothing but his own skill and wit.
But in the end, the peer pressure to perform and stay absolutely perfect in public cooking matches withered him. We have seen numerous times that Joichiro loves experimenting with weird dishes and recipes ; the Dried Sardines Garnished with Strawberry Jam would be a prime example of that personality trait. Since Joichiro eventually felt obligated to polish his art to perfection rather than being able to goof off and create weird dishes at his leisure, he lost the motivation to continue competing at Totsuki.
Joichiro avoided going to BLUE because he felt he had lost sight of his reason for cooking. Following Senzaemon Nakiri's advice, he went on a world tour to discover new cooking methods and vent off the accumulated pressure.
It's not the fact he wasn't enjoy cooking, but rather he despise the fact that nobody is brave enough to do challenge with him.
In the manga after the 50v1 shokugeki, he still having fun cooking challenge with his dorm mate.