Back in his youth Saiba was the best cook in Totsuki, but after he had a 50 vs 1 shokugeki match it's like he stopped having fun cooking, why?
Joichiro suffered from burn-out and had to leave the school to ease the pressure on his shoulders.
Joichiro is widely considered to be the best Totsuki student of his generation by his peers. As you mentioned, he managed to win an improbable 50 vs 1 cooking match -- armed with nothing but his own skill and wit.
But in the end, the peer pressure to perform and stay absolutely perfect in public cooking matches withered him. We have seen numerous times that Joichiro loves experimenting with weird dishes and recipes ; the Dried Sardines Garnished with Strawberry Jam would be a prime example of that personality trait. Since Joichiro eventually felt obligated to polish his art to perfection rather than being able to goof off and create weird dishes at his leisure, he lost the motivation to continue competing at Totsuki.
Joichiro avoided going to BLUE because he felt he had lost sight of his reason for cooking. Following Senzaemon Nakiri's advice, he went on a world tour to discover new cooking methods and vent off the accumulated pressure.