As we know, we can avoid Sharingan-based genjutsu just by avoiding eye contact. Can't we avoid Infinite Tsukuyomi just by not looking at the moon?
The Infinite Tsukuyomi uses the moon to reflect the user's Rinne-Sharingan. Sharingan based genjutsu can be avoided by avoiding eye contact , yes. But not all. However the working of the Infinite Tsukuyomi differs from that of normal Sharingan genjutsu. The user's eye is reflected on the moon's surface. And then the moon shines or illuminates.
From Infinite Tsukuyomi wiki article :
This technique can also be reflected off the moon, upon the technique's activation, the light from the moon's eye illuminates the entire planet, turning it bright as day and making it impossible to escape. All living things are bathed in its penetrating light and immediately placed under the genjutsu.
Remember the part where Sasuke puts up the Susanoo to protect the team from the light produced by the Infinite Tsukuyomi moon? That was because the Infinite Tsukuyomi activates when the light falls upon the individual. The Susanoo blocked the incoming light rays and prevented the genjutsu. The moment the light penetrates the body, the person falls under the genjutsu. So, in order to avoid the Infinite Tsukuyomi, the person has to avoid the light given off by the moon. That is why the others couldn't escape it. Not looking at the moon doesn't help because it's the light that induces the genjutsu and not the moon's surface.