( Sorry about my bad grammar and vocabulary, feel free to edit my answer )
From what i could figure, the most accepted answer would be on manga ver. The anime version seems lack of explanation while i haven't watched the anime "Completely"( i watchedthe first minute part where shouya lost his bicycle and met nagatsuka. I watched it on youtube) but i did read the manga completely( the reason maybe because they want to say "Hey, i made animation movie drama, story about deaf girl that got bullied and the person who bullied hers have suicidal-mind because he regret from what he did from the past to the deaf girl. I would give you 1/3 content of the story, if you want more, go buy some of our manga".
So it was started in Volume 1, Chapter 5, page 2 to 6 Here are some of the pages (read from right to left)
Page 2-3
Page 4-5
Page 6
He have such anxiety disorder with other people. He pissed of because his own negative thought. Look at the yellow marker, he hate everything, even the friend of friends that he hated. SO that's the reason, why the entire class had "X" mark on their faces.