( Sorry about my bad grammar and vocabulary, feel free to edit my answer )
/!\Spoiler Warning
The content from this point on contains spoilers. Read it at your own risk. Blockquote
From what i could figure, the most accepted answer for this kind of question is based on manga. The anime version seems lack of explanation .
The reason of this case is kind like this:"Hey, i made animation movie drama, story about deaf girl that got bullied and the person who bullied hers have suicidal-mind because he regret from what he did from the past to the deaf girl. Oh, and it has a manga version. I would give you 1/3 content of the story, if you want more, go buy some of our manga". And of course i have finished the manga.
I haven't watched the anime, well i do watch that but just some scenes, skipped some part.
( i watched the first minute part where shouya lost his bicycle and met nagatsuka. I watched it on youtube)
It was started in Volume 1, Chapter 5, page 2 to 6 Here are some of the pages (read from right to left)
Page 2-3
Page 4-5
Page 6
He have an
anxiety disorder with other people. He feel pissed off because his own negative thought to the other. Look at the yellow marker, he hate everything, even the friend of friends that he hated.
SO the question "Is there a concrete pattern behind why and when the X is put on a person" already have an answer. For question "why does Ueno get the X on and off?" i will answer that after a few hours later. Soon I will edit my answer when i am have much time to write