Spoiler Warning
.TL;DR
So the summary of this answer is:
The "X" mark can be symbolized as "Not My Friend" or "No interest to interact with them". When Ishida feels pissed off because of an act or the attitude of someone/something, the "X" mark would be put on their faces. And when would the "X" mark be taken down? When Ishida thinks like "I like him/her, such a nice person" about someone or something.
Because Ueno is so hypocritical, that make Ishida so pissed and happy at the same time. The X mark on Ueno's face was first marked in volume 3, and it came off in Volume 7 (the last volume), when Ishida tried to change himself. In volume 7, all the 'X' marks that everyone had in Ishida's perspective have already come off
The content from this point on contains spoilers. Read it at your own risk.
From what I could figure, the most accepted answer for this kind of question is one where you can find in the manga version. Although I have not watched the anime in full, the anime version seems lacking in explanation. And of course I have finished the manga.
1. What is the significance of the X on people's faces from Ishida's perspective?
The X's first appeared in Volume 1, Chapter 5, page 2-6. Here are some of the pages (read from right to left)
Page 2-3
Page 4-5
Page 6
He has an
anxiety disorder being with other people. He feels pissed off because of his own negative thought about the others. Look at the yellow marker, he hates everything, even the friend of friends that he hated. The "X" mark can be symbolized as "Not My Friend" or "No interest to interact with them". So that's the reason why the entire class have "X" mark on their faces. For Nagatsuka why he got "X" mark from the first, there is something that Ishida hates about him. For the other people like Sahara that doesn't have "X" mark when they met on the train, because Ishida has no reason to hate Sahara for some reason (exclude his classmate nor neighborhood that he hate). He will put the "X" mark down when someone wants to acknowledge him more and he trusted it.
2. Why does Ueno get the X on and off?
It's explained in Volume 3, Chapter 18, Page 12-17 (read from right to left).
Page 12-13
Page 14-15
Page 16-17
Ishida said that he doesn't care about his ex-friend (aka all his elementary school friends) in pages 12-13, but when he met Ueno for the first time since junior high school, he did care. At first, Ueno's face didn't have the "X" mark but in chapter 21, pages 9-10, Ueno did something terrible to Shouko, and that made Ishida really pissed of with Ueno's attitude. So that's why Ueno got the 'X' mark on pages 15-16.
Page 9-10
Page 15-16
