The writing system used is likely just rotated katakana, mixed with some similar looking shapes.
The second line in the image looks like the katakana テナ二 (transliterated tenani), and the fourth line is almost a perfect match for 二ヒサ (nihisa). The other lines also contain some katakana characters, mixed in with some other plausible looking characters that don't correspond to actual kana.
As for the language itself, the katakana characters don't correspond to any phrase or word I could think of that would fit in this context. Likely, the animators just drew some random katakana and changed up the shapes of some of them to make it look like another world's script. I haven't seen the anime, so I can't vouch for whether any of the other scenes have meaningful phrases, but it's likely that there isn't any deeper meaning behind them.