So far, all of the customers came from the same world. And all of the characters share some kind of connection. Some met each other in their world, some are from same country, some know about others from books or tales. Most of them also know same language, which allows them to read the menu.
Kuro is easy one, as she lives on the broken moon, which is visible from the ground shots. And she knows Aka, who obviously lives in same world as Aletta, who personally knows Ms. Minced Meat Cutlets, and so on.
One character who is unrelated to most is the Lizard man. But Lizard men and their tribes living in marshes were mentioned by other characters, from what I remember.
Another reason to believe the door doesn't open to different worlds, is that the door was created and linked by Yomi the Hero, using artifact from that world. And there would be no reason for her to link it to different worlds, she didn't knew about or cared about.