The sign says ようじや (youjiya) but because it's written in hiragana the meaning is a bit ambiguous. The や suffix in the name of a store would always be 屋, which just means store, or it can also refer to the head of such a store (e.g. 魚屋 can mean either a fish store or the dealer selling you the fish). So this is a store for ようじ (youji), but the meaning of that is still ambiguous because ようじ written in hiragana is a homophone that could have several meanings.
The meaning that Aoi interpreted is 幼児, which means infants and young children. So Aoi thought it was a store to buy children, hence the second screenshot you've given. This is the most common somewhat sensical interpretation of ようじ when written in hiragana in this context. It could also mean 用事 (errands) in other contexts, but that doesn't fit here.
But ようじ can also mean toothpick written as 楊子 (or 揚枝 less commonly). Usually one uses 爪楊枝 (tsumayouji) to describe the toothpicks themselves but the 爪 (here tsuma, meaning claw or nail) can be omitted in principle. This is the meaning here, so the shop is literally a store for toothpicks (and likely some related merchandise).
I think that many Japanese people would not know how to interpret the sign ようじや, so Aoi's misunderstanding is understandable. However, I don't think very many people would think that it was a store for 幼児 (children) either, so Aoi's leap of reasoning there is somewhat comical.
As a cultural note, there actually is a toothpick specialty shop in Japan. It's called Saruya and located in Tokyo (link to a blog post about the store). As far as I know this is the only store of its kind in Japan, so the shop in Ro-kyu-bu is fictional (being in Kyoto). In any case, Japanese culture takes toothpicks more seriously than most other cultures, so the existence of such a store is not entirely impossible, but it's still quite implausible.