There's a myriad of reasons, from the original author not liking it and thus preventing further releases if he was given that kind of power, to people on the production team dying unexpectedly.
But really, mostly it's just sales.
Twilight Q released during the late 1980s which was a special period for OVAs. In the early 1980s VHS tapes became cheap enough to reach general consumers. Anime was sold to the video store for about $200 per episode and the unthinkable happened. Japanese fans actually bought them at those prices that were intended for a video store.
There was a dedicated segment of fans willing to spend $200 per episode and then there were lots of people creating OVAs and selling to these type of people. Mostly small "studios", too. Maybe operating out of someone's apartment. Life was good back then and people had money to spend on anime consumption and production. If it didn't work out after a few episodes they'd just stop and try a different project.
It was a period of experimentation. Anime OVAs were a new thing. Prior to the 1980s, they didn't exist at all. So people mostly had fun with them, going for ultra violence and nudity, which you wouldn't get to see on TV or in a movie theater, for that matter. The first pornographic anime released, too. (Some people will tell you that there were earlier pornographic anime, like the 1970 Cleopatra, but they are wrong.)
There are many unfinished OVAs from that time. But there are also many unfinished TV shows nowadays. It's the same principle. If it sells enough, you'll likely get to see more of it.