I know that if you open the 8th Gate, you will get enormously powerful but at the end you will die.
In that case, how did Guy know Jutsu that can be performed in that state, given that against Madara was the first time he ever opened it? You can't just invent new Jutsu on the fly.
Okay, it's hard to explain, and i might not even be right, but I'll try my best. Sadly, there's no tldr.
What's different? Guy didn't invent the jutsu in the sense that Kakashi invented the Chidori. Kakashi spent years perfecting it. It's something he initially needed to use hand signs to perform every time.
Why?
The jutsus (more specifically taijutsu techniques) Guy used — Evening Elephant (Sekizo) and Night Guy — are byproducts of attacking with all eight gates opened. It's not something you have to learn; it's something you are just able to do with that much power flooding from your body.
The jutsu are just names he used to describe the phenomenon of having incredibly OP punches and kicks that can also generate a dragon-like aura (for some reason). He didn't actually know what would happen, unless he witnessed his dad do the same thing and he told Rock Lee what it was like.
Evidence From Narutopedia (Sekizo, Night Guy):
The Evening Elephant ... can be performed by those users who have opened all Eight Gates.
The Night Guy ... can only be performed by opening all Eight Gates.
^ They imply that it's something that you can naturally do with the eight gates opened.
When Guy initiates the technique, he emits an enormous amount of chakra ... the blood vapour of the Eighth Gate manifests into a red, flickering dragon-like aura ... He then dashes forward at such an extreme speed that the space within the immediate vicinity of the technique is distorted, making it impossible for the target to defend against it. Guy then deals an immensely powerful kick which carries such force that it is capable of obliterating the target's body...
^ Essentially, Guy is gathering power, running really fast, and kicking really hard. That's not really something that you have to practice and invent, like the Rasen-shuriken.
Examples: Kinda like when Obito and Madara became the Ten-tails Jinchuriki. They could do all this fancy truth-seeking ball stuff. Not because they spent time learning it, but because they were just able to, with the help of Ten-tails and huge sage power. Or when Naruto was able to heal people and Sasuke was able to teleport as soon as they got Hagoromo's chakra.
Sigh... Nevertheless, there are too many caveats and exceptions in the Naruto universe. Everything they say isn't necessarily true (e.g. Gaara's absolute defence is not absolute, Hashirama's unique wood release can be used by several other people). I mean, your first sentence is refuted by the fact that Guy actually survived (thanks to Naruto). So it's quite hard to make sense of anything.
More info - If you're not satisfied, look at this guy's explanation, which is good in the sense that it uses common sense: How do you train in a jutsu that kills you if used?
To understand how Guy managed to master the Hachimon Tonkou (Eight Gates Release Formation), let us take a look to how jutsus are learned in Narutoverse.
There are 3 ways a jutsu can be learned.
Learn it from a teacher
This might be the easiest way to learn a jutsu. You have a teacher teach you how to perform a certain jutsu. Most ninja learn jutsu this way through the academy. Example of jutsus learned through this method: Bunshin no Jutsu, Kawarimi no Jutsu, Sasuke learning Chidori from Kakashi, Naruto learning Rasengan from Jiraiya.
Learn it from a scroll/book
This is a method that is harder than the first method. You learn it from a book. I no longer have the copy with me but my father used to have a book that teaches kungfu moves. In that book there are explanations and illustrations on how to perform certain moves. You learn the techniques by performing as the book says. Example of jutsus learned through this method: Naruto learning Kage Bunshin no Jutsu, Orochimaru learning various jutsus using scrolls he stole from the library.
Learn it yourself (a.k.a. invent it)
This method might be the hardest of the 3. You learn the jutsu by creating it yourself. You can invent it through seeing natural phenomena or other people performing a certain jutsu, copy it and change several aspects of it to make your own jutsu using the same principles. Example of jutsus learned through this method: Minato learning Rasengan, Kakashi learning Chidori/Raikiri, Naruto learning Rasen Shuriken, Tobirama learning Kage Bunshi no Jutsu.
How does Guy knows how to perform jutsus in the Hachimon Tonkou?
To answer this we need to identify how Guy learned the jutsu and his training method. Guy learned the jutsu from Might Duy, his father. Guy's training method consists of extreme physical training, such as walking around the village for 500 laps using his hands, thousands sit ups and image training. The image training was mentioned in the manga (and perhaps the anime as well) as one of methods he used to train himself with the reasoning being training his ability to predict enemy movements. Guy also trains by sparring with Kakashi to enhance his fighting senses.
Since Hachimon Tonkou is a taijutsu, let's talk about real life taijutsu training. Taijutsu in real world, such as Kungfu, Karate, Kendo are mastered through training the said moves. A complete newbie can perform Iai Slash, but they would perform it badly as they only copy the movement in general. What differentiates a master from those newbies is that they perform the same move over and over through training until they can use the move with super precision (perfect foot-steps, sword drawing timing, slashing direction, power, speed).
Might Guy might learned the jutsus used in Hachimon Tonkou by learning the moves itself in normal state or 7th gate release state, which is the closest he can get to without killing himself. He learned the jutsu by training the moves over and over in the 7th gate release state while also doing image training since he can't perform the jutsu for real. Combining that with the points I bold above and fighting sense honed through experience of real combat and sparring with one of the best shinobi ever (Hatake Kakashi), it is very possible for him to perform the jutsus splendidly despite that being the very first time he used it for real because he already mastered the basics to perform that.
It is the same as when you do a chemistry practical test at school despite knowing only the theory (including the steps). Can you do it? Yes, but it might be a bit sloppy since you never do it for real. If you have practiced the movements (pouring the chemicals, measuring it, using the pipette) using a substitute liquid (like water for example) you would be able to perform it greatly even if that was the first time you use the chemicals.
Though this is a bit dissapointing, we do not fully know the truth. We do know for a fact that Guy's Father Duy was in the same situation as he was, but was weaker (the eternal genin). He learned the Eight gates from an unknown source. Duy claims it took him 20 years to master the eight gates. The only piece of knoledge we have on the subject is simply summarized to:
By undergoing intense training, one can learn how to open these gates allowing the user to surpass their own physical limits at the cost of extreme damage to their own body.
In the same paragraph on the wiki, the Gates
limit the overall flow of chakra within a person's body. The basis for the idea of the chakra gates comes from the body's limits on the functions within it. This makes the body much weaker, but it keeps the body from expiring too soon.
So we know that the Gates are simply, Literally, Chakra limiters, and Chakra flow increases strength, speed, and other physical traits. In the anime, it is shown that to open the gates, you need only Overload them, which forces them open. Therefor, it does make sense that one who spends years if not decades training intensely, like Duy or Guy, would eventually accidentally overload the first one. it is also possible to do so if they are in battle in a critical moment. Either way, Knowing there are gates (sharinan and byakugan can see chakra flow, so they can probably also see the gates) and learning they can be opened through force is probably all you need to open at least the first 7. Guy has been shown to open several at once, and for the first 7 only ever had to clench his muscles to do so. The first 7 can probably just be opened by sheer force which can be learned and mastered by training.
The problem then is simply the 8th gate, which is shown to require a puncture to the gate itself, which is located in the heart. The issue is simply how they could figure that out. As we know, there has only been one master of the gates at a time, so there are 2 likely senarios, either 1, they watch their master use it (Guy knew his dad was going to use the 8th gate against the 7 Ninja Swordsman) and witnessed it, realizing that was the key to opening the gate. The second idea is simply that they cant force it open, either through lack of training (to not risk accidentally activating it) or its just locked that hard, and the puncture works to make it easier to open. If you know about openin the first 7, it is completely understandable that you would figure out how to make it easier to open, though I doubt you want to puncture yourself every time, except when you do not expect to live long without doing so.
The last thing to think about is the Moves he uses. Well like almost every move besides Rasen Shuriken were invented off screen. By who and what we do not know, But what is clear is that Guy can not use Ninjutsu, as well as Lee. Every Jutsu they used was some kind of Taijutsu, usually a simple punch or kick, sometimes enhanced with the power of the Gates.
Morning Peacock was simply punches so fast that the air friction alone set them ablaze.
Daytime Tiger merely used the strength to compress a massive amount of air into a very small point, then release it in a massive shockwave.
Evening Elephant was to create air cannons with your punches
Night Guy Was nothing more than a Chakra enhanced Super kick.
In the end, all these moves are Simple punches or kicks, which could be performed at lower or any level, with or without gates, but with reduced effectiveness. Like most forbidden Jutsu, They were also likely written down in scrolls, recorded as useful/effective moves, but with high risks attached. It only takes one time to test a move, and one survivor to record it. We also know Might Duy had reached all 8 gates, so he could also have had time to invent some of these moves. Beyond that, We are just looking at fairly simple moves, for a guy that is undeniably a Genius Taijutsu specialist, and only 2 of them are doable in the 8th gate, both of which are just a punch and Kick, simple moves in principle.
