Here's an updated answer to your question. Note that this will contain spoilers from the manga Shingeki no Kyojin: Before the Fall and I'm confident this is accurate as of Volume 14 of the mentioned manga, which was where I stopped reading. Also, the events in this manga occurred approximately 50-70 years before the breach in Shiganshina District in the main Shingeki no Kyojin manga.
Before the invention of the Vertical Maneuvering Equipment, the Survey Corps could only counter Titans using swords, rifles and bombs for close-combat and cannons for long-range. This was shown in Volume 2 of the manga. However, this was just a stopgap and the most it could do was delay the movements of or distract a Titan due to its fast regenerative abilities. The swords were initially made using a naturally-growing/wild plant called Iron Bamboo before it would be replaced with ultrahard steel after the final design of the Vertical Maneuvering Equipment was completed.
Angel Aaltonen, a craftsman, with the help of his assistant, Corina, and another craftsman, Xenophon Harkimo, were able to come up with an early model of the maneuvering equipment. This was mostly shown in Volumes 10-11 of the manga as Angel's flashbacks. This equipment could only move vertically (upwards and downwards). Also, unlike the current maneuvering equipment which has a control grip at the handle of each sword, the control grip for the early model was in one hand while the other holds the sword.
How did they come up with this idea?
It was Corina who suggested on thinking up on a device that allows them to fight 'on equal terms' with Titans. So, the invention of the first maneuvering device happened, to make up for the height difference of a human and a Titan. But, during this time, they were still unaware of a Titan's 'weak spot.'
How, then, were they able to 'learn to kill' a Titan?
After Angel, Corina and Xenophon returned from the Industrial City where they made the first model of the maneuvering device, a Titan was able to get inside the gates of Wall Maria, which ate Corina. Angel, wanting revenge, first joined an expedition to attempt to capture a Titan and find its weaknesses. This almost ended in failure but with the death of his close friend, Sorum, who sacrificed himself by blowing up a Titan's head when he was about to be eaten, they witnessed how the Titan did not regenerate and die (they saw lots of smoke being emitted from the Titan's body before disappearing). They initially presumed that the weak point might be the head or somewhere in the throat and neck. To test this, another expedition was led in secret by the Survey Corps and this time, Angel himself was able to kill a Titan, the first human to do so using the early model of the maneuver device, but the credit was given to the then-captain of the Survey Corps, Jorge 'The Hero', despite Jorge himself not wanting to.
In Volume 6 Chapter 19,
Jorge mentioned that they were not sure of the exact location of the 'weak spot' but this was what he said: '..based on the way Angel delivered the killing blow, we can assume that it is somewhere on the back of the neck. Angel was able to confirm that it was not actually the throat, as we initially surmised..' Therefore, it was Angel who discovered the Titan's weakness but after this expedition, he would disappear for roughly a decade before being convinced by Sharle to go back and help finish the device.
In Volume 12,
during their journey back to the Industrial City in Xenophon's workshop, it was Sharle who suggested that the control grip be placed on a sword's handle. After Sharle's suggestion, it took them one month to finalize the Vertical Maneuver Equipment's design and build it. Thus, mankind's first weapon that can actually kill Titans was finally finished.