why does Homura have to die within her soul gem in order not to betray Madoka
i can only think of 2 instances you are referring to
when Kyubey asks if Homura wants to spend eternity trapped inside her barrier as a witch. Homura says that Mami and Kyoko will kill her instead order to free the people who were drawn into her Labyrinth and thus the Isolation Field which prevents anything getting out
Homulily being marched off to a guillotine. this is explained in the 7th dot point on Observations
Homulilly appears to have half of her head cut off, which is a reference to the guillotine in her barrier, guillotines are usually symbols of punishment and shame, which may reflect Homura's guilt for not being able to protect Madoka.
this comes after she's found out that the Incubators have partly succeeded in capturing Madoka within the Isolation Field and plan on studying her
Even if Kyubey confirms the existence of God Madoka, it doesn't mean he has the power to manipulate it to his advantage. After all, God Madoka's power is greater. That's why Madoka is a God, isn't it?
but that was their end goal and to do that they needed to confirm the existence of the Law of Cycles and then observe how it works so they can interfere and control her
Kyubey says his kind wants to observe the Law of Cycles, which to them had only been a hypothesis. Homura insists on knowing why, claiming that if Kyubey considers curiosity irrational,
but Homura realizes that his goal is to control Madoka and attacks Kyubey with her witch powers. Kyubey evades the attacks and admits that by observing the Law of Cycles the incubators could interfere with and control it. This would allow them to turn magical girls into witches, reinstating the old system that provided the incubators with more energy.
Rebellion/Synopsis > Part D (3rd paragraph)
as quoted above the Incubator's goal is still the same, collecting energy and as explained at the end of the series/second movie, after Madoka's wish was granted and the new universe was created, the "current" system of energy collection isn't as good as the old one with the witches which Homura explained to Kyubey where it comments how that would have been a better system but they had no way of confirming the existence of the Law of Cycles.
Would the Incubators be able to control Madoka? well yes given time. remember this is a species who can rip out the souls of young girls, fasion the soul into a gem and give them great powers, powers enough for one girl to be able to travel though time and another recreate the universe