It seems at first that the technique is simply a magic trick with no limit or explanation. But I think it's much deeper than that.
What is ki?
Ki is the raw essence of energy in a body, and when a body has exactly no ki left, the body dies and the soul, essence, mind, whatever you want to call it, is sent to otherworld. All attacks require ki, heck even raising your arm requires a slight amount of ki. "Ki attacks" are attacks in which damage is not physical, but energy is shot from the body at an opponent or obstacle. Majin Buu's "magic transform beam" is no more mystical than any other attack, it's what he does with this ki that matters.
Ki control outside the body
Goku showed in many, many instances that he can control a ki attack long after it has left his body, and this is what Majin Buu does in this attack. Instead of the Kamehameha or other ki attack which is designed to deal a powerful impact, Majin Buu is clever enough to attack the opponent's ki directly, and ultimately drain their body of all ki, rendering them lifeless. He can at this point shape the remaining matter however he wants, but I don't believe it has any flavor, just an appearance.
How to defend against Buu's attack
When Buu uses this attack on common humans, he only needs to use a small amount of ki to wipe them out. For a whole crowd of people, I don't suspect that Buu even feels the energy usage. However, when he does this against a more powerful opponent, he must put more ki into the attack than the fighter has in total. This is still no real problem for Buu, as nobody can contest his ki... until Vegito comes to "PLAY, PLAY, PLAAAY!". Vegito's unimaginable ki was strong enough so that when Buu used his attack, it only drained a significant amount of ki from Vegito, leaving him very much alive if only a bit...compromised. He has enough energy leftover, plus Goku's ability to work around any disadvantage, to put up a strong fight even as a piece of candy, forcing Buu to reverse the effects.
How'd he do that?
Now, Buu clearly underestimated Vegito and didn't put even half of his energy into that attack. Maybe with full power he could have succeeded in leaving both Goku and Vegeta lifeless and edible. But my question is, did he have to use the same amount of energy to revert Vegito to human, or did he get the original energy back, like performing the technique backwards? I don't fully understand this, and if he actually gave energy to Vegito, I don't get why he would do that as a response to getting his ass whooped by candy. This is a hole in my logic, and I can't defend it. This is just my best theory. Hope it inspired creative analysis, and hail Son-Goku.