What would have happened to Gon, Killua, and Kurapika if they had lost the coin game against Gotoh? Would Gotoh really have killed them, or was he bluffing all along?
First of all, this is speculation, since you asked about 'what if'.
I believe to answer your question, we first need to understand Gotoh's personality. From what the story depicted, we can fairly well assume that Gotoh cared deeply for Killua. With that in mind, we can say that regardless of the outcome of the game, Killua would not have been killed.
As for Gon, Kurapika and Leorio, I believe he wanted to test them, probably by seeing how they would react to the different situations he created during the game. The reason for that would be because the Zoldyck are assassins, which meant that they for sure are hated by a lot of people (e.g. victims of the family's assassinations) and any strangers posed a threat, Gon & Co were no exception. The fact that they passed the Testing Gate and Canary was proof that they were no ordinary people and carried the potential of harming the Zoldyck.
During the game itself, he tried to see how far Gon & Co would go for Killua, as far as doing something that would not benefit them in any way; saying that he would kill Canary, a member of Zoldyck's household, if they lose (in the event Gon wanted to harm the Zoldyck and lost, resulting in Canary getting killed, it would be in Gon & Co's advantage to have less people to fight against, and thus would likely not take the game as seriously, but this is straying a bit too far).
At that point, if Gon really lost, it can be hard to say what would have happened, but I would tend to believe that following Gon's reaction, Gotoh determined then that they meant no harm to Killua. Should Gon have lost, Gotoh could possibly have instead refused them meeting with Killua assuming that Gon was not strong enough to protect Killua if needed.
Conversely, I believe that if the way Gon reacted during the game was different, hinting that Gon couldn't be trusted, only then, would Gotoh attempt to kill the three of them.
Gotoh would have killed them. He's a butler charged with ensuring a clan of assassin's aren't annoyed by petty visitors and don't have to kill anyone unless they're paid to do so.
... and the series would have been much shorter
Gotoh never had the intention to kill them. He had spares in his hand and was just toying with their emotions. Gotoh would never harm killua's friend over a game of heads or tails. Killua would go godspeed in a instant.