The people say it's in the Grand Line, but I can't find anything on how they knew. In the opening with his last words, Gold Roger only says he left it in one place, not that he left it in the Grand Line. Did I miss something?
The one piece in itself is a guess. If you remember, Luffy even yelled at Ussop in Shabody for asking such a question, and for once Luffy was actually trying not to think about the possibility of the one piece being nonexistent. However, we know that:
There are 4 Red Poneglyphs, and each one points to a certain location. Once they are laid out, these four locations will have Raftel in the middle.
However, we do not have actual evidence that Raftel is in Grand Line. Now that you mention it, probably, just maybe, it is somewhere else. (The irony; people were looking for it for so long in Grand Line and if it's not there then really, what a disappointment.)
As far as we know, it could be anywhere, even on top of the Red Line, we don't know. Note, at times Rayleigh's facial expressions really did suggest wild thoughts in readers, like the one piece does not exist or that the Void Century is such a simple puzzle after all.
In Conclusion: The people assume it is in Grand Line because Gol D. conquered it, and short after that, he got executed.
The people guessed that it was there, but it seems Roger was implying it was in Raftel.
One translation I read was exactly:
"Want my ultimate treasures? It's possible... I will give them to those who can find them. I have gathered everything in the world and already hidden them in "that" place".
Another one read:
"in that one place"
And a third:
"in one place"
While literally these are identical, there is an insinuation with the word "that". It implies that the speaker has already indicated where he is talking about or you already know where he is talking about. As the former is not the case (he didn't say it), people understood it to be the latter. As everyone knows Gol D. Roger and his crew were the only ones ever to reach the end of the Grand Line and Raftel, the legend quickly became that the One Piece was there.
I cannot tell from the kanji, whether or not this implication was conveyed in the same way in the original Japanese. The reaction from his statement, however, is that he was implying Raftel and everyone understood this.
There is, however,the process of elimination as well, as people searched everywhere for it but no one was able to check that one rumored place just yet.
As far as we know the only people who has actual information about One Piece are the former Roger pirates. Everyone else, both readers and characters, are only guessing about One Piece.
Every new piece of information that turns up (Crocus explaining navigation, the poneglyphs, the Whitebeard-flashback, etc.) points to Raftel, so it seems like a pretty good guess. But it's still only a guess.