Usually we would treat the manga as canon for most works, but as most people usually only have watched the animation or played the video games, would in this case the manga still be considered canon over the anime? Or would it be the videogame and manga based on it?
Canon usually refers to the original work rather than the most popular one. As such, I would imagine that the original Japanese manga is canon.
The other mediums can be canon as well; the anime for example is a very important part of the franchise. Stil, the manga would probably still take precedence.
Canon is a very broad term that can be as strict or lenient as the user who uses it, so it's hard to answer this question definitively.
The video games, manga, and anime are all different stories. Each is canon only to it's own plot. Since the video game came first however, we can assume the video game is considered canon unless you are speaking directly about the anime or manga, in which case that universe takes precedence. In the absence of information from the video game, I would say information from the anime comes next, since it came out before the manga. (Though arguably the manga fits the games better then the anime, therefore it takes the spot at second most canonical.) It entirely depends on how you see the Pokemon universe.
Pokemon adventures is often considered to be more canon to the games. Even though the manga was not made by Satoshi himself, you can see how canon it is to as Satoshi Taijiri once stated, "This is the comic that most resembles the world I was trying to convey."
While the current producers of the Pokemon cartoon series have tried to make all the official movies to be canon to the cartoon series. As the series is only canon to things relating to Ash Ketchum.