In the case of many anime where the manga came first, things that are included in the anime but not the manga (such as filler arcs) seem to be not considered totally canon to the series, indicating that manga is generally what is considered to set canon. One example of this is the Quincy arc in Bleach. For series where the anime came first, such as Code Geass, is the anime then considered to set canon?
What is considered canon is generally decided by the author(s) or license holder. I think you need to get a better idea of the definition of canon.
the original work from which the fan fiction author borrows
a descriptor of specific incidents, relationships, or story arcs that take place within the overall canon
So, to directly answer your question, if there was only a manga and anime, and the manga came first, then the manga is canon. I say only, because many manga and anime are based off of light novels or visual novels. In that case, the light novel or visual novel would be canon.
If the anime came first, then it will probably be whatever the license holder decides is canon. This is an assumption, because I think anime writers give up the rights to their work.
When a series is presented in several media, most of the versions will usually be said to be "based on" some other version, and whichever version is not "based on" anything of the others is canon. Under normal circumstances, that one is the one that came first.
But this is only a general guideline. Sometimes things get strange. Consider Revolutionary Girl Utena, which has no fewer than four presentations: the manga, the TV series, the movie, and the manga of the movie. The movie-manga is based on the movie, of course, but the other three are considered separate canons. Sort of. Like I said, Utena is strange.
Canon is whatever the rights holder (people who own the IP) says it is. Canon is always changing. Just look at the American comic industry, DC and Marvel are always retconning and reinventing the canon.