SingerOfTheFall's answer has covered the relation between anime and light novel: volume 1-3 & 5, skipping 4. For the detail, according to Japanese Wikipedia:
- Season 1 adapts volume 1 & 2 of the light novel almost faithfully (+ volume 7's "The Red of the Apple, the Blue of the Sky" on the DVD-exclusive episode 7)
- Season 2 adapts volume 3 & 5, skipping 4 (+ volume 7's "The Wolf and the Amber Melancholy" on the OVA episode 0)
Given the above fact, the anime covers volume 1-8 of the manga with quite heavy modification. According to Japanese Wikipedia:
- Beginning of light novel (LN) 1
- Middle of LN 1
- End of LN 1 - Beginning of LN 2
- Beginning - mid of LN 2
- Middle - end of LN 2
- End of LN 2 + LN 7's "The Wolf and the Amber Melancholy" (anime season 2's OVA episode 0)
- Part of LN 3 + Beginning - mid of LN 4
- End of LN 4 - Beginning of LN 8 (LN 5-6 are simplified)
For the manga, you might start from volume 7 instead, for the skipped volume 4 of the light novel.
Note: as of July 2017, the manga is still on-going and consists of more than 8 volumes. The above list ends at volume 8 which covers volume 5 of the light novel because that's the last volume that got adapted into the anime.