I bought an English adaption of Baudelaire's Flowers of Evil, and hopefully I can answer your question.
Because they share the same title, I will refer to Baudelaire's work as "The Flowers of Evil" and the anime/manga as "Aku no Hana".
Prepare for some reading:
Similarities between the Aku no Hana's Story & Baudelaire's poems
Both Baudelaire and Kasuga have an obsession with a female "muse", In Kasuga 's case - Saeki.
Baudelaire has several poems with Muse in the title that refer to her grace and perfectness in a world full of woe (for him at least). In many of his other poems, his muse is mentioned through other features of the world - like "the brilliant sun".
He idolizes this woman, wanting her to remain untouched by the evil of the world - that she finds her happiness, of course not with himself - he isn't worthy.
We see this in Aku no Hana repeatedly, when Kasuga refuses to believe Saeki can be happy with him as her boyfriend, the fact that he steals her gym clothes, when he
explicitly calls her his muse.
Hating the world, but wanting to love it:
Both Kasuga and Baudelaire experience the world in a despairing light, generally the blame is placed on themselves for not being able to see the wonder in the world.
Extract from The Evil Monk:
My soul is a tomb where - bad monk that I be -
I dwell and search its depths from all eternity,
And nought bedecks the walls of the odious spot.
For the majority of the story, Kasuga is coaxed by Nakamura to do perverse deeds - Kasuga generally resists, wanting to be a normal citizen and enjoy his life if he can.
Acceptance that they cannot enjoy the world:
As time grows on in Aku No Hana, Kasuga more readily accepts his depravity and falls to Nakamura's ideas quickly, as they slowly become the only excitement that exists in his life.
Baudelaire also accepts that he will never enjoy the world as other people do.
The allure of evil & the evil Beauty
Baudelaire has a second female figure in several of his poems - perhaps it is a concept of beauty itself or someone particular in his life. This figure is of immense beauty, but revels in despair. In the poem "All Entire", the Devil himself comes to the poet's side to complain about this figure's alluring evil.
If Saeki is the muse, this figure is certainly Nakamura in the anime. Kasuga slowly falls into complex feelings for her, knowing she is a bad influence, but clinging to the enjoyment that life provides.
The show depicts Nakamura in a constant evil light. The only time we gain an insight into her true self is a glimpse of her diary, where it is written about her joy of finding a pervert equal to herself. In a way, Aku no Hana is actually about Nakamura's unveiling rather than Kagura's fall into despair.
Extract from Hymn to Beauty:
From God or Satan? Angel, Mermaid or Proserpine?
What matter if though makest-Blithe voluptuous sprite-
With rhythms, perfumes, visions- O mine only queen!-
The universe less hideous and the hours less trite
The final lines show a little bit of the relief Baudlaire recieves from embracing this Beauty's evil. We see this in Kasuga's actions too when he gradually starts to enjoy the despair and Nakamura's wiles.
As a side note, we can see Kasuga's conflict of loyalty to Saeki or Nakamura to be a reflection of his internal conflict to either conform or reject society. Saeki selflessly forgives him repeatedly, willing to accept him. Nakamura, on the other hand is full of angst and rejection.
A good few themes are in Flowers of Evil that don't appear in Aku no Hana. Baudelaire makes repeated use of seas, waves and water in his poems is one that springs to mind
As mentioned in @krikara's answer, there is indeed a few poems mentioning death, Baudelaire seems to alternate these poems between the theme of erotisism of the darkness in death and treating Death as a proof to his meaningless existence - that it is all for naught once you are dead.
He also often sees the world through lenses of death, in which he describes beautiful surroundings, but through images of decay, loss and ruin.
So? Get to the point already!
All of these strong parallels strongly indicate that Kasuga's personality is quite like Baudelaire's, and as of such he was taken by the work. We all know it's easy to get obsessed by things you enjoy (look at all these anime questions I've answered :v) and Kasuga probably became obsessed with the book as a result of this similarity of minds.
Add in his situational circumstances - his fascination with Saeki, his seduction into darkness by Nakamura and you can see how a work would be influential on his life.
Although, Kagura does mention at one stage (on the mountainside I believe) that he enjoyed reading Baudelaire's works to feel more intelligent than the others in his class, despite not understanding them. So you can make a judgement call on that too.