While Yakuza these days are mostly semi-legitimate, they still extort and is the largest criminal organization in the world. Just like how there's a taboo with tattoos in Japan because they're associated with Yakuza, the same connotations come with speaking like the way Yakuza do.
I believe it refers to じゃねえ だぜ 便所 (ja nee, da ze, benjo 'it isn't' 'it is' 'toilet') and the like, which are hardly exclusive to the Yakuza in real life. They are simply masculine and very rough. While all Yakuza probably use them, not all the people who use them are Yakuza. They're used to make the heroes sound more rough and masculine, to appeal to that sort of demographic. Of course, while it was popular a while ago, nowadays anime characters who use them are laughing stocks.
(There is Yakuza slang, but I don't know any of it. Anime Yakuza have a rather defined pattern of speech based on Hiroshima-ben, but that's probably inaccurate.)