The only consistent drawback of remaining possessed is having your first child being the reincarnation of a devil from old hell. The main goal of a loose soul is to be returned, and since they do this through the first offspring of their host (see below), they wouldn't want to adversely affect said host too drastically.

Aside from that, like you say, certain girls that host a Loose soul have a particular "power", such as Kanon's invisibility, or Tsukiyo's shrinking. Even if not having a certain "power" as such, the Loose Soul can still heavily affect its hosts personality (e.g. Shiori, who has no specific power, but reads each book in the library and lives a secluded existence). These are all unique to a particular girl, and even though they vary greatly in severity, they are still definitely drawbacks that would leave less than desirable living conditions if left unchecked.
Deciding on the most potent drawback depends on the girl, I suppose, but there aren't any positives to take away from being possessed!