According to interviews with co-founder Fred Patten, the Cartoon/Fantasy Organization (C/FO) was the first anime/manga fanclub set up in 1977. Based in Los Angeles, the group started expanding into several spin-off groups in the 80s, including another major group in New York:
1980s C/FO New York meeting.
Most ways of accessing anime & manga were limited back then. You either had to know someone over in Japan, or visit one of very few Japanese bookshops. Nearly none of which having English translations.
(src)
Whilst there may have been some small gatherings of friends before the C/FO, it was the first group to be open to the general public.
Many of the early members of the C/FO became prominent figures in the emergence of furry fandom.
Unfortunately, due to political in-fighting the splinter groups gradually ceased to exist, leaving only the original L.A. branch active. Their current website was last updated in 2013.
Even today, Los Angeles is one of the most important places in english-speaking territory for anime, hosting one of the largest conventions - Anime Expo