@N33R this pretty much.
I would not wanna associate with these toxic men either. Like ewwww. dudebro cult is something else.
Should not be too hard to built a alternative to that. Especially talking about stuff like a bookclub.
Like hell I did manage to find some women-dominated spaces (Lolita in this case on reddit) where they do not mind me being there as a male. Just be respectful and enjoy the fashion and talk about it. (I guess it does help though that like the fem side of Lolita Fashion) There is Ouji which you could pull off as a guy as well though.
Though I guess count in the fact that I like crossdress as well so I feel way safer in a women dominated space generally.
That said I would not try to join some explicitly female-only groups. If they not want me thats fine too. Plenty of other spaces where they not mind right?