"One study even found that the more men there are taking a maths test in the same room as a solo woman, the lower women's performance become. And, surrounded by men, she herself may come to grudgingly believe that women are indeed naturally inferior in maths.
The mind turns from a focus on seeking success (being bold and creative) to a focus on avoiding failure, which involves being cautious, careful and conservative. A mind that is struggling with negative stereotypes and anxious thoughts is not in a psychologically optimal state for doing taxing intellectual tasks.
Stereotype effects have been seen in women who: record their sex at the beginning of a quantitative test (which is standard practice for many tests); are in the minority as they take the test; have instructors or peers who hold sexist attitudes or have just watched women acting in air-headed ways in commercials."
Delusions of Gender by Cordelia Fine
Add to that the pressure of having to represent all women: indeed, every time a woman fails against a man it is held as proof of the unfitness and inferiority of our entire sex. I also remember a study that showed men try harder when facing a woman, they really make it a point to destroy her, because it's not just a game to them it's about maintaining their dominance. Worth noting that male chess players have been sexually harassing and threatening female chess players since the begining. You make a piss poor political commentator if you refuse to take any of this into account (or are simply ignorant of those facts) as you complain about women not wanting to compete in a male environment.
And lastly, I'll say it again, even if none of this happened, women don't need your permission to spend time together and play together and exist without men. We don't owe men our attention.