I understand your reasoning here, but I think there's more to it, or I just understand it in a different way. Most of the discussion against the idea of misandry as a concept I've seen has been pushback against trans men talking about the oppression they face, and calling that oppression transmisandry/transandrophobia. And the common refrain against that is "men can't be oppressed for being men."
I've used the example of black men being oppressed to refute this before — not in the sense of black people *in general* being seen as hyper-masculine (which I'm not disputing) — but in the sense that anti-black racism targeted at black men specifically has differences to anti-black racism aimed at black women or at black people in general.
Bigoted rhetoric can be un-gendered, in that it is targeting a group as a whole, and not aimed at either men or women respectively.
Non-gender specific racism: black people are lazy, do drugs, are generally inferior in some way
Racism aimed specifically at black women/misogynoir: welfare queens, perceiving black women as unfeminine, hypersexual, angry etc.
Racism aimed specifically at black *men*: gang involvement, absent fathers, hypersexual and hyperviolent, "super-predators" etc.
All three of those things are related, but function differently. Intersectionality is often (and correctly) used to explain how people who are marginalised in multiple ways have that marginalisation compounded, but I think intersectionality also applied to scenarios where a way in which someone is marginalised and a way in which they aren't is *also* compounded. I don't think it's accurate to say what black men experience is purely racism, and that gender dynamics do not play into that at all. I think that to call black men's experiences of gendered racism purely racism is to consider men the un-gendered default against which women are differentiated. And I think something is lost in doing that.
In a white-supremacist, patriarchal society, black men's access to male privilege is mitigated by their being black. It's by no means non-existent, but I'd argue it is hampered somewhat. In the same way, I'd argue that white marginalised people's experience of bigotry is mitigated by their whiteness. In the same way as I think viewing black men's experience of racism as un-gendered is reductive, I think viewing any bigotry against white marginalised people as un-racialised is reductive. It's a perspective that I think unwittingly presents the experiences of those favoured by the oppressive structures we live under as the default, normal experience — and the experience of anyone else as "other".
That's not to say that misandry as an isolated concept is something I believe in. Obviously, under patriarchy alone, men are not oppressed in any real systemic way, and while men may be harmed by the patriarchal gender norms they're taught to uphold, that harm is not the same as systemic oppression. But I don't think we can look at any oppressive structure in a vacuum. It's all connected, and in any given situation, one aspect of someone's perceived identity might take the back seat to another, like how class allegiances might outweigh solidarity with someone's racial group or sexuality if money was on the line.
Patriarchy is complicated in that men are in two positions at once under that framework: both the owner and protector of the women considered their property, and the aggressor/predatory force that property needs to be protected against. [Women are also in two positions at once, both the "valued" property of men, and a commodity to be exploited.] Under patriarchy, men are at odds with other men, in a way that white people are not at odds with other white people in a racial context. This puts black men in the position of being at odds with white men in two ways at once, which I think explains some of the emphasis on black men as aggressors towards white women. It's a concern about racial purity, and about patriarchal ownership of white women being the exclusive right of white men.
So while I don't think "misandry" on its own is a real thing, I *do* think someone being a man can be a factor in how they're oppressed for other reasons.
[And also that saying trans men can't call what they experience transmisandry/trandandrophobia because "men can't be oppressed" is reductive, and that saying what they experience is purely transphobia or is just misogyny is both missing something, and kind of pointless fighting about the name people affected by an issue give that issue.]