I really appreciate this reading of Gaston, because I think he too often gets reduced to haha-big-dumb-guy. He's canonically not stupid. He uses big words and quotes Macbeth! He comes up with a clever plan to get Belle on the spot, and when he later implements that plan and it starts going off the rails, immediately comes up with a new plan and knows how to get everyone on his side. He's intelligent! So yeah, I can buy into the idea that the dumb masculine things he does are more for show.
One of the actors who played Gaston on Broadway said that when he thanks Belle for calling him "positively primeval" it is not because he's dumb and doesn't know what it means, it's because he knows exactly what it means but thinks it's a compliment, and I've adopted this interpretation of this exchange. It goes along perfectly with your analysis. He's trying to appear like Big Strong Primal Man and then Belle (to his mind) titters and essentially says "ooh yes you're such a Primal Man," so he's happy that it worked, thanks her, and immediately suggests they got look at his trophies (dead animals) for more evidence of how primal he is.
He also thinks Belle is playing along during his proposal, because she does. When he cages her inâwhich is a total bodice-ripper romance trope that you still will see in fanfic constantly because lots of women really do love that performance on masculine aggression (as is their rightâI find it hot in the right context too, no kink shaming) and Gaston totally knows thisâshe flutters her eyelashes at him and gives him the stock Demure Lady response.
I will take issue with the idea that Beast tries to perform masculinity with his anger and intimidation. I don't think this is a performance; these are his genuine reactions and sincerely chosen behaviors. That's the real him. And the point is that the real him simply changes to not want to do these things anymore. (Though I headcanon he will still have a bit of a temper for his whole life because that's just part of his personality and no one is perfect.)
But otherwise, I agree with everything you said about Beast's arc! Yes, he does reject masculine displays as the movie goes on. Beast is given the chance to play Typical Man by Cogsworth when he suggests Beast give her "the usual things: flowers, chocolates, promises you don't intend to keep." It's not malicious, of course, but it is stereotypical. Beast rejects this idea and decides to be unconventional and give her the library.
He doesn't do the typical Tough Guy and fight back against Gaston at firstâwhich, granted, is just because he wants to die, not because he thinks it's morally wrong. But then when he does fight back and has the chance to deliver the killing blow, he shows mercy and gentleness and lets Gaston go, again eschewing Tough Guy behavior specifically because he knows it's wrong.
Gaston, of course, does the opposite, and dies for it.
Great analysis OP! Thanks for sharing this!