Was Circe breaking xenia by inviting Odysseus' men in to eat and then cursing them? Can you punish someone for trespassing when you've welcomed them as guests? Is it self-defense to preemptively curse men, or does the "we lied to protect ourselves because we are weak and feeble women" justification fall apart because she's such a powerful witch goddess?
So, the TL:DR here is "she broke the rules, but only if we assume the rules APPLY to her, which they very much might not."
I will admit — I am biased about this. My undergraduate thesis was about Circe, in the context of how she compares to both other Odyssean women and other morally questionable women in Greek mythology. And I got VERY attached to her character, specifically my understanding of it...
...Which is why I seethed at Madeline Miller's version of the story nearly as much as I did Christopher Nolan's. As you said, she is a powerful witch goddess, not a vehicle through which to explore how vulnerable women in ancient times often were. She's pointedly the exception to that rule! Circe should be powerful, which is why the novel that bears her name isn't really about her at all. In the Odyssey, everything we see and hear about her suggests that she is extremely independent, confident, capable and self-assured. She sleeps with Odysseus, either because she wants to or as a power play, but she has no need for him. She's surrounded by loyal handmaidens as well as domesticated beasts that respond only to her. She has to kowtow to the Olympian gods, but not to anyone else, up to and including her own father. She's a corruption of ancient Greek ideas of femininity — that's what makes her scary (to ancient audiences), and that's why she can't be expected to play by the rules.
But she did break the rules.
According to xenia, once you welcome someone into your home as a guest — which you are obligated to do! — you cannot harm them unless they directly try to harm you first. Because that means that they have broken xenia already and you have no obligation to uphold it either.The idea that the Ithacans were trespassing or that Circe cursing them was some kind of self-defense is kind of nonsensical...
...If we assume Circe is beholden to the standards of xenia even though she's a goddess. Which she probably isn't, at least not by default!
It is in fact a whole Thing in Greek mythology that gods do not dine with mortals, or if they do they do so in disguise and/or as part of a religious ritual. And Circe is alternately called a nymph and goddess — she's not a mortal. She's not necessarily obliged to follow the rules of xenia. She has no default obligation to play host to anyone who lands on her island, or to treat them courteously. She can be as callous as any of the gods.
And notably, the events of the Odyssey and specific way Circe is defeated does reinforce this interpretation. Because her changing her tune towards Odysseus and his men happens because even more powerful gods strongarm her into it. Athena, because she likes Odysseus, sends Hermes to go give Odysseus a drug, molly, that counteracts Circe's pharmaka (a term that, interestingly, can mean both poisons or medicines). Odysseus, immune to her magic, threatens her with a sword, and Circe realizes then that 1. This guy has the backing of more powerful gods than her, and 2. That he is Odysseus, a hero who was long ago prophesied to come to her island. (We never get to hear exactly what those prophecies were, but Circe and Polyphemus both state that Odysseus' arrival was foretold. But generally, prophecies were about important shit.)
Anyway: Odysseus is able to best Circe only because more powerful gods than her are on his side. That...kind of does imply she could otherwise do whatever she wanted to mortals and have it be fine.
It's also worth noting that in the original Odyssey, she does change her tune. After Athena and Hermes intercede on Odysseus' behalf, Circe does a total 180 and becomes a genuinely good host. The two of them sleep together, and then Circe turns Odysseus' men back, provides the crew with good food and wine and fine clothing, and after doing so for a full year she finally sends Odysseus off on his journey with provisions and specific, very useful advice about how he might (or might not) get home. She's the one who tells him to go to Tiresias in the Underworld, to warn him about Scylla and Charybdis, and to tell him not to eat her father Helios' cattle.
Admittedly the whole of Circe's part in the story is told in first person by lying Odysseus, so we take it with a grain of salt, but...he doesn't exactly have motive to lie and make her look better. Only worse. So I think we can't call Circe wholly unreasonable, at the absolute least.