The important piece to understanding "casual oaths" is to remember that words don't have meanings in themselves; words have the meanings that they create in shared speech communities. If everyone knows your words to mean X, even if the literal content is Y, then your words mean X.
This is why it's not a lie to tell a fairy tale. Or, you know, to engage in self-deprecating humor, or sarcastically say "and obviously no one would ever do anything that silly". These are all denotatively false statements, but they aren't lies because they aren't interpreted to mean their denotation.
If you stand up and take the Hippocratic Oath at the end of a MD graduation, the meaning you create isn't "listen to these specific words and hold me to them". The meaning you create is "I have just graduated from medical school and am participating in the ritual that accompanies that graduation." And subsequently "violating" that "oath" doesn't say anything about your trustworthiness, because no one was interpreting you as making a serious promise to begin with.
(In the limiting case: if you're acting in a play on stage and your character swears an oath, you would never expect that oath to be binding on the actor. Even though the actor just stood up in front of a thousand people and spoke the oath!)
And you ask "But then how can we distinguish real oaths from performative oaths?" And the answer is that there's no shortcut because all communication is contextual. But we should take seriously promises that are offered seriously in a serious context. And we should take performatively promises that are offered performatively in a performative context.
There's a corollary question, which is where this started on the subject of marriage vows. How can you force other people to take your oath seriously, rather than performatively?
And the answer is, you can't. Meaning is created contextually through communities, and if your community chooses to interpret your words as meaning one thing you can't stop them. (Much like you can't use a slur and then explain "but I mean this lovingly!" You might genuinely mean it lovingly; but that doesn't matter if that's not the meaning it creates for your audience).
In your marriage, you can make a serious oath to your spouse, and both of you can intend that seriously, and it will be a serious oath. As binding on you as your conscience and your spouse dictate.
You can make the oath in front of a community that chooses to accept that oath as serious. And then the community can hold you to the oath, and bind you to it, as much as you abide by that community's norms and wish to remain a member of that community.
But you can't take a community that does not wish to accept your oath as a serious one, and force them to. Because, as the tortoise said to Achilles, you can't force people to accept a meaning they choose not to accept.