What you’re going to do is go to Wiktionary.org and search the terms you want to work backward from. In this case we want to search Sir, and we also want to search Madam, which is what Ma’am comes from. 

On the page for Sir, we click “etymology” under English or scroll down to it, and we get the history of the word. How far back you go in this history can vary by what kind of word you’d like to use. In this case we know the history goes sir > sire > French Sire (master, sir, lord) > Latin senior (elder) > Latin Senex (old). I like to go all the way back to the Latin, but let’s hold that thought. 

Now that we have Sir identified, we’ll check out Madam, from which we get the history ma’am > madam > madame > Old French madame (”my” and “lady”) > post-classical Latin mea domina, which also means generally “my lady” although it has a more specific meaning we’ll get to shortly. 

So we have a couple of options! 

We can take “Senex” which is more closely related to the masculine “Sir” but is in itself generally neutral, and come up with “Sen”, which has no meaning in Latin on its own but we’re not speaking Latin, we’re speaking English, which shortens everything anyway. 

We can also look at “ma domina” and take that apart -- domina and dominus concern the home, the physical building, using the same root we get “domicile” and “domain”. So you could click through from domina to dominus to domus, and go with “ma domus”, since domus has connotations of household, family, etc. Ma Domus might shorten to M’us. It could also shorten to “ma’do”, but that’s two syllables and I like to retain the syllable count of the original words. And also M’us or even just Mus sounds like you’re saying Moose. Which, Moose is a pretty cool name to call a nonbinary friend, but may be taken amiss by strangers. It strikes me that M’us could be used as a term of respect specifically for someone in your family -- a parent or grandparent, a cousin or zaza. There’s a hint of familiarity there. 

We could go one step further and look at the implications of the word origins -- both are addressing a superior in rank, but “sir” emphasizes age, while “ma’am” emphasizes economic power. Now, if we want to break away from both of those we could decide that instead we want to respect a different kind of power -- say, the power of a teacher we trust and look up to. Wiktionary tells us that teacher derives from the verb “teach”, and at the etymology of teach we find several variants including techen, taecan, taikijan, taikijana, and deyk (as a prefix). I rather like Deyk, because a) it shortens nicely to Dey, b) if you’re talking to someone you respect it’s sincere but if you’re talking to someone you don’t respect it’s easily sarcastic, and c) if you’re talking to someone you don’t respect you can throw a little k in, so that it sounds like you’re calling them Dey but you’re actually calling them Dick. 

 Of these options I really do prefer Sen. It sounds nice, it’s not a homophone for anything weird, and it implies respect for the person’s experience. If I were writing a novel with a nonbinary honorific I might go with Dey just because there’s more scope for wordplay and nuance, but in actual life I think Sen’s quite nice.