For the original Dissidia, it's possible that it started development around the same time the 3D remake did, and by the time that was shown off, they'd already locked in Onion Knight. It's also possible that they wanted a character to represent the 3 "endgame" jobs of the Famicom version (Ninja, Sage and Onion Knight) and Luneth is more associated with the Warrior class and jobs like it (i.e. physical jobs). Basically, Onion Knight represents the famicom version and III entirely to an extent, much like how WoL represents FF1.
As for why it keeps happening - more specifically, why Luneth's party never shows up if Onion Knight does first - I can think of a few possibilities:
SE feels like things would get complicated if they had the main characters of two different versions of the game co-existing, so they take the easy way out (Kinda rings hollow when Opera Omnia had played coy with XII and T's Ivalices being different or adding Jack and his gang (for that matter, did WoL ever interact with SoP!Astos? Or acknowledge that he also fought a dark elf named Astos?). Also friendly reminder that WoFF has Refia in it and WoFF characters were in OO. Is Refia just Enna's OC in the Dissidia timeline? My only explanation in this case is that there might've usually been enough of a degree of separation in terms of those games' casts interacting that wouldn't be there for III and the devs don't want to make a "definitive" statement on the situation.*
Square-Enix seems to just avoid III content when it comes to characters beyond slightly over the bare minimum in general (it took almost 3 acts for III to get characters beyond CoD and Onion Knight in Opera Omnia, Therhythm for its entire run just had Onion Knight, Cloud of Darkness and Cid). From descriptions, it seems that each game has a department to handle them and refer to for spinoffs (hence the "Tactics/XIV pause" in OO when Yoshi-P's team were working on XVI), but I have no idea who handles III. One would assume someone from Team Asano, but I don't get that vibe with how III tends to be treated in spinoffs and stuff like the Octopath mobile game crossing over with IV and VI, but not III.
The Magic The Gathering crossover had a mandate (can't remember where the interview is) that stated that only 16 games were getting cards: Remakes were explicitly barred alongside spinoffs (the article highlights VII, but this would logically trickle down to III). But again, spinoffs and the like were also banned, while they're represented in Dissidia, so that can't be it.
I'm personally inclined to believe somewhere between options 1 and 3.
* Though funnily enough, DeNA had already gave Luneth's party some form of relation to Onion Knight in Record Keeper, but another game of theirs - Pokemon Masters - handled a similar situation much more easily and gracefully; In Masters, the events of Scarlet happened in the universe that the game takes place in with the girl protagonist (yes, Pokemon has a multiverse, don't worry about it) and had the boy protagonist who went through the events of Pokemon Violet (which had some minor yet fundamental differences) appear through a portal. It directly acknowledges that the boy is from a different dimension and that he has his own versions of the girl's friends with him getting along with her group very quickly. I dunno, it's just funny in light of S-E apparently being terrified of having Luneth's party and Onion Knight co-exist in anything resembling a story concept.
