But also like with what I commented on the last post about how the main difference between book Moriarty and Librarians Moriarty is appearance description and I proposed certain things reflect Sherlock (or John's) altered perception of reality.
But anyway, what if John Watson has godawful eyesight but doesn't actually know he does. And what if Moriarty has this elderly woman or man who works for him and lives with him as like a cook. And they're close so he goes places with them. So the entire time John thinks he's talking to Moriarty it's Moriarty's bemused cook while the real Moriarty stands somewhere nearby kind of just letting it happen.
Sherlock could easily tell John "Hey, that's not Moriarty. That's Moriarty over there," but he's not reading Watson's memoirs so he has no idea this is even a problem. He just thinks Watson has a perpetual annoyance with Moriarty's cook.