It’s not addressed often, but based on what little we’re told, it’s very likely that Ah-Duo was (and still is) the love of his life.

She was the Emperor’s childhood sweetheart. She was his only consort for over a decade. Ignoring how rare that is—how it would have been encouraged for a prince to take more than one concubine—he refused to take on any other consorts for the first ten years after Jinshi's birth. He only accepted other concubines after the former emperor passed away, and he ascended the throne.

For as the current emperor, he could no longer avoid the obligation of taking other consorts, producing heirs—especially since he knew Ah-Duo could no longer bear him children.

Despite losing her womb, the child Ah-Duo gave him is the only one that has survived into adulthood. A child he dotes on, loves dearly, and who he deeply wants to be his heir, despite Jinshi's clear objections. And it makes sense why he'd want Jinshi to succeed him. He's the only child he bore with the woman he loves most, who bears such an uncanny resemblance to her, who shares her intelligence, charm, and cleverness.

He loved her so much that he kept her—a woman who could never get pregnant again—in the rear palace for over eighteen years. It hurt their relationship to have her caged there. The one solace they had was being able to be near Jinshi as he grew up.

And by the time the Emperor gives in to the political pressure to replace Ah-Duo with a younger (fertile) consort, he does something that we haven't seen any other emperor do so far: he allowed Ah-Duo to leave the rear palace. Not only that, he gave her the Southern Imperial Villa: an entire compound (comparable to a small palace itself) to call her own in the royal capital, where she can live her life comfortably and privately. It isn't true freedom—not really—since she doesn't have jurisdiction to travel the world as she's always wanted, but it's a hell of a lot more freedom than any other consort has ever received up to that point.

And he leaves the palace often to visit her there. This is extremely uncommon; in theory, an emperor should have no further reason to interact with a former consort if the woman cannot serve the baby-making purpose she was chosen for. He gave her a residence in the royal capital because wants to keep her close. He wants continued access to her, even after she's comfortably installed in her new life outside the palace.

Jinshi knows all of this, and is painfully aware of how easily history could repeat itself with him and Maomao. Jinshi knows how trapped and stifled Ah-Duo felt in the rear palace, knows it strained her relationship with the Emperor, knows that it ultimately pushed her away from him, emotionally. It did not affect the Emperor's love for her, but it did affect Ah-Duo's love for him.