I would go with Richard. From my understanding, Hyacinth wasn't always like how she is during the show. At some point, something made her believe she could/should be of a better class.
This is when the candlit suppers, the bragging about her sister (You know, Violet. She's the one with the Mercedes, swimming pool, and room for a pony) and her constant cleaning began.
From Richard's perspective, the woman he loves and build a family with has gone completly, utterly insane. The man fought with his bosses, who wanted him to take an early retirement. Richard really did not want to spend time his wife.
They can't take a simple car ride without expecting him to open the door, listening to her remarks and warnings about lorries and sudden changes of directions/destinations because she saw someone from the bridgeclub she wants to impress.
Not to mention when their son, Sheridan, calls constantly to "borrow" money for one reason or another. Richard is clearly done with throwing money into that bottomless pit, but Hyacinth always ends up either giving Sheridan the money or make Richard give them.
She is even ashamed of their last name, HIS last name. Always making people say it the "right" way. Hyacinth doesn't seem to care about Richard and his needs.
Their neighbours are AFRAID of the woman, her family thinks she is nutty and most of the community tries to keep her out of events. People moving to the neighbourhood are even warned about Hyacinth and her antics.
People look at Richard with pity for being stuck with this nutcase of a woman. It is commendable that he hasn't already strangled her with the phonecord.