When people invoke this sentiment, i think they often make the same error seen in, say, calls for stronger online content moderation: a lack of consideration for who is going to do this work.
In the US today, around a quarter to a third of aides and caregivers for the elderly are immigrants [1, 2], the vast vast majority of whom are women. (From what I can tell) a major reason for this is that nursing or social work degrees/certification programs provide a straightforward (well, relatively) pathway to citizenship or permanent residency; and many simpler caregiver jobs only require a high school diploma, which is useful for immigrants in the US who may not be able to attend college. While I'm sure many of these workers have a sincere passion for caregiving, in a world without severe immigration restrictions and these obvious pipelines the numbers would not be this skewed. I do not think most of these people would put these careers as their first choice.
For starters, they do not pay very much. Average base salary on Indeed for caregiver jobs was $16.83 an hour. For seconders, how often have you had to deal with older relatives making sexist, racist, or otherwise uncomfortable comments at family gatherings? Frankly, lot of old people suck to be around! So do a lot of mentally ill people! Mentally ill old people (so, a lot of the ones receiving home care or living in facilities) can extra suck to be around!! Sexual harassment, violence, and abuse towards workers is common in home care and facilities. [1, 2, 3]. Many caregivers of color experience racism from coworkers, patients, or patients families. Now, many of these people are not fully in control of their thoughts or actions (hence why they require care). And even if they were, being a racist abusive serial harasser doesn't mean somebody deserves to be tortured or medically neglected or left to die.
But we aren't talking about abstract hypotheticals and who "deserves" what. We are talking about real people. When people say things like "the worst person you know still deserves care", it feels like they are expecting the care provider to just manifest out of the ether, and be willing to do whatever is needed without complaint. How much money would someone have to pay you to visit your shittiest relatives for eight hours a day? How long do you think you could retain a warm, kind bedside manner while being sexually harassed, called slurs, or physically assaulted? A week? A month? This is what providing care for the elderly, disabled, and mentally ill can entail, and this is what needs to be addressed if we want to prevent abuse and neglect of patients.