And this is why you need ethical theory not just a general preschool sense of not being mean!
Compassion is a great place for ethics to start but a terrible place to finish. There is so much farther to go. You have to be able to interrogate your feelings and ask questions like:
Why do I care about these people and not those people? Is it because these people are more worthy, or is it just a psychological quirk like "these people have more in common with me"?
Is the action I am moved to take actually going to solve anything?
Is my goal the wellbeing of others, or to soothe my own conscience so I can go back to ignoring it?
I know it would be nice and easy if we always had the correct moral intuitions on first blush every single time, but that seems pretty unlikely to me!