Lily and Sirius are the only characters who use the insult "toerag" in canon:
“You think you’re funny,” she said coldly. “But you’re just an arrogant, bullying toerag, Potter. Leave him alone.” “I know James Potter’s an arrogant toerag,” she said, cutting across Snape. “I don’t need you to tell me that. But Mulciber’s and Avery’s idea of humor is just evil. Evil, Sev. I don’t understand how you can be friends with them.” “What’s she got against werewolves?” said Hermione angrily. “Scared of them, I expect,” said Sirius, smiling at her indignation. “Apparently she loathes part-humans; she campaigned to have merpeople rounded up and tagged last year too. Imagine wasting your time and energy persecuting merpeople when there are little toerags like Kreacher on the loose —”
Headcanon that Sirius started using the phrase because of Lily (uh, ignoring the less than ideal comment in which he uses it…). People who spend a lot of time together often talk like each other and catch phrases from each other etc.
