MYS

But he wants to be a hero. And there's someone right in front of him who looks like they need saving. So against quite literally all his better judgement, Keigo holds out a hand to Touya and offers him at least a roof over his head and an occasional meal.

We've got:

Complaints about Izuku and Katsuki's "inadequacies" ...

... that both of them are deeply affected by:

And physical violence that's played as "comedy" ...

... but then is referred back to in more serious scenes:

(sorry for the gif, the anime makes it much clearer that Izuku is flinching here)

Now Toshinori and Mitsuki obviously weren't intended to be abusers the same way Endeavor or Kotaro were. As awful as Horikoshi treats the manga's abuse storylines, the Todorokis and Tenko do suffer realistic psychological effects of the abuse, their abusers are either "redeemed" or killed off, and nothing about the abuse is ever portrayed as even close to comedic.

But tbh, based on the information we have, I understand people interpreting both characters as abusers. They're meant to be toxic adults whose violence is exaggerated but whose actions and words hurt their charges nonetheless. It's not a huge jump.