"What if Jeremy actually did something horrible to his family."
I don't know... Because the butler is nice to Jeremy. And someone who works for rich people wouldn't be able to tolerate a rich child who did something horrible and got away with it. The butler is kind of definitive proof that Jeremy's family is in the wrong. And based on the butler trying to help Jeremy avoid his older brother, and the fact that Jeremy was who Lucas felt safest calling when Grayson hurt him, I think Jeremy's older brother is the problem.
Maybe Jeremy's older brother did something terrible to Jeremy and Jeremy's family tried to insist on sweeping it under the rug, refusing to get the law involved because they don't want their family to look bad, but his older brother did whatever it was again Jeremy's freshman year when Jeremy thought he'd be free and able to go live on campus for college, so Jeremy told... and all hell broke loose. Jeremy's family blamed him and did everything they could to bury whatever happened. Maybe if it got out, they even threw Jeremy under the bus and made it look like his fault or made it look like he was exaggerating for attention, and then to further prove their point, they made him live at home as a "Look, he has no problem with his brother and everything is perfectly fine here" thing.
Or it's still an older sibling who did something to him, but it's the missing one and that one got sent off to "heal" because the family refuses to admit that whatever he did to Jeremy was anything other than a misunderstanding or something that can be fixed if he reflects on it and just works on himself for a while.
Or maybe whatever older sibling did something to Jeremy is a cop, hence Jeremy's discomfort when he saw the group of cops (who would, like Jeremy's family, look out for Jeremy's brother and vilify Jeremy).