Dyke Changeling

I feel that Call of Cthulhu particularly fails to capture anything that makes its antagonists terrifying in the sanity mechanics and it usually just manifests as a penalty for interacting with otherworldly creatures.

Alien does a fear mechanic that's similar to sanity but with a strong short term panic focus.

The only system I know that seems to have made a real effort in innovating here is Unknown Armies.

Eureka: Investigative Urban Fantasy is a TTRPG about solving mysteries, where someone in the mystery, the fellow investigators, or your investigator that you play may or may not secretly be paranormal.

The 'sanity' mechanic in question is called Composure. An investigator starts with 7 Composure. Things like financial stress, being threatened or attacked with a weapon, or seeing a gorey corpse are placed from +3 to -3 on the Tiers of Fear during character creation. When that fear comes up, they roll applying its modifier, losing Composure on a low result.

An investigator's bonuses are capped by their Composure, so if they would normally be a charismatic person with +3 Manipulate, at 1 Composure they can only add +1.

If an investigator would lose Composure while they have none, it starts eating into their health, and can even kill them from the stress.

To regain Composure, investigators have to eat three square meals or get a full night's sleep, or from a direct Comfort roll from other investigators. But for investigators that are monsters, their options are much more limited, usually they have to engage in their True Nature which often involves killing and eating people.