Doc's father already didn't approve of his sciencing, so he was already failing as a daughter. And with how eccentric he is, he didn't particularly interest potential suitors, so he wasn't really wife material either. Failing at two of the major things that women were supposed to be, he may have decided to just throw out the whole 'woman' thing while he's at it. He's always wanted to be a man, so why not do it? What does he have to lose?
I don't know how much Doc knows about healthcare and surgery, but I believe he would have taught himself enough to be able to do some of his medical transition himself. Also, he used to be pretty wealthy, so he could have paid for surgery and hormones if he couldn't/didn't want to do it himself.
As the years pass and Doc continues to keep to himself, people in Hill Valley forget that the strange, eccentric inventor used to be a woman.
And then he meets Marty, who at that point is still going by his deadname. He knows he doesn't like being a girl, maybe he knows he wants to be a boy, but he doesn't know it's a possibility.
One day, the gender thoughts won't leave him alone, so he hesitantly asks Doc if he's ever wished he was a woman. (If it's normal for men to wish to be women, then it's normal for Marty to wish to be a boy and therefore he can go on trying to ignore this.) Doc has a brief moment of panic thinking Marty has somehow found out he's trans. (He hasn't.)
They talk and eventually Marty admits (though he tries to make it sound casual) that he wishes he was a boy. Doc just pauses for a moment and offers Marty some of the testosterone he brews in his garage (thank you @bishops-severed-torso). And that's how Marty finds out Doc is also trans.
Doc helps get Marty a proper short haircut, new more masculine clothes, and gets him started on T. The T part is really illegal, but then again Doc conned stolen plutonium from terrorists, so this is not even out of character. In both timelines, Marty continues to speak with a higher voice at home and Marty's parents just aren't really paying enough attention to notice any other effects of T.
Anyway, trans Doc (and Marty)
