NGL Saizo being so harsh on Asugi, while not necessarily justified, makes a lot of sense considering his own relationship with his dad, and I'm honestly surprised it isn't spelled out anywhere.
Saizo (and Kaze)'s dad was killed by Kotaro at least a few years prior to the game. With that, Saizo suddenly became head of the family and the representative of the Saizo line, in addition to his duties to Ryoma. Not to mention his own investigation into Mokushu, which led to his revenge which led to the loss of his eye. And as per his and Kaze's support, he didn't share his troubles with anyone else, not even his brother, and had been dwelling on his own shame for a long time.
Fanon tends to place Saizo the 4th's death around when our Saizo and Kaze were teenagers. Which means, essentially, Saizo suddenly had a lot of responsibility and trauma put onto him as a teenager, and as the eldest son, and presumably had a lot to live up to. Depending on how young he was, he might not have been ready to take on that role, and from there on out he had to figure things out on his own, without any guidance from his dad. And also his own issues he is very terrible at dealing with.
Saizo the 4th died in a relatively peaceful time compared to the timeframe of Fates, i.e. full-on wartime. Compared to his dad, our Saizo lives, and has his son, in much more dangerous times, especially considering the family business and specifically his duty to protect the king. He could, and is prepared to die any day. But he has a son who (in his mind, to not let his ancestors down) will be the next Saizo.
So Saizo feels the need to train his son for the role, even if it means pushing him insanely hard, to make sure that Asugi will be more prepared than him to take on the same weight and duties that he once had to, when he himself dies while his own son is still a teenager.
Implicitly, Saizo also looks up to his dad a lot, which adds to his pressure to live up to the Saizo name. And then consider that Kaze, in comparison, pretty much never brings up their dad outside of his conversations with Saizo.
For all that their dad was respected / loved / whatever by one son, the other doesn't really seem to think about him. There are a lot of reasons why this could be, but Saizo Sr. probably wasn't close with Kaze to begin with, likely since Kaze wasn't the heir. Not to mention, Kaze and Saizo began their training at a very young age as well, which is why they're so skilled at their work.
Which is to say, Saizo Sr. might not have been the best dad either, and Saizo is using him as a reference because he respects him, and there aren't exactly any other fathers around to teach him how to parent. In short: generational trauma!
And as for why Kaze and Midori aren't like that? Mostly because Kaze wasn't heir, didn't have any of these burdens, and is far enough removed from the situation to see how placing such pressure on someone so young hurt his brother, if not just himself. He would rather not hurt his own daughter in the same way, especially since he doesn't have to. So Midori is free to not be a ninja, and pursues her own interests instead, and is very close to her dad.
However, a huge part of Midori's character is that she's very capable despite her age, is still working in a field that allows her to be relatively self-sufficient, and her dad was the one who taught her. Kaze is also preparing his daughter for the possibility of losing him early, so she can at least survive on her own when he dies. Which is a subtle similarity to how he himself was raised - even more generational trauma, but gentler at least? Kaze probably doesn't know how else to raise a kid either, but he's trying to be different from his father and brother at least.
TL:DR Saizo's relationship with Asugi is the way it is because of literal fatherless behaviour, and so is Kaze's with Midori to a certain, less messed-up extent.
