I often see some really fun speculation about Zenos and Varis and the Galvus family and why their relationship became as shit as we see it in canon, but most of the time I think something people miss is that Varis was around 20ish when Zenos was born and he was a career soldier. We are told Varis was a brilliant strategist and from what we see in game he must have been a hella fine fighter when he was more active, but he was also very much defined by being a member of the Garlean military (contrasted with his uncle Titus who has always been framed as more of a pure politician).
We don't know what rank he was at any given time in his life, but considering how dismissive Solus was of him in general (probably as a result of comparing Varis to his late father Lucius, who was Solus' favorite son) and that the other Legati all seem to be on the older side (Gaius and Valens are both 56, Quintus, Vergilia and Basch should be in the same age range based on context clues and while the Nael we meet in game is younger at 38, she is explicitly pretending to be her older brother who was the actual Legatus — Zenos at 26 is an extreme outlier) I think it's reasonable to say that for many of Zenos' formative years Varis was probably trapped somewhere on the military ladder, even if he started at a higher initial rank due to being a member of the royal family.
So I think the root of the distant, cold relationship between Varis and Zenos is pretty clear: Varis was not there for Zenos growing up at all, because he was busy following his Legatus around, trying to impress his way into a promotion. His was likely an arranged marriage to begin with and producing an heir more an expectation than something he personally wanted, but even if he wanted to step up and be a good father to Zenos after his wife died... I'm not sure he had much of an opportunity to do so, honestly. Not without giving up his military and political ambitions and completely conceding any power and influence he had in the royal family. Definitely not when he was still at best a Tribunus, but probably not even when he was a proper Legatus (anybody remember how pissed Varis was when Gaius ignored the order to come the fuck home and help stabilize the empire and ran off to conquer Eorzea because that's what Solus would have wanted instead?). A Legatus holds responsibility for their legion, appears to be always stationed with their legion rather than commanding them remotely and they follow the orders of the emperor. We know that Varis had been stationed in some pretty remote places (Nangxia and Corvos were mentioned directly and it makes perfect sense that Solus would want to send Varis as far away as he could, if only so he wasn't around to remind Solus how much more inferior he was compared to Lucius) and I somehow can't imagine him having much of a direct influence on how Zenos was brought up from a continent away.
There was the incident with the Corvosi sword tutor abusing Zenos, but I honestly can't imagine Varis deliberately installing a tutor who he expected to endanger his son's life. He only had the one heir, losing Zenos would have been a big blow to his ambitions to the throne, so that entire incident was likely one giant oversight that came to bite him in the ass in short order.
The way I imagine that whole scenario went was something like... okay, imagine you have a son somewhere back in the homeland you only see a handful of times a year and otherwise only know about from the reports of the servants and his tutors. He's a really smart kid, but something of a problem child, completely unable to connect with his peers at all, so you likely stopped forcing him to interact with other kids of his age group/class because having your kid creep out (or possibly even hurt) the kids of your political allies is a very fast way to lose those political allies. His studies are progressing well, all things considered, but he also keeps going through tutors at an alarming rate (understandable for a neglected child, but that's probably not what Varis thought of when he received note of the nth tutor resigning in the span of as many months) and the acceptable pool of tutors is not infinite so you very much want him to stop. So, you send a warning: if Zenos keeps scaring off the regular tutors, he will get one he won't enjoy tormenting. Dangle some reward in front of the Corvosi swordsman so he has reason to take his tutoring role seriously, knowing full well that his resentment will seep through no matter what and make his teaching harsh and likely unpleasant, then wait for Zenos to regret this change in tutors. After Zenos was sufficiently cowed by the bad tutor, dismiss the Corvosi, rehire a regular one and hope that Zenos has learned his lesson, he should be smart enough for that.
Not a complicated plan, but it backfired spectacularly because Varis doesn't know Zenos on a personal level whatsoever. At the time of the incident Zenos was 14 and Varis still only showed up briefly and it's implied that he left almost immediately afterwards, so the most likely explanation is that he was in Garlemald to report in or accept new orders before he traveled back to his Legion stationed fuck-knows-where. Assuming he started spending more time in Garlemald after he became Legatus Augustus (being the High Legatus in command of the entire Garlean Military), he must have been in his late 30s at that point and Zenos was old enough that he left for whatever military training or academy they sent terrifying young men from a noble background to be bullied into being useful soldiers of the Garlean military, so their interactions would still have been minimal.
Most of their direct interactions likely happened when Zenos was already an adult, between a general and his superior commander, then between a prince and his emperor. They are only family by blood, completely passing each other by in life and they only know of each other whatever they have gathered second hand or projected onto the other and I think that's both fascinating and deeply tragic.