Some of the stuff from the TotK Master Works would have been so great for the game that either Nintendo is struggling to understand what its fans want and cut a ton of stuff that fans craved, or some of this art was created specifically for TotK Master Works and isn't true concept art. Stuff on the cutting room floor is often tantalizing, but some of this seems too perfect for a game with a full development cycle. (Here's a rhetorical question) Seriously, why was some of this stuff cut?
I think that perhaps what you and I might call “nuanced writing” or even “intriguing worldbuilding” isn’t a concern for most people who play Zelda games. This is nothing more than my own impression, of course, but I get the sense that the vast majority of series fans care about one of two things: gameplay mechanics, or being able to see conventionally attractive young people have a happy ending.
I don’t have an uncle who works at Nintendo, so I don’t have any special insight into the company’s decisions, but I’d assume the devs were aware they’d be competing with Final Fantasy XVI and Elden Ring. If I had to guess, I’d say that the decision to tone down the initial “dark fantasy” elements of Tears of the Kingdom was based on practical issues related to marketing. Instead of trying to beat FromSoft and Squaresoft at their own game, I think the Zelda devs wanted to appeal to the core audience of the series, and they considered that core audience to be professional gaming journalists obsessed with unique gameplay mechanics and creative influencers on social media who ship Zelda/Link.
Accordingly, I think the bulk of the artbook accurately reflects the primary concerns of the developers. There’s page after page of visual references for Link’s outfits and weapons, as well as sparsely annotated concept art for the temples and shrines. I’m still working my way through the 100+ pages of text at the end of the book, but most of what I’ve encountered so far is a basic summary of what we see in the game. In terms of the more “shocking” revelations, such as Sonia and Rauru having children, I can understand why these elements were cut for the sake of streamlining an already bloated product.
At the same time, I agree with your assessment that Nintendo seems to be struggling to understand what its fans want, and I think this is part of the reason why Tears of the Kingdom was awarded a slew of near-perfect reviews from gaming websites but then more or less forgotten by the broader culture.