IMHO two things can be true at once.
"Brown" doesn't mean brown people in LOTR-ese the way we use it to refer to non-black POC, it means tanâlike, if the Chinese, a reputedly pale ethnicity, can call a light medium skin tone black while still referring to fellow indigenous Chinese, English authors from the 40s can surely call light medium skin tones brown without referring to nonwhite Brits.
But also there's no good reason for somebody to flap his gums about "politically correct" blah blah blah. That's absolutely a dog whistle. He shouldn't have thought that way and he definitely shouldn't have said it.
Another thing entangled in all this is there's a way to do normalize diversity with casting without falling into the tokenizing trap that corporations have developed a taste for, and without pretending that all logic goes out the window in any world where dragons exist. The Shire hobbits are very set in their ways and suspicious if polite to foreigners so it's implausible on many levels for them to be multiracial, but hobbits of color in bustling Bree is quite another story. Meanwhile a certain author recast a certain character as black in what might be inferred as a desperate attempt to reclaim her social justice credentials only for it to backfire by injecting new, problematic subtexts to the relationship dynamics of the characters. We do need to be more progressive in casting but we need to do so thoughtfully.