It's because humans are primates.
Back in the Cretaceous period, flower plants burst on the scene and flourished in dozens of lineages by cooperating with animals. Pollinating insects were first, but many plant lineages discovered that producing tasty fruit with hardy seeds hidden inside could be useful to disperse, as larger animals would eat the fruits and excrete the seeds elsewhere. This is why many fruits are brightly colored, sweet-smelling, and packed full of sugars. Flying animals were especially useful seed dispersers, for obvious reasons.
Most early land vertebrates were diurnal, and could see three main colors (red, green, and blue). When dinosaurs evolved, and then birds, they made good use of this ability, and even now sight remains by far the most important sense for birds (you know how they say if you touch a chick the mother will no longer rcognize it because it smells of human? False. Birds don't recognize each other by scent; that's mammal stuff.)
Mammals, however, were driven into comparatively marginal niches during the age of dinosaurs: underground, and by night. (Well, not all: there were large, diurnal mammals in the Mesozoic, even badger-like Repenomamus that actually fed on small dinosaurs -- but our own ancestors were more the tiny burrowing variety). In such conditions, sight is less useful. Early mammals sacrificed much of their brain capacity for sight processing in favor of olfaction, and even lost the receptor for red in their retina, reducing to a two-color vision. This is why most mammals communicate with scents, and this is why they cannot see red.
As mentioned, fruit plants were especially buddy with birds, who could spread their seeds far and wide. No wonder that many fruits turn red or yellow as they ripen, to become obvious for birds. They also started filling themselves with substances that were selectively poisonous to mammals -- for example the capsaicin of chili peppers, which directly stimulates mammal pain receptors but does nothing to birds.
Then some mammals became primates. They started foraging again in daytime (as most mammals would, later on), and picked up what excellent source of food symbiotic fruit could be. So they re-evolved that lost third color receptors, becoming some of the few mammals that see in three colors, so they could recognize ripening fruit like birds do. And they (we) evolved metabolic pathways to neutralize many of the poisons fruits and leaves defend themselves with, even some that were never meant to be eaten at all. Deadly alkaloids like caffeine and nicotine, or the theobromine of chocolate, the organosulfides of garlic and onions, and the capsaicin of chili peppers too -- deadly for most mammals, but not for primates. And later, of course, ethanol, the toxic waste of fermenting fungi.
Incidentally, at some point a bad mutation broke our gene to synthesize vitamin C, but our ancestors didn't notice, because they got all vitamin C they needed from fruit anyway. Most mammals can produce it by themselves, but we monkeys (and, in a completely separate development, guinea pigs) need to get it from food, which is why scurvy was a scourge of primate sailors for centuries.