I believe this is the case being talked about by Silas P Silas:
As in almost all cases, this isn't a case of a computer DELIBERATELY being made to discriminate, it's a case of a computer RECKLESSLY being made to discriminate. Which isn't even a tiny bit better, but is an important distinction for when we talk about the dangers of this kind of thing.
AI is trained off of available data. Available data is about the real world, where racism and sexism and all that shit exists. Unless you carefully put effort into making sure that bigotry is EXCLUDED from the training, it will by default be included.
We've known this forever. There is no excuse. They don't bother, or they do a shitty job at it, because they want to make more money and carefully curating your own custom dataset without racism in it is not only extremely difficult, but very expensive.
