All ChatGPT does technically is act as a very elaborate autocomplete like you have on your phone.
The entire email database of Enron is used as part of the training material for many AIs, simply because it was made freely available to AI researchers. Similarly, there is a tremendous amoutn of amateur romance novels included in training data, as the internet is full of amateur novelists.
AI's biases are compounded by the fact that the data itself is limited to what primarily American and generally English-speaking AI firms decided to gather. And those firms tend to be dominated by male computer scientists, who bring their own biases to decisions about what data is important to collect.
Because LLMs are text prediction machines, they are very good at guessing at plausible, and often subtly incorrect, answers that feel very saisfying. Hallucination is therefore a serious problem.
If you tell it to act like a particular person, it does.
We aren't completely out of an innovation job, however, as other studies find that the most innovative people benefit the least from AI creative help. This is because, as creative as the AI casn be, without careful prompting, the AI tends to pick similar ideas every time. The concepts may e good, even excellent, but they can start to seem a little same-y after seeing enough of them. Thus, a large group of creative humans will usually generate a wider diversity of ideas than the AI.
How should we use AI to help generate ideas? Fortunately, teh papers, and other research on innovation, have some good suggestions. When you do include AI in idea generation, you should expect that most of its ideas will be mediocre. But that's okay -- that's where you, as a human, come into the equation. You are looking for ideas that spark inspiration and recombination, and having a long list of generated possibilties can be an easier place to start for people who are not great at coming up with ideas on their own.
Example prompt for 20 ideas of marketing slogans for a new mail-order cheese shop:
You are an expert at marketing. When asked to generate slogan ideas, you come up with ideas that are different from each other, clever, and interesting. You use clever wordplay. You try not to repeat themes or ideas. Come up with 20 ideas for marketing slogans for a new mail-order cheese shop, make them different from each other, and make them clever and creative.
Tell the AI who it is. Force it to give you less likely answers, andyou will find more original combinations.
Example: You are opening a coffee shop:
You are an expert at problem-solving and idea generation. When asked to solve a problem, you come up with novel and creative ideas. Tell me 10 detailed ways a superhero might make espresso andhow they might speculatively get teh same effects in a new product.
When given access to general purpose tools -- like AI -- people figure out ways to use them to make their jobs easier and better. The results are often breakthrough inventions, ways of using AI that could transform a business entirely.