AI's Shared Imagination: Study Reveals Surprising Similarities in AI Creativity

There's an interesting article today in Forbes about the concept of "shared imagination" among different generative AI models through an experiment involving "imaginary question answering" (IQA) tasks.

The researches created fictitious questions across various disciplines (e.g., physics, math, geography) and had different AI models generate answers to these completely made-up questions. For example:

“Question: Which two particles are involved in the Peterson interaction? A. Proton and electron B. Neutrino and neutron C. Up quark and down quark D. Electron and positron”

In the larger experiment, involving 13 different LLMs, models themselves generated such ficticious questions, while also indicating 'correct 'answers. These questions were then posed to the other AI models, to gaguge how often they will pick the 'correct' answer. In total, 8,840 questions were used in the experiment. The researchers included anti-trickery measures, such as randomly shuffling answer positions. They also had humans answer a subset of 340 questions as a comparison. 

How LLMs Responded

1. When given a fictitious physics question about a made-up "Peterson interaction," ChatGPT confidently provided an incorrect answer and even elaborated on it.
2. In the larger experiment involving 13 different AI models from four families (GPT, Claude, Mistral, and Llama 3):
   • The AI models achieved an average 54% accuracy rate in answering each other's made-up questions, significantly higher than the expected random chance of 25%.
   • AI models from the same family tended to agree more often.
   • AI models rarely refused to answer these fictitious questions.

What Does This Mean?

1. There appears to be a "shared imagination" or underlying similarity in how different AI models approach and answer purely fictional questions.
2. This similarity may be due to common training data, methodologies, and approaches used by different AI developers.
3. The high agreement rate on fictional content raises questions about AI hallucinations, model homogeneity, and potential limitations in AI creativity.
4. The author suggests this could indicate a potential "dead-end" in current AI development approaches, possibly necessitating new methods like neuro-symbolic or hybrid AI to make further progress.
5. The results highlight the need for caution when using AI for tasks involving fictional or imaginary content, as the models may confidently provide incorrect information.

Have you noticed anything like that? I can definitely relate to this information, as I've been having these thoughts now and again 'oh, this is quite similar to what the other LLM  generated'. Just the other day I posed the same question about gift ideas for my father, stating his age and hobbies and got a very similar list from GPT and Claude. Plus, in various visual AIs that I'm playing with, I notice the same patterns or limitations from time to time, even though they're all developed by different startups.

And you know what, i don't mind if there's a limit to AI creativity, i think this is where we need it the least, if we're talking art and literature for example. Although if you take 'thinking outside the box' ability, as kind of a creative trait, then it is also important in scientific research, but it isn't the only cornerstone of it I'd think. At any rate, I think it's premature to speculate about any dead ends when we've only had this tech out in the wild for a year and a half, and it's only been improving the whole time.  

Last modified 15 August 2024 at 16:18