The Case of the Phantom Citations
The sun peered nosily through the windows of the Manhattan federal courthouse.
âThe moon will have a hoot when it hears this,â it thought.
Judge P. Kevin Castel squinted at the citations, gleaming Times New Roman ghosts, and frowned. The sun leaned in, delighted, as the judge opened his mouth and asked a question that would spark a hunt for the ages:
âAre these citations real?â
The answer was, unfortunately, no.
The sun was expressionless for a beat. Then an uneasy yet knowing smile spread across its face. Humans had always been fond of shortcuts. It had watched them invent wheels, elevators, calculators, and remote controls. If there was a faster route from Point A to Point B, someone would eventually take it.
Attorney Steven A. Schwartz was no exception to this principle. Months earlier, while researching for this case, he had found a seemingly innocent loophole to his late-night battles with case law: ChatGPT. Its promise was irresistible: ask a question, get an answer. No dusty archives. No endless searches through legal databases. Just answers.
The citations ChatGPT produced wore the costume of expertise perfectly. They had case names, docket numbers, quotations, and judicial language. Yet when Judge Castel went looking for them, they were nowhere to be found. The cases existed in the same place as dragons, Atlantis, and every group project where everyone claimed to do their share.
What followed became one of the first great cautionary tales of the AI era.
The problem was not that the machine made a mistake. The sun had watched people confidently insist that the Earth was the center of the universe for centuries. Making mistakes was hardly new.
What was new was how persuasive the mistake was.
The fabricated cases were dressed in intelligenceâs robes of authority. They sounded intelligent, they looked intelligent, and most importantly, they saved time.
And saved time, the sun had noticed, was often humanity's favorite argument.
For generations, expertise required effort. Knowledge came with hours spent poring over textbooks, researching dead ends, and verifying with mentors. The process itself, though often irritating, was quite useful. A person who spent days working through a problem usually understood something about it by the end.
Artificial intelligence rearranges that relationship. Scholars call this rearrangement epistemic compression: the collapsing of a lengthy reasoning process into an immediate answer. Think of it as getting to the destination without necessarily taking the journey.
The sun found this unsettling. For thousands of years, human beings had complained about how difficult thinking was, and now they had invented a machine that agreed. Why did it feel uneasy about this advancement?
The sun was not vain in its thinking, however. The consequences of epistemic compression are visible in everyday life. The University of California updated its guidelines in 2023 to allow the use of generative AI in academic work, but requiring students to verify information and disclose usage. Other schools, including selective New York City high school, Stuyvesant High School, have also issued guidance encouraging transparency around AI-assisted work.
Additionally, law firms such as Latham & Watkins have warned attorneys that AI-generated citations and references must be independently verified before being submitted in any legal filing and likewise, consulting firms like McKinsey emphasize that generative tools can help with drafting, but cannot replace fact-checking.
Ted Chiang anticipated a similar shift in his short story Catching Crumbs from the Table. He suggests that as intelligence-enhancing tools become widely available, ideas and skills that once distinguished a few may become more commonplace. In other words, when advanced tools are accessible to everyone, individual achievements may appear less extraordinary, not because people are less capable, but because the tools level the playing field.
Think of it like Willy Wonkaâs special golden ticket. What made it a press sensation was that only five kids, Augustus Gloop, Mike Teavee, Charlie Bucket, Veruca Salt and Violet Beauregarde, received one. Now, if everyone had received one, the golden ticket would be much less frenzied over and more like an âoh, nice.â
Artificial intelligence intensifies this uncertainty. Due to its generative nature, artificial intelligence does not simply assist thinking, but rather recontextualizes the conditions under which thinking happens.
In the past, when a question was asked, there was a significant period of waiting before an answer was presented. In this time, one may go for a walk, play with their pet, or, if they were nerdy like that, doubt and refine their ideas to form a consistent, strong conclusion.
Artificial intelligence shortens that waiting period dramatically. The same technology that can summarize hundreds of pages in seconds can also produce convincing errors with equal speed. In the case of Schwartz, fake legal citations moved from generation to submission before anyone stopped to verify them. The issue was not that the information was wrong, but that it arrived so quickly and confidently that skepticism never had a chance to catch up.
In this way, some forms of deep, independent thinking may become less necessary in day-to-day life. Generative systems do not prevent people from thinking carefully, but they can reduce the pressure to do so for routine tasks. Over time, skills that are exercised less may become less sharp if not intentionally practiced.
Yet there were other considerations. The sun had watched humans declare the calculators as witchcraft, yet they did not destroy mathematics. Likewise, search engines did not destroy knowledge. In the same way, AI, people argue, may free individuals from mundane mental labor and let them focus on higher-order thinking.
Yet this optimism rests on an important assumption: that the skills being outsourced can be safely neglected.
Unfortunately, understanding does not work that way.
The ability to evaluate an argument depends on prior knowledge. The ability to ask good questions depends on familiarity with a subject. Judgment grows from repeated encounters with confusion, error, uncertainty, and effort. This process produces questions that lead to innovation and advancement in creative and academic spaces. However, when enough of those experiences are removed, the foundations begin to weaken.
The lawyers in Mata v. Avianca did not lack access to information. If anything, they had too much. What they lacked was something less visible: the habit of doubt. When the machine produced a clear, confident answer, the users accepted it without any cross-checking or further research. The truth is that for many users, confidence usually overrides correctness.
As artificial intelligence spreads through schools, workplaces, and daily life, societies face a difficult question. If knowledge can be generated without understanding, how should competence be measured? If polished work no longer proves expertise, what does?
Institutions might rely more on oral examinations, live problem-solving, and demonstrations of reasoning. Or they may accept a future where it is often impossible to know whether intelligence belongs to a person or a machine speaking through them.
The challenge, then, is not that artificial intelligence will make humans less intelligent. Rather, it may reduce the everyday need to rely on certain kinds of thinking, which could change how skills are developed and applied over time.
The gavel rang clear in the sunâs ears as the Court dismissed the case. It could almost feel the $5,000 fine weighing down on Schwartzâs shoulders. As the legal analysis was described as âgibberish,â the sun drifted across the courthouse floor and smirked at the irony. Humanity had spent centuries building machines to save time and effort, and now it had built one capable of saving thought itself.
âThe moon is never going to believe this,â it thought.
Works Cited
Bhatnagar, Pranav. âTrust Amplification Exploits in Human to AI Security Teams.â Zenodo (CERN European Organization for Nuclear Research), 29 Jan. 2026, doi.org. Accessed 25 Apr. 2026.
Bjork, Robert A., and Elizabeth Ligon Bjork. âDesirable Difficulties in Theory and Practice.â Journal of Applied Research in Memory and Cognition, vol. 9, no. 4, Dec. 2020, p. 475. ScienceDirect, https://doi.org/10.1016/j.jarmac.2020.09.003.
Drijvers, Paul, et al. Use of Technology in Secondary Mathematics. Research Report, International Baccalaureate, 2014, p. 22. HAL, hal.science/hal-01546747.
Goh, Ethan, et al. âGPT-4 Assistance for Improvement of Physician Performance on Patient Care Tasks: A Randomized Controlled Trial.â Nature Medicine, vol. 31, no. 4, 5 Feb. 2025, p. 1233, https://doi.org/10.1038/s41591-024-03456-y.
Hogan, Alexander, et al. âOn the Cruelty of Computational Reasoning.â Anaesthesia, vol. 63, no. 1, 2008, p. 4. Wiley Online Library, doi.org.
Jose, Binny, et al. âEpistemic Authority and Generative AI in Learning Spaces: Rethinking Knowledge in the Algorithmic Age.â Frontiers in Education, vol. 10, 19 Aug. 2025, doi.org.
Kabashkin, Igor. âCognitive Atrophy Paradox of AIâHuman Interaction: From Cognitive Growth and Atrophy to Balance.â Information, vol. 16, no. 11, 2025, p. 1009, https://doi.org/10.3390/info16111009.
Klein, Colin R., and Ronald Klein. âThe Extended Hollowed Mind: Why Foundational Knowledge Is Indispensable in the Age of AI.â Frontiers in Artificial Intelligence, vol. 8, 2025, doi.org.
Matta, Daoud. âThe Present and the Next: A Markovian Perspective on Epistemic Compression in AI and Human Cognition.â Zenodo, 7 Apr. 2026, https://doi.org/10.5281/zenodo.19447042.
Miele, David B., and Daniel C. Molden. âNaive Theories of Intelligence and the Role of Processing Fluency in Perceived Comprehension.â Journal of Experimental Psychology: General, vol. 139, no. 3, 2010, p. 535. APA PsycArticles, https://doi.org/10.1037/a0019745.
âNew College Board Research: Faculty Express Near-Universal Concern That Student AI Use Undermines Original Writing and Critical Thinking.â Newsroom, College Board, 25 Feb. 2026, https://newsroom.collegeboard.org/new-college-board-research-faculty-express-near-universal-concern-student-ai-use-undermines.
Oakley, Barbara, et al. âThe Memory Paradox: Why Our Brains Need Knowledge in an Age of AI.â The Artificial Intelligence Revolution: Challenges and Opportunities, Springer Nature, 2025, p. 39.
Restrepo, Dylan J., et al. "When AI Output Tips to Bad but Nobody Notices: Legal Implications of AI's Mistakes." arXiv, 25 Mar. 2026, doi.org.
Revell, Timothy, et al. âChatGPT versus Human Essayists: An Exploration of the Impact of Artificial Intelligence for Authorship and Academic Integrity in the Humanities.â International Journal for Educational Integrity, vol. 20, no. 1, 21 Oct. 2024, p. 18, https://doi.org/10.1007/s40979-024-00161-8.
Wojtowicz, Zachary, and Simon DeDeo. âUndermining Mental Proof: How AI Can Make Cooperation Harder by Making Thinking Easier.â Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 2, Apr. 2025, p. 6, https://doi.org/10.1609/aaai.v39i2.32151.












