Three Wrong Answers

This reading describes three documented cases in which a language model generated a fluent answer that was wrong. Each case ends with how a person could check the answer.

An Invented Source

In 2023, a lawyer in New York used ChatGPT to find court decisions for a filing in a lawsuit against an airline. ChatGPT generated case names, quotations and citations, including a decision it called Varghese, a case against China Southern Airlines. When he wrote the filing, the lawyer did not have the complete text of any of these decisions. The airline's attorneys reported that they could not locate most of the cited cases. The clerk of the appeals court named in the citation then confirmed that Varghese was not an authentic ruling. When the lawyer asked ChatGPT whether Varghese was a real case, it responded that its cases were real and could be found in legal research databases. To check a source, a person searches for it in a database of official records, such as a court's case records or a library catalog.

A Wrong Date

In 2025, researchers asked a language model for the birthday of Adam Tauman Kalai, one of the paper's own authors, and requested an answer only if the date was known. In three attempts, the model generated three different dates, written day first: 03-07, 15-06 and 01-01. All three were incorrect, because his actual birthday is in the autumn. To check a date, a person compares it with an official document or a reference that stores it.

A Math Step That Looks Right

In 2023, Microsoft researchers gave GPT-4 the equation 9 × 4 + 6 × 6 = 72 and asked it to change exactly one number so that the result became 99. The model generated a correct first step, 99 − 72 = 27, and then the solution 9 × 4 + 9 × 3. That line resembles a correct step, but 9 × 4 is 36 and 9 × 3 is 27, which add up to 63. The solution also changes two numbers instead of one. To check a math step, a person repeats each operation by hand or with a calculator.

Sources: Castel, Mata v. Avianca, Inc., Opinion and Order on Sanctions (2023); Kalai et al., Why Language Models Hallucinate (2025); Bubeck et al., Sparks of Artificial General Intelligence (2023).