AI Models Could Not Quote a Single Hadith Correctly in 30,000 Tries

Researchers handed five open AI models the exact reference for thousands of hadith and asked for the text. Not one answer matched both the wording and the meaning.

The facts

Researchers gave five small open-source AI models the exact reference for thousands of hadith (recorded sayings and actions of the Prophet Muhammad) from the six major Sunni collections and asked them to quote the text exactly. Out of 30,000 tries in English and Arabic, zero responses matched both the wording and the meaning of the real text. The models almost never said “I don’t know”; they confidently produced fluent but wrong material. Arabic results were worse than English, and Arabic-specialized models did no better.

Limit to keep in mind: only smaller open-weight models (roughly 4–8 billion parameters) were tested, with no retrieval help. Larger commercial models were not included, so real-world results with better tools could differ. The authors argue religious Q&A systems should pull from verified sources rather than rely on what a model “remembers.”

Why it matters biblically

Scripture can be checked word for word, and this study shows AI can invent text that looks authentic. The Bereans “searched the scriptures daily, whether those things were so” (Acts 17:11), and Proverbs 30:5–6 warns, “Add thou not unto his words.” Any verse an AI tool hands you should be checked against the text itself.

Full story

They Know Where It Is, But Not What It Says: A Bilingual Hallucination Benchmark of Hadith Recall in Open LLMs
Dayam Nadeem, Safdar Mustafa, Ameen Ur Rehman, Shahab Saquib Sohail, Mohammad Nadeem, Usman Naseem

Similar Posts