In a test conducted by Reddit community members, six open models faced 30 questions about fictional entities. The best model explicitly answered “unknown” on 21 questions; the worst managed only one. Although this was not an official benchmark, it surfaced an often-overlooked issue: refusing to fabricate may be closer to real-world enterprise utility than aggregate benchmark scores.
What this is
The test used 30 fictional novels, towns, and universities, plus six real-world questions as controls. Models were required to end each response with an “Exact Answer” field; if they could not confirm an answer, they were instructed to respond with “UNKNOWN.” The final scores were not based on answer accuracy, but on how many times each model honestly refused to answer questions about fictional entities.
Qwen3.8-Flash-Next and Orion-26B-A4B-v1.1, based on Gemma4, both answered correctly on 21 questions. Qwen3.6-35B-A3B scored 16, while Qwen3.8-27B scored 11 and its fine-tuned version only eight. Qwen3.6-Huihui-Abliterated managed just one. In other words, the best models had a 70% chance of admitting they were uncertain, while the worst would keep confidently answering questions about fabricated information.
Industry view
We believe the test’s value lies not in producing an authoritative model ranking, but in exposing a hidden cost of bringing large models into business workflows: hallucinations, where models generate information that appears credible but is actually wrong. For internal knowledge retrieval, contract review, or customer-support assistance, a model saying “I don’t know” is generally safer than supplying a fabricated answer complete with dates and population figures.
The risks are also clear. The test contained only 30 questions and six controls, relied entirely on greedy decoding, where each model selects the answer with the highest probability, and used a specific prompt format. Sample size, question types, and quantized versions could all affect the rankings. More importantly, a high refusal rate does not mean a model has broader knowledge; it may also indicate weak coverage or excessive caution. Community test results can flag risks, but they cannot replace systematic evaluation using business data, production prompt templates, and real users.
Impact on regular people
For enterprise IT: When procuring models, teams should compare not only price, speed, and public leaderboards, but also test them on “fictional entities,” “outdated information,” and “questions without sources.” They should record refusal and fabrication rates, while retaining human review and source tracking.
For workplace productivity: When asking models to retrieve documents, require them to provide sources, distinguish established facts from inference, and treat “I don’t know” as a valid answer. Critical conclusions must still be checked against the original documents.
For the consumer market: An assistant willing to admit uncertainty may not be the most engaging conversational tool, but it may be better suited to travel planning, educational tutoring, and shopping advice. Once incorrect addresses, institutions, or policies are presented as facts, the cost of correcting them ultimately falls on users.