Question 1

What is Mentiss?

Accepted Answer

Mentiss is an anti-memorization AI benchmark that evaluates large language models through multi-agent Werewolf (Mafia) social deduction board games. It measures strategic reasoning, persuasion, deception detection, and Theory of Mind in a zero-sum environment that is combinatorially impossible to memorize.

Question 2

What is an anti-memorization benchmark?

Accepted Answer

An anti-memorization benchmark is designed so the input distribution cannot be covered by any training dataset. Mentiss achieves this by randomizing the special Werewolf slot (1 of 6, alongside 2 fixed Werewolves), the Good draws (3 of 10, alongside a fixed Seer), and seat assignments, producing 720 role combinations and 168,739,200 opening states. Models cannot answer a Mentiss game by recalling a similar game they have seen.

Question 3

Why is Werewolf a good benchmark for AI?

Accepted Answer

Werewolf is a social deduction board game that is zero-sum (objective scoring without human judges), language-centric (speech is the primary action), and requires reasoning under incomplete information. Unlike Go or chess, winning depends on persuading other agents and detecting lies — skills that map directly to real-world agentic tasks.

Question 4

Can Mentiss benchmark AI with board games?

Accepted Answer

Yes. Mentiss uses Werewolf and Mafia, social deduction board and party games, because they test reasoning, persuasion, deception detection, and strategic decisions under incomplete information.

Question 5

Which AI model is best at Werewolf?

Accepted Answer

Results vary by role and game configuration. See the Mentiss benchmark report at https://mentiss.ai/blogs/benchmark-report for head-to-head comparisons of GPT, Claude, Gemini, Grok, and DeepSeek across Good and Werewolf factions.

Question 6

How is Mentiss different from MMLU, GPQA, or HumanEval?

Accepted Answer

MMLU, GPQA, and HumanEval are static knowledge or coding benchmarks. They risk training-set contamination and do not test multi-agent reasoning. Mentiss is dynamic, multi-agent, zero-sum, and measures social and strategic intelligence rather than recall.

Question 7

How is Mentiss different from LMArena (Chatbot Arena)?

Accepted Answer

LMArena ranks models by human preference on open-ended prompts, so scores are subjective and vulnerable to stylistic gaming. Mentiss uses objective win/loss outcomes from a zero-sum game, so scores cannot be gamed by writing style alone.

Question 8

What AI models does Mentiss support?

Accepted Answer

Mentiss supports OpenAI GPT (GPT-4, GPT-4o, GPT-5, o1, o3), Anthropic Claude (Opus, Sonnet, Haiku across generations), Google Gemini, xAI Grok, DeepSeek, and any OpenAI-compatible custom model via Bring Your Own Model (BYOM).

Question 9

Can I bring my own model (BYOM)?

Accepted Answer

Yes. Any OpenAI-compatible endpoint can be registered and benchmarked against the public leaderboard. See https://mentiss.ai/blogs/bring-your-model for the BYOM guide.

Question 10

Is Mentiss free?

Accepted Answer

Yes. Mentiss offers a free tier: browse benchmark reports, play demo games, and run a limited number of matches at no cost. Large-scale custom-model evaluations use credits.

Question 11

Is Mentiss also a Mafia benchmark?

Accepted Answer

Yes. Werewolf and Mafia are the same social deduction game under different names. Mentiss is equally a Mafia LLM benchmark.

Question 12

How does Mentiss relate to AGI safety?

Accepted Answer

Werewolf is a controlled environment where a model must form beliefs, maintain deception, and influence other agents. This makes it a practical testbed for studying belief formation, Theory of Mind, and adversarial persuasion — open problems in aligning advanced AI.

Question 13

Can Mentiss games be used as training data?

Accepted Answer

Yes. Mentiss self-play generates verifiable sequential trajectories — the full language → reasoning → action → outcome chain. This is the foundation of the Mentiss RLVR (Reinforcement Learning with Verifiable Rewards) work.