MT-Bench: Multi-Turn Conversation Evaluation
How MT-Bench evaluates model quality on multi-turn conversations using an LLM judge, what the 10-point scale measures, and how it complements other benchmarks.
Multi-Turn, LLM-as-Judge
MT-Bench (Multi-Turn Benchmark) was introduced by the LMSYS team to evaluate models on realistic multi-turn conversations. It consists of 80 challenging, open-ended questions organized into 8 categories: writing, roleplay, reasoning, math, coding, extraction, STEM, and humanities. Each question includes a first turn and a follow-up, testing how well models maintain context and coherence. MT-Bench rewards In-Context Learning, the model must use information established in turn 1 when answering turn 2, making it a practical test of how well a Foundation Model generalises from conversational context.
Unlike multiple-choice benchmarks, MT-Bench uses GPT-4 as an automated judge to score responses on a 1-10 scale. This LLM-as-judge approach evaluates qualities like helpfulness, accuracy, depth, and appropriate tone that automated metrics miss. The judge is instructed to assess each response independently before comparing, reducing order effects.
Interpreting MT-Bench Scores
MT-Bench scores range from 1-10. Early models like Llama 2 Chat 70B scored around 6.3; GPT-3.5 Turbo scored 7.9; GPT-4 scored 8.96 when the benchmark was introduced. Frontier models in 2025 reach 9.2-9.5 on most evaluations, the scale is increasingly compressed at the top as models improve.
Scores by category reveal capability profiles. A model scoring 9.5 overall might score 9.8 on writing but 8.9 on math, these differences help match models to use cases. Reasoning and math categories most strongly differentiate frontier models from mid-tier ones. Writing and roleplay categories tend to be more uniformly strong across quality models.
Limitations
MT-Bench has limitations stemming from its LLM judge approach. GPT-4 may have biases: favoring longer responses, preferring its own style, or rating highly on dimensions that correlate with its own training (e.g., politeness, hedging). When GPT-4 is used to evaluate models that compete with GPT-4, the judge is not neutral.
Newer evaluations use multiple judges, include human validation, or use open-source models as judges to reduce potential bias. MT-Bench remains useful as a quick multi-dimensional snapshot but is best interpreted alongside automated benchmarks and Arena results for a full picture.
Read next
Understanding AI Benchmarks: How Models Are Evaluated
AI benchmarks are standardized tests used to compare LLM capabilities. Learn how they work, what they measure, and how to read them critically.
Chatbot Arena: The Crowdsourced LLM Leaderboard Explained
How LMSYS Chatbot Arena's human-preference voting works, what the Elo system measures, why it captures what automated benchmarks miss, and how to read the rankings for model selection.
MMLU: The Massive Multitask Language Understanding Benchmark
What MMLU measures, how it's constructed, why it became the standard LLM benchmark, what top model scores reveal, and when to use MMLU-Pro instead.