MMBench

Although frequently mentioned on social media and Zhihu, existing discussions are mostly introductory articles and retweets; real-world user evaluation feedback is limited. However, the detailed official tutorial (e.g., from OpenMMLab) explaining the CircularEval strategy provides solid methodological backing, indicating MMBench has been adopted as an important benchmark in the multimodal community. MMBench provides a carefully curated dataset of thousands of multiple-choice questions covering fine-grained abilities such as perception and reasoning. It introduces the CircularEval strategy, using ChatGPT to convert free-form model outputs into predefined choices, greatly improving evaluation stability for open-source models that struggle with direct option output. This yields more robust and reproducible comparisons across models. The dataset and evaluation code are publicly available on GitHub (official source), and can be used free of charge with no API fees. Users need basic Python programming skills to integrate their model outputs; the benchmark itself does not offer model hosting or compute resources. Running costs depend on self-hosted inference environments.