Vision models fail to top 60% on Moonshot benchmark
A new evaluation workflow for Moonshot's PerceptionBench benchmark reveals that even leading vision-language models fail to cross a 60 percent accuracy threshold on fine-grained visual tasks.
An end-to-end evaluation workflow has been designed for Moonshot's PerceptionBench, a multimodal benchmark that assesses fine-grained visual perception across ten atomic capabilities. The benchmark contains 3,000 total rows, with 40 percent newly authored and 60 percent decomposed from existing datasets, representing a 1.63 GB download. The evaluation framework supports local models like HuggingFaceTB/SmolVLM2-2.2B-Instruct and API-based models such as gpt-4o-mini. It uses a robust multi-stage streaming strategy, base64 image decoding, and automated judging to test how well models handle tasks like OCR, counting, depth understanding, and hallucination detection.
The benchmark's leaderboard highlights a stark performance ceiling for modern vision-language models, as no frontier model has managed to exceed 60 percent overall accuracy. GPT-5.6-Sol leads the pack at 59.7 percent, followed closely by Kimi K3 at 58.5 percent, Claude-Fable-5 at 57.2 percent, Gemini-3.1-Pro at 56.2 percent, and Seed-2.1-Pro at 55.0 percent. Larger open models lag further behind, with Qwen3.5-397B-A17B scoring 47.5 percent, Gemma-4-31B at 40.7 percent, and GLM-4.6V at 32.5 percent. Across almost all evaluated models, perception hallucination, or Hallu, remains the weakest capability.
For AI practitioners, this structured workflow provides a highly reliable method to diagnose the visual limitations of their models. The framework implements both rule-based grading and an LLM-assisted judging protocol modeled after the original paper's use of a GPT-oss-120B judge, which demonstrated a 99.7 percent agreement rate with human evaluators during a 300-sample audit. By calculating bootstrap confidence intervals and analyzing performance across specific difficulty slices, developers can move beyond simple accuracy averages. This allows practitioners to run targeted ablations, such as comparing crop-to-region techniques against full images or testing how different image resolutions affect fine-grained recognition.
This is our own summary of reporting by MarkTechPost
