The Problem with Traditional AI Benchmarks

If you’ve been keeping up with the latest AI model releases, you’ve probably noticed a trend: every new model is breaking records on some benchmark leaderboard. Gemini is #1 on Chatbot Arena, OpenAI’s o3 scored 25% on the Frontier Test, and DeepSeek is dominating the MMLU benchmark. But let’s be real—what do these scores actually tell us about an AI model’s real-world value?

Not much.

LLMs today are fine-tuned to game these benchmarks. Labs are mining prompts and training models to score high rather than to perform well in real-world tasks. In other words, the AI arms race has turned into an academic exercise where models are optimized to look good on paper rather than drive business impact.

This is where Hao AI Lab’s latest research comes in—and why benchmarking AI with video games like Super Mario might actually be the key to solving real-world problems.

Why Super Mario is the Perfect AI Testbed

Hao AI Lab’s latest research project, GameArena: Evaluating LLM Reasoning through Live Computer Games, is taking a different approach. Instead of scoring models on static benchmarks, they’re testing how AI agents can reason, adapt, and make decisions in live gaming environments.

Think about it:

• Super Mario requires strategic thinking, real-time decision-making, and adaptability.

• There’s no static dataset—every playthrough is different.

• An AI agent has to process visual inputs, predict obstacles, and act accordingly.

• It’s closer to how AI will have to operate in real-world applications like robotics, autonomous navigation, and workflow automation.

Hao AI Lab pitted top models—Claude-3.7 and GPT-4o—against Super Mario, with surprising results:

• Claude-3.7 outperformed GPT-4o, relying on simple heuristics to navigate the game efficiently.

• GPT-4o, despite being more advanced, struggled—likely because its architecture prioritizes language rather than real-time action.

What This Means for AI & Business

Hao AI Lab’s work proves a critical point: AI that succeeds in controlled benchmarks doesn’t necessarily translate to useful AI.

For businesses looking to leverage AI, the key question isn’t “Does this model rank #1 on Chatbot Arena?”—it’s “Can this model actually solve my business problem?”

Benchmarks should measure:

• How well AI improves efficiency in workflows.

• Whether it makes smarter decisions that lead to better outcomes.

• The actual economic impact of deploying the model.

Right now, the AI industry is too focused on leaderboard supremacy. But what businesses need is applied AI that drives measurable results.

The Future: AI Benchmarks that Matter

Hao AI Lab’s research is an important step toward rethinking how we evaluate AI models. Instead of measuring abstract performance on static tests, AI should be tested in real-world, dynamic environments—whether that’s gaming, financial modeling, or customer service automation.

At PixelPac, we believe AI’s value lies in execution, not theoretical supremacy. Our approach is to help businesses navigate the AI landscape by focusing on solutions that actually move the needle—not just models that score well on a test.

What’s your take? Should AI benchmarks prioritize business impact over leaderboard dominance? Let’s discuss.