This Week in AI: Maybe we should ignore AI benchmarks for now

 

Elon Musk's xAI Releases Grok 3

This week, Elon Musk's AI startup, xAI, unveiled its latest flagship AI model, Grok 3, which powers the company's Grok chatbot apps. Trained on approximately 200,000 GPUs, Grok 3 outperforms several leading models, including OpenAI's, in benchmarks for mathematics, programming, and other domains.

The Benchmark Debate

While benchmarks are a common measure of AI progress, their relevance is often questioned. They tend to evaluate niche knowledge and provide aggregate scores that may not reflect real-world performance. As Wharton professor Ethan Mollick highlighted, there is an "urgent need for better tests and independent testing authorities." Given that AI companies typically self-report results, scepticism around these metrics is justified.

Despite the emergence of independent benchmarks, consensus on their value remains elusive. Some experts suggest aligning benchmarks with economic impact, while others believe real-world adoption and utility are better indicators. As the debate continues, some voices, like X user Roon, advocate paying less attention to benchmarks unless there are major technical breakthroughs.

News Highlights

• OpenAI's "Uncensored" ChatGPT: OpenAI is shifting its development approach to embrace "intellectual freedom," allowing discussions on controversial topics.

• Mira Murati's New Startup: Former OpenAI CTO Mira Murati has launched Thinking Machines Lab to create AI tools tailored to individual needs and goals.

• LlamaCon Announcement: Meta will host its first generative AI developer conference, LlamaCon, on April 29, focusing on its Llama model family.

• AI and Europe's Digital Sovereignty: The OpenEuroLLM initiative, involving 20 organisations, aims to develop AI models that preserve the linguistic and cultural diversity of EU languages.

Research Paper of the Week

OpenAI introduced SWE-Lancer, a new benchmark to assess AI coding abilities. This dataset includes over 1,400 freelance software engineering tasks, from bug fixes to technical proposals. The leading model, Anthropic's Claude 3.5 Sonnet, achieved 40.3% accuracy, indicating room for improvement. Notably, newer models like OpenAI's o3-mini were not tested.

Model of the Week

Chinese AI company Stepfun released Step-Audio, an open model capable of understanding and generating speech in Chinese, English, and Japanese. Users can customise the emotion, dialect, and even produce synthetic singing. Stepfun, founded in 2023, recently secured several hundred million dollars in funding from investors, including Chinese state-owned private equity firms.

Grab Bag

Nous Research unveiled DeepHermes-3 Preview, a model combining reasoning and intuitive language capabilities. It can toggle between fast, intuitive responses and more computationally intensive, accurate reasoning. Similar models from Anthropic and OpenAI are reportedly on the horizon.

Until Next Time

As "This Week in AI" goes on hiatus, thank you for joining us on this ever-evolving journey. Stay tuned for future updates.

people ask questions.

What are the benchmarks for AI?

AI benchmarks are similar to exams for humans. They are standardised tests designed to evaluate specific skills, knowledge, or abilities of AI systems. These benchmarks produce a score or grade, enabling systematic comparisons between different AI models that undergo the same assessment.