---
title: "Are the New GPT-OSS Models Any Good? We put them to the test."
description: "We ran OpenAI’s new GPT-OSS 20B & 120B through our open-source optimizer. See how they stack up on speed, cost, and accuracy."
url: "https://www.datarobot.com/blog/testing-gpt-oss-models/"
authors:
  - "Stefan Hackmann, Staff Machine Learning Researcher, DataRobot"
  - "Alex Conway, Principal Software Engineer, DataRobot"
  - "Dr. Debadeepta Dey, Distinguished Researcher"
  - "Mark Steadman, Data Science Engineering Architect"
  - "Matthew Hausknecht, Principal AI Researcher"
published: "2025-09-16"
updated: "2025-09-13"
category: "AI for Practitioners"
language: "en-US"
---

# Are the New GPT-OSS Models Any Good? We put them to the test.

OpenAI hasn’t released an open-weight language model since GPT-2 back in 2019. Six years later, they surprised everyone with two: **gpt-oss-120b** and the smaller **gpt-oss-20b**.

Naturally, we wanted to know — how do they actually perform?

To find out, we ran both models through our open-source workflow optimization framework, [syftr](https://www.datarobot.com/product/syftr/). It evaluates models across different configurations — fast vs. cheap, high vs. low accuracy — and includes support for OpenAI’s new **“thinking effort”** setting.

In theory, more thinking should mean better answers. In practice? Not always.

We also use syftr to explore questions like “[is LLM-as-a-Judge actually working?](https://www.datarobot.com/blog/llm-judges/)” and “[what workflows perform well across many datasets?”](https://www.datarobot.com/blog/silver-bullet-agentic-ai-workflows/).

Our first results with GPT-OSS might surprise you: **the best performer wasn’t the biggest model or the deepest thinker.**

Instead, the 20b model with *low thinking effort* consistently landed on the [Pareto frontier,](https://www.datarobot.com/blog/pareto-optimized-ai-workflows-syftr/) even rivaling the 120b medium configuration on benchmarks like FinanceBench, HotpotQA, and MultihopRAG. Meanwhile, high thinking effort rarely mattered at all.

## How we set up our experiments

We didn’t just pit GPT-OSS against itself. Instead, we wanted to see how it stacked up against other strong open-weight models. So we compared **gpt-oss-20b** and **gpt-oss-120b** with:

- qwen3-235b-a22b
- glm-4.5-air
- nemotron-super-49b
- qwen3-30b-a3b
- gemma3-27b-it
- phi-4-multimodal-instruct

To test OpenAI’s new “thinking effort” feature, we ran each GPT-OSS model in three modes: low, medium, and high thinking effort. That gave us six configurations in total:

- gpt-oss-120b-low / -medium / -high
- gpt-oss-20b-low / -medium / -high

For evaluation, we cast a wide net: five RAG and agent modes, 16 embedding models, and a range of flow configuration options. To judge model responses, we used GPT-4o-mini and compared answers against known ground truth.

Finally, we tested across four datasets:

- FinanceBench (financial reasoning)
- HotpotQA (multi-hop QA)
- MultihopRAG (retrieval-augmented reasoning)
- PhantomWiki (synthetic Q&A pairs)

We optimized workflows twice: once for accuracy + latency, and once for accuracy + cost—capturing the tradeoffs that matter most in real-world deployments.

## Optimizing for latency, cost, and accuracy

When we optimized the GPT-OSS models, we looked at two tradeoffs: **accuracy vs. latency** and **accuracy vs. cost**. The results were more surprising than we expected:

- **GPT-OSS 20b (low thinking effort):
    Fast, inexpensive, and consistently accurate. This setup appeared on the Pareto frontier repeatedly, making it the best *default choice* for most non-scientific tasks. In practice, that means quicker responses and lower bills compared to higher thinking efforts.
- **GPT-OSS 120b (medium thinking effort):
    Best suited for tasks that demand deeper reasoning, like financial benchmarks. Use this when accuracy on complex problems matters more than cost.
- **GPT-OSS 120b (high thinking effort):
    Expensive and usually unnecessary. Keep it in your back pocket for edge cases where other models fall short. For our benchmarks, it didn’t add value.

![Figure 01 latency](https://www.datarobot.com/wp-content/uploads/2025/09/Figure_01_latency-1024x509.png)

*Figure 1: Accuracy-latency optimization with syftr*

![Figure 02 cost](https://www.datarobot.com/wp-content/uploads/2025/09/Figure_02_cost-1024x520.png)

*Figure 2: Accuracy-cost optimization with syftr*

## Reading the results more carefully

At first glance, the results look straightforward. But there’s an important nuance: an LLM’s top accuracy score depends not just on the model itself, but on how the optimizer weighs it against other models in the mix. To illustrate, let’s look at FinanceBench.

When optimizing for latency, all GPT-OSS models (except high thinking effort) landed with similar Pareto-frontiers. In this case, the optimizer had little reason to concentrate on the 20b low thinking configuration—its top accuracy was only 51%.

![Figure 03 latency financebench](https://www.datarobot.com/wp-content/uploads/2025/09/Figure_03_latency_financebench-1024x770.png)

*Figure 3: Per-LLM Pareto-frontiers for latency optimization on FinanceBench*

When optimizing for **cost**, the picture shifts dramatically. The same 20b low thinking configuration jumps to 57% accuracy, while the 120b medium configuration actually drops 22%. Why? Because the 20b model is far cheaper, so the optimizer shifts more weight toward it.

![Figure 04 cost financebench](https://www.datarobot.com/wp-content/uploads/2025/09/Figure_04_cost_financebench-1024x768.png)

*Figure 4: Per-LLM Pareto-frontiers for cost optimization on FinanceBench*

**The takeaway:** Performance depends on context. Optimizers will favor different models depending on whether you’re prioritizing speed, cost, or accuracy. And given the huge search space of possible configurations, there may be even better setups beyond the ones we tested.

## Finding agentic workflows that work well in your setup

The new GPT-OSS models performed strongly in our tests — especially the 20b with low thinking effort, which often outpaced more expensive competitors. The bigger lesson? More model and more effort doesn’t always mean more accuracy. Sometimes, paying more just gets you less.

This is exactly why we built syftr and made it open-source. Every use case is different, and the best workflow for you depends on the tradeoffs you care about most. Want lower costs? Faster responses? Maximum accuracy?

[Run your own experiments](https://github.com/datarobot/syftr) and find the Pareto sweet spot that balances those priorities for your setup.