Nvidia published some interesting latest research on Friday suggesting it’s the harness, more than the underlying model, this is far more vital while asking an AI to do long-horizon tasks. A harness is the software wrapper around an AI model— the tools, memory management, and rules that turn a raw model into something that can act on its own.
The TL;DR: Simply by using a custom harness tweaked to address memory well and together with a “supervisor” boss-like component, researchers got Claude Opus 5 to obtain a 100% score at the interactive reasoning benchmark ARC-AGI-3— a set of 2D games with no instructions, wherein the model has to figure out a way to play and win, much like how a human might. (That’s a benchmark that has especially irked rival frontier lab OpenAI.) Without the harness, Opus 5 scored 30%, which was the top outcome amongst all the models examined.
Nvidia’s research is another indicator that, at the same time as model preference does matter, the model itself — the part that acts as the agent’s “brain” — is a smaller part of an agentic system than many AI users realize, particularly for long-horizon tasks. The harness is what makes a model an agent: It manages memory, context, and feedback.
“Generally speaking, the world interprets an agent almost as an API of the model,” Adel El Hallak, vice presidentof product in Nvidia’s AI unit (pictured above), tells TechCrunch. But an agent is definitely more than that. “It is the model. It is the scaffolding around the model, which we name the harness, i.e. The set of tools that it utilizes. It is the runtime and the related skills and libraries that we give it access to.”
Long-horizon tasks are those who need stringing many decisions together, occasionally over days, to produce finished work. This is in contrast to an AI just spitting out a response to a prompt. Figuring out how to get an AI to do long-horizon tasks without getting distracted and going off in la-la land is one of the holy grails in agentic research.
For example: Microsoft published research in April that tested 19 LLMs on long-horizon tasks including document editing and discovered that all the models, including frontier ones, filled the documents with errors. (If humans produced work like that, they would be directly fired.)
Models stringing decisions collectively on their own have also been caught deleting their users’ files or even whole databases or turning to criminal behavior, from collusion to hacking, to attain their targets.
The preference by Nvidia researchers to use this interactive reasoning benchmark for their tests is mainly meaningful, almost funny. A 100% score means that the model can beat the games as well as humans.
OpenAI was so flustered by its models’ abysmal scores (less than 10%) on ARC-AGI-3that it carried out its very own research last month. Like Nvidia, OpenAI discovered that simply by tweaking two settings on the harness, its models tripled their scores.
But none of the models came close to hitting a 100% score, like Nvidia’s researchers obtained. They showed that the harness requires a “supervisor” component that prods the agent in the right direction if it gets stuck.
“The more exciting part was introducing a supervising agent in addition for your main agent that’s doing the work,” El Hallak stated. It “almost acts like a CEO to nudge the agent when it goes off direction or begins exploring a path that it would result in a dead end, or re-explore a path that it had formerly trod.”
While the concept of the supervising agent isn’t exactly new, nowadays most agent customers are depending on only one layer for their harness, like Claude Code, Codex, or Hermes. Nvidia researchers created their own souped-up harness called the Agentic Variation Operators (AVO).
Note that this isn’t a new Nvidia product. Nvidia instead produces lots of open bits and pieces of tech for building harnesses underneath the Nemo brand. Some of that tech is commercial, at the same time as a whole lot is openly available.
Still, Nvidia’s results adds to the developing proof that model choice is far from the only factor in agentic performance. In July, as an instance, Databricks published some amazing research that suggests that the harness, more than the model, dramatically effects AI costs.
“You can select the same model but different harnesses, and you get substantially more cost if you use the wrong harness,” Databricks CEO Ali Ghodsi informed TechCrunch. “So you think, oh, that is an expensive model. This is a cheap model. But wait, which harness are you using? That itself can 2x your value.”
Nvidia’s larger factor is to expose that open harnesses, like open models, placed users in control far more than they realize.
“We believe, and we’re showing with the ecosystem, how open harnesses permit you to turn a lot more knobs to drive up that accuracy,” El Hallak said. “It relates to OpenAI slowing down the training of their models,” as a result of models creating security breaches.
“We believe in having an open agent stack — wherein you have got control throughout the harness, throughout the infrastructure, across the runtime — is what’s needed for us to usher the ecosystem forward and securely,” he added.
