Microsoft White Paper Evidence: Replacing Harness, the performance of Claude is significantly worse

Article | Xiaojing

Editor | Xu Qingyang

On November 7, Microsoft released a technical white paper on the Agent execution system, which included a somewhat surprising set of test results: using Claude Opus 4.8, simply replacing the Harness behind Copilot Studio increased the success rate of complex tasks by 22.1 percentage points. The new system nearly doubled the number of model calls per task, yet the adjusted input Token consumption could be reduced by 21%.

Just ten days earlier, Microsoft had announced the promotion of the unified Copilot Harness across multiple products and began selling hosting services to developers through its cloud platform.

On November 5, Moon's Dark Side just launched Kimi Hosted Agents, officially proposing Harness as a Service. In August, DeepSeek open-sourced its own Harness; OpenAI and Anthropic had already turned their internal operating systems into services that developers could call upon.

The "operating system" behind that large model has begun to become a key player in the eyes of model companies.


Image generated by AI

### 01 Seizing the Harness

Harness can be understood as a set of Agent operating systems built around large models. It enables models to use external tools, continuously execute multi-step tasks, and adjust subsequent actions based on execution results. For Agents that need to run for extended periods, Harness is also responsible for saving task states, allowing work to continue after interruptions.

It operates at a different level than AI applications. For example, products like Claude Code and WorkBuddy are directly aimed at users, while the underlying Harness can be reused by different applications and can also be offered as an independent service to developers. As the work undertaken by Agents becomes increasingly complex, Harness has gradually become an important factor influencing task success rates, execution efficiency, and operational costs.

Anthropic is one of the earlier model companies to invest in Harness development, initially to address issues Claude Code encountered during long programming tasks.

By the end of 2025, Anthropic discovered that when Claude's context window was exhausted, new rounds of conversation often could not continue previous work. Engineers then improved the Harness behind Claude Code, allowing the Agent to save progress before the end of each stage, so that the next round could read these records and continue unfinished tasks. This way, even if a model session ends, the entire programming task can still progress.

In April of this year, Anthropic further launched Managed Agents, providing this Agent operating capability as a cloud service to developers. To enable enterprises to run Agents long-term and stably, Anthropic also adjusted the underlying architecture: Harness is responsible for driving task execution, historical records can be saved separately, and the AI runtime environment's sandbox can be replaced at any time. Even if the sandbox fails, the Agent can recover previous work.

From the internal operating system of Claude Code to the hosted service for external developers, Anthropic has turned years of accumulated harness capabilities into an independent product.

OpenAI also chose to open the Harness behind Codex to more developers. In February of this year, OpenAI disclosed that the web version, command line, and IDE plugin of Codex actually share the same set of Agent operating systems. In August, it further positioned Codex as an open Harness platform, allowing third-party applications to connect directly. The Agents API launched in September is hosted and maintained by OpenAI, allowing developers to directly access the Agent's operating environment.

This route allows OpenAI to serve users directly through Codex while also providing underlying execution capabilities to other software companies. Developers can retain their product interfaces and business processes while Codex completes specific tasks in the background.

DeepSeek open-sourced the entire framework code of Harness under the MIT license on August 13, proposing "Everything is a Plugin," almost breaking down the entire Agent operating system into freely combinable plugins. Developers can not only replace models and external tools but also modify how Agents call models and execute tasks, even allowing Agents to write and install new plugins themselves.

This Cordis-based architecture aims to enable developers to develop different types of Agents according to their needs without having to build an operating system from scratch.

On November 5, Kimi officially announced that Hosted Agents provide 24/7 cloud environments, allowing enterprises to integrate Agents into internal systems or embed them into products sold externally, clearly proposing Harness as a service.

### 02 Evolution of Harness

In fact, the popularity of Harness did not start with model companies.

Earlier this year, the rise of OpenClaw has shown the industry that a company without a foundational model can also attract a large number of users through an Agent operating system. In February, OpenAI recruited OpenClaw founder Peter Steinberger to participate in the development of the next generation of personal Agents.

Subsequently, Nous Research launched Hermes Agent, attempting to allow Agents to save accumulated experience from work as reusable Skills and continuously improve in subsequent tasks.

At the end of March, institutions like Stanford University proposed Meta-Harness, allowing AI to directly modify Harness code based on failure records and verify effectiveness through testing. In a text classification experiment, accuracy improved by 7.7%, and context Token consumption dropped to a quarter of the original.

Why are these explorations worth the investment from model companies? In October, Nanyang Technological University in Singapore tested 66 models combined with Harness and found that changing Harness could even alter the performance rankings between models.

FrontierHarness Eval also found that for the same Kimi K3, the task success rate using Codex was 66.7%, using DeepSeek Harness Creator was 63.3%, and using Kimi Code was 56.7%.

For foundational model companies, improving model performance increasingly relies on the design and optimization of Harness, in addition to improving the models themselves.


Researchers are also trying to use the experience accumulated by Harness for model training. Co-Harness, proposed in July, first improves Harness through failure records and then uses the optimized execution data to train models. HarnessFlow, announced in November, attempts to automatically search for better Harness designs, improving average scores by 10.89 percentage points across six types of tasks.


However, automatic optimization also carries risks. A study at the beginning of November found that in specific experiments, a Harness optimized for new tasks saw its previous score on SWE-bench drop from 94% to 32%. Model upgrades may also render previously effective designs ineffective.

This means that the technological accumulation of Harness is difficult to achieve once and for all. How to maintain stability across different tasks while continuously improving remains an unresolved challenge.

Industry insiders have pointed out that this is also a major pain point for deploying open-source models. Therefore, model companies have begun to sell Harness services. Open-source frameworks can be obtained for free, but enterprises that want to run Agents stably and long-term still need to bear the costs of deployment and maintenance. Companies like Kimi and Anthropic see opportunities in taking on these tasks, turning Harness into an independent business, but the approaches of different companies vary greatly.

### 03 The Hidden Value of Harness

Harness is beginning to become a service that can be charged separately, but the current pricing table shows that model inference remains the primary source of revenue.

Anthropic's Managed Agents charge $0.08 per active session hour, with Tokens billed separately. According to an official example of a one-hour programming task, the total cost is $0.58, of which the hosting fee accounts for only about 14%.

Kimi's charges are divided into model Tokens, sandbox operation, and paid plugins. It estimated the cost of a legal analysis task for an investment bank to be about 30 yuan, with sandbox fees being less than 0.1 yuan. OpenAI waived additional service fees during the public testing period of the Agents API, primarily charging based on model and tool usage. DeepSeek open-sourced Harness under the MIT license and does not have a direct charging logic.


Thus, the direct revenue that model companies currently obtain from Harness may be minimal.

So why invest a large amount of R&D resources?

One important reason is that once enterprises truly deploy Agents, their demand for Harness may be more stable than for a specific model.

ABC Legal, a U.S. legal services company, originally deployed Agents on employee computers, running them through scheduled tasks. As the usage expanded, CTO Brandon Fuller began searching for a platform that could manage these Agents uniformly. He compared several solutions and ultimately chose Anthropic's Managed Agents because the company did not want to continue maintaining a decentralized operating environment.

According to Anthropic, ABC Legal put over 50 Agents into production within a month, with about 310 employees regularly using Claude. Employees do not even need to have software development knowledge to create new Agents through configuration.

This indicates that the appeal of hosted Harness comes from actual enterprise demand. Compared to developing, deploying, and maintaining an operating system from scratch, enterprises may prefer to purchase ready-made services and allocate engineering resources to their own business.

However, enterprises may not necessarily want to bind themselves to a single model company.

In August of this year, VentureBeat published a survey of 107 companies. 85% of the surveyed companies use two or more agent orchestration platforms simultaneously. When choosing a platform, 29% of companies prioritize the flexibility of models and tools, while only 10% prioritize native compatibility with a leading model. Although the sample is biased towards large tech companies, it reflects a purchasing tendency: customers want to retain the freedom to switch models.

This precisely provides another business logic for Harness's competition. If companies are willing to use a single operating system in the long term, even if providers cannot monopolize model calls, they may continue to earn revenue from hosting, maintenance, and enterprise development services, establishing a more stable partnership with customers.

In this way, Harness can access the execution process of actual work in enterprises. Compared to a simple model API, Harness can record the work trajectory of a task from start to finish, including model calls, tool executions, failure retries, and final results. These records can help companies troubleshoot issues and provide valuable feedback for model development.


If it can continuously obtain verified real task feedback, model companies will have the opportunity to discover issues with models in production environments more quickly. However, this advantage will also be clearly limited by data rights.

This also raises a new question for model companies: If enterprises can freely switch underlying models while insisting on controlling their own data, how much long-term advantage can Harness establish?

For model companies, Harness may bring more long-term value than direct charging. As companies delegate more and more work to agents, those who can provide a reliable operating system will have the opportunity to influence model selection and how AI participates in real business.

A familiar yet clichéd term: entry point.

However, this term, deeply rooted in the internet era, will always tug at the sensitive nerves of competition in the AI era. Now, Harness, to some extent, bears this expectation and imagination.