Author: Kyle Jeong (Growth Engineer at Browserbase)

Compiled by: Daniel

Edited by: Cage

This article is adapted from the analysis of Astra Computer Use capabilities by Kyle Jeong, Growth Engineer at Browserbase, originally published on his personal blog. For the past three years, their team has been working hard to apply the capabilities of AI Computer Use to real business.

The article breaks down how the Astra model works in conjunction with Codex Harness: Codex initiates a continuously running code environment, connecting to browser and desktop operation tools; Astra reads the text and button structure in the interface through accessibility mode and comprehensively judges through screenshots. The model writes the operations to be executed as code and hands them over to Codex's tools to complete.

Astra incorporates a large amount of professional environments and data during training and improves its strategy selection and error correction capabilities through reinforcement learning. The model can understand interfaces and select actions more accurately, which reduces trial and error and repetitive checks, allowing the same task to be completed with fewer interactions.

This release shows us a leap in new model capabilities: Computer Use capabilities allow Agents to cover more long-tail scenarios of white-collar work, inspiring users to actively attempt to delegate their daily tasks to Agents.

After Astra's release, the model's Aha moment appeared in many visual coding scenarios that previously required professionals to develop operations. For example, someone encountered Blender for the first time thanks to Astra and created a 3D model for real estate sales. Such examples of quickly seeing completed products spread rapidly online, igniting many people's desire to try and share.

The penetration of the Agent requires repeated capability expansions to convert onlookers into users. Seeing familiar cases at a glance is often more persuasive than evaluation scores. In user feedback, real estate practitioners saw others' experiences using visual coding and thought about integrating it into their own listings. Video workers also observed others using Astra to manage video workflows and proactively tried it out. These examples made more users aware that these tasks could be handed over to an Agent. Such expansion of long-tail scenarios will help the token consumption and agent penetration rates take another step forward.

01.

A Brief History of AI Computer Use

In October 2024, Anthropic launched the Computer Use capability alongside Claude 3.5 Sonnet. They adopted a purely visual approach, meaning the model determines how to interact with the computer screen through screenshots. The model conducts post-training based on pixels and returns pixel coordinates and actions in JSON format, for example:

"action": {"type": "click","x": 156,"y": 50}

Subsequently, driving tools like Stagehand or Playwright convert these outputs into actual interactions on the browser or computer.

After Claude 3.5 Sonnet, other labs began to release similar visual Computer Use models, training models to recognize pixels on the screen. OpenAI released Operator and computer-use-preview; Google DeepMind added Computer Use capabilities for Gemini 2.5 Pro.

However, these models are not perfect. Due to post-training being pixel-based, labs must choose a specific viewport size, such as 1288 × 711, and maintain that throughout the training process. When using these models in windows of different sizes, they tend to malfunction and start missing buttons.

Another obvious limitation is that these models rely purely on visual methods for application interaction, while some more complex interactions in applications are not visible "just by looking."

Numerous experiments have emerged around purely text-based solutions, as well as hybrid Agents combining DOM (Document Object Model) and visual elements. Standard Intelligence's FDM-1 is an intriguing Computer Use experiment: it encodes Computer Use through video instead of static screenshots.

FDM-1 is a Computer Use model developed by AI company Standard Intelligence, characterized by learning how people operate computers through continuous screen recordings.

02.

What Makes Astra Different?

To understand why Astra is different, we must first return to the 5.6 model family and discuss the Codex/ChatGPT harness. Computer Use is both an engineering problem of the harness and a research problem for the model: the model determines what to do, while the harness is responsible for execution. To excel in Computer Use, both issues must be addressed.

Computer Use has been popular for a long time, but it has not proven itself reliable enough for production environments. When OpenAI launched Computer Use capabilities in the Codex app, many developers began using it daily, gradually understanding how powerful this capability is.

Codex opens a built-in or local browser to complete designated tasks. Tasks can be performed in the background, allowing the user to continue using the browser for other matters.

Computer Use in Codex is faster and more accurate than Atlas, and it can accomplish tasks such as creating charts and invoking other tools by writing and executing code.

Astra builds on the capabilities present in 5.6, further enhancing speed and reducing costs.

To understand how this is achieved, we can start with how computer interfaces are organized. Today, most computers provide a graphical user interface (GUI) visible to us. We look at the pixels on the screen and decide where to click, which is similar to how early models operated.

What drives model progress today is a feature originally launched for visually impaired users: to assist those with vision or hearing impairments, every website in Chrome provides an "accessibility mode." This maps the content on the screen into an Accessibility Tree. This requires applications to expose the semantic information of user interface elements, allowing these elements to be operated upon.

The accessibility tree in the browser looks roughly like this:

The model can read it much more easily, as it removes all the code used to define the visual presentation layer; for webpages, this primarily involves CSS and class names.

The tokens used by the accessibility tree are fewer than those in screenshots, yet they can provide equally rich or even better screen context. Astra utilizes this tree to issue Computer Use commands such as clicks, inputs, and key presses. Each website in Chrome automatically generates an accessibility tree, which means they are inherently compatible with Astra out of the box.

03.

Architecture

Astra's architecture is quite simple:

When a Computer Use session begins, Codex starts a Node REPL to maintain session state and provides interfaces for browser operations or native Computer Use. The Agent chooses which set of interfaces to use based on the task.

Whether facing a native application or a browser, the Agent observes the page through text, screenshots, or a combination of both to accurately grasp the current state as much as possible. Subsequently, it decides what actions to execute through code. Today's Computer Use essentially operates in code mode. OpenAI's API documentation even suggests that all Computer Use be executed using code.

An output example is as follows:

{"type": "function_call","name": "exec_js","call_id": "call_123","arguments": "{\"code\":\"await page.getByRole('searchbox').fill('browser automation'); ...\"}"}

The local service (named CodexComputerUseIPC-5) is responsible for executing operations. The execution encapsulation layer converts the selected element into a native element ID (if the model chooses to use coordinates, it can also convert to coordinates), and then transmits JSON-RPC messages through a native pipe, matching requests and completing execution using request IDs.

In practical use, OpenAI recommends using Playwright and PyAutoGUI separately as frameworks to control browsers and computers.

Playwright: An open-source browser automation tool developed by Microsoft.

PyAutoGUI: An open-source mouse and keyboard automation tool created by programmer Al Sweigart.

Next, Astra checks the results of its operations, as successfully sending a request does not necessarily mean the operation has taken effect. The model requests to observe again, comparing the current state with the expected state.

This loop continues until the task is completed. Since Computer Use naturally requires maintaining state, the Node REPL must run continuously throughout the session.

Node REPL: Represents a continuously open code workspace intended to retain previously created variables, page objects, and intermediate data, allowing subsequent operations to continue from the previous step without needing to set up the environment each time.

In the above table, Astra is the only model that mandates automatic review. Astra employs a Guardian strategy to conduct safety audits on planned Computer Use before permitting execution.

Guardian uses GPT 5.6 Luna as the backend classifier, assessing the current workflow and any potential risks, then returning results classifying them as high or low risk. If deemed high risk, subsequent operations will trigger a review. The operations are then handed over to a safety review module, which conducts a complete assessment of the proposed actions.

{  "risk_level": "high",  "user_authorization": "low",  "outcome": "deny",  "rationale": "..."}

Safety reviews consider multiple aspects: the operations and their parameters the AI is ready to execute, relevant references in the conversation history (including user authorizations), the environment and permission settings for the main task, existing records and images in the code execution environment (REPL), and the review request along with its justification. The reviewer can complete the review without access to the complete accessibility tree.

Common Guardian interceptions include:

•Permission grants: Whether explicit authorization for the granted permissions and recipients has been obtained.

•Logins and account actions with significant consequences: Whether the user has explicitly authorized these actions.

•Submission of sensitive data: Whether permission has been granted for both the data itself and its destination.

•Clicks that may have significant consequences: The actual state of the interface and the impact of the click; whether form entries or settings are incorrect; whether they comply with user directives.

•Bypassing restrictions: Whether alternative paths have been authorized.

•Destructive operations: Whether there could be substantial state loss or irreversible damage.

•Accessing out-of-scope private data: Whether access falls within the overall authorized task scope.

In the alignment benchmark, Astra performs significantly better than previous models.

04.

A Representative of Speed

Compared to GPT 5.6, Astra has an additional layer of review, but it is actually faster. This is due to a smarter model requiring fewer interaction rounds.

Astra was trained on 100,000 GB300, consuming a significant amount of computing power. However, this model is also much smarter and has received ample RL training in Computer Use environments. Smarter → fewer rounds required to complete tasks → faster task completion. In this case, inference speed is not the decisive factor affecting Computer Use capabilities; when the generation speed exceeds 300 tokens per second (TPS), the speed of action execution becomes the bottleneck.

The execution framework has also undergone some optimizations, such as WebSocket preheating, connection reuse, and using previous_response_id for incremental requests. However, these optimizations do not correspond to the speed of the actions themselves but rather relate to the tool startup times.

05.

Current Drawbacks

Astra is indeed impressive, but some drawbacks can still be identified. Astra may make mistakes in the observation phase, obtaining incomplete accessibility tree states, insufficiently detailed screenshots, or outdated visuals. Between observing and executing actions, the status of the interface may also change. The accessibility trees in some applications can change frequently, causing operations to fail.

Long-duration Computer Use remains a challenge that has not been fully resolved. Due to effective contextual compression, Astra can run for extended periods while maintaining high execution quality; however, when Computer Use runs continuously for several days or even weeks, its performance in such cases has yet to be observed.

06.

The Frontier of Computer Use

Computer Use is just beginning to become user-friendly. We have witnessed its progression from frequently failing at trivial tasks to mining diamonds in Minecraft faster than a ten-year-old. People are finally starting to realize how much work can be delegated to AI.

As models shift from visual and screenshot-based methods to using accessibility trees, their performance has vastly improved. The prospect of more computing power also means better models at lower costs; as we continue to push the boundaries of post-training, models will keep improving in the specific domain tasks selected for training in the labs.

Astra combines speed and accuracy with strong safety measures to ensure Agents will not be hijacked. With each iteration of the Computer Use model, we move closer to achieving Computer Use capabilities suitable for production environments.

The future of software is to have AI do the work on behalf of humanity, allowing humans to focus on the problems that require thought.

免责声明：本文章仅代表作者个人观点，不代表本平台的立场和观点。本文章仅供信息分享，不构成对任何人的任何投资建议。用户与作者之间的任何争议，与本平台无关。如网页中刊载的文章或图片涉及侵权，请提供相关的权利证明和身份证明发送邮件到support@aicoin.com，本平台相关工作人员将会进行核查。