Skip to content

Computer Use Drivers

A driver is the surface the model controls. Pick a browser, a desktop, or a terminal.

A driver decides which machine the model uses. Change the driver, and the rest of your computer use code stays the same.

Pick a driver

PlaywrightSandboxYour own computer
MachineA browser page for each threadA Linux desktop sandbox for each threadYour mouse, keyboard, and main screen
Environmentbrowserlinuxwindows, mac, or linux
Runs onYour serverAny sandbox providerThe machine that runs chat()
navigate toolYesNoNo
zoomYesNoYes
cursor_positionNoYesYes
Setupplaywright-core, then download ChromiumA provider and a desktop image@jitsi/robotjs, and permissions on macOS

Playwright

If the task stays inside a web page, use Playwright.

Install the packages:

shell
npm i @tanstack/ai-computer-use @tanstack/ai-computer-playwright playwright-core zod

Then download Chromium one time:

shell
npx playwright-core install chromium

Then give playwrightComputer() to createComputerUse:

ts
import { createComputerUse } from '@tanstack/ai-computer-use'
import { playwrightComputer } from '@tanstack/ai-computer-playwright'

const computerUse = createComputerUse({
  driver: playwrightComputer({
    startUrl: 'https://tanstack.com',
    display: { width: 1280, height: 800 },
  }),
})

playwrightComputer() takes these options. All of them are optional:

  • startUrl: the URL that each new page opens. Without it, the page stays blank.
  • display: the viewport of each page in pixels. The default is 1280 x 800.
  • browser: a Playwright Browser that makes the pages. createComputerUse never closes it.
  • launch: the options for chromium.launch(). When you give browser, the factory ignores them.

Without browser, Chromium starts on the first action. computerUse.close() closes it.

The back and forward mouse buttons go back and forward in the page history.

Do you already have a Playwright page, for example in a test? See Share one page.

Linux desktop in a sandbox

If the model must use any app on a Linux desktop, use a sandbox. Each thread gets its own desktop, away from your server. The driver works with every sandbox provider: E2B, Daytona, Docker, and the others.

Install the packages. This example uses E2B:

shell
npm i @tanstack/ai-computer-use @tanstack/ai-computer-sandbox @tanstack/ai-sandbox @tanstack/ai-sandbox-e2b zod

Then do these steps:

  1. Set E2B_API_KEY in your server environment.
  2. Give sandboxComputer() a provider that makes desktop sandboxes. The E2B desktop template is one.
ts
import { createComputerUse } from '@tanstack/ai-computer-use'
import { sandboxComputer } from '@tanstack/ai-computer-sandbox'
import { e2bSandbox } from '@tanstack/ai-sandbox-e2b'

const computerUse = createComputerUse({
  driver: sandboxComputer({
    provider: e2bSandbox({ template: 'desktop', timeoutMs: 60 * 60_000 }),
  }),
})

Warning

Set the provider timeout above idleTimeout plus the longest time that one thread stays in use. E2B kills a sandbox at its timeoutMs, also when the thread is busy.

sandboxComputer() takes these options:

  • provider: the sandbox provider. It makes one sandbox for each thread.
  • display: the screen size of each desktop in pixels. The default is 1024 x 768.
  • persist: keep the state of each thread after the thread closes or goes idle. Set one or both:

How the sandbox driver behaves:

  • There is no navigate tool, because a desktop has no address. The model opens the browser on the desktop.
  • When a thread closes or goes idle, createComputerUse destroys its sandbox, unless you set persist.reconnect. With persist.snapshots, the driver saves the files first.
  • One scroll step is 100 pixels.
  • The driver quotes every key name and all typed text before they go to the shell.

The sandbox image

The image must have these tools:

  • xdotool and xdpyinfo.
  • scrot. Without it, the driver takes screenshots with ImageMagick import, but they do not show the mouse pointer.

When no X display runs, the driver starts one with Xvfb. If the image has startxfce4, the driver also starts it.

The E2B desktop template has all of these. For Docker, Daytona, and the other providers, use an image with the tools. On Debian or Ubuntu:

dockerfile
RUN apt-get update && apt-get install -y xvfb xdotool scrot x11-utils

If a tool is missing, the driver throws an error with its name.

Keep the desktop for later

Give sandboxComputer() a persist.reconnect store. A user who comes back to a thread a day later gets the same desktop, for example with the browser still signed in.

  1. Store one sandbox id for each thread, next to your threads.
  2. Set the provider to pause the sandbox, not to kill it. With E2B, use onTimeout: 'pause'.
ts
import { createComputerUse } from '@tanstack/ai-computer-use'
import { sandboxComputer } from '@tanstack/ai-computer-sandbox'
import { e2bSandbox } from '@tanstack/ai-sandbox-e2b'

declare const db: {
  getSandboxId: (threadId: string) => Promise<string | undefined>
  setSandboxId: (threadId: string, sandboxId: string) => Promise<void>
}

const computerUse = createComputerUse({
  driver: sandboxComputer({
    provider: e2bSandbox({
      template: 'desktop',
      timeoutMs: 15 * 60_000,
      onTimeout: 'pause',
    }),
    persist: {
      reconnect: {
        get: (threadId) => db.getSandboxId(threadId),
        set: (threadId, sandboxId) => db.setSandboxId(threadId, sandboxId),
      },
    },
  }),
})

With persist.reconnect, the driver works like this:

  • The first call of a thread creates a sandbox and stores its id.
  • A later call reconnects to that sandbox with provider.resume. If the sandbox is gone, the driver creates a new one and stores the new id.
  • The driver never destroys a kept sandbox. The provider timeout pauses or removes it.
  • If the setup of a reconnected sandbox fails, the driver keeps the sandbox and throws the error. The next call tries again.

What each provider keeps:

  • E2B with onTimeout: 'pause' keeps the memory and the disk. Open apps come back as they were.
  • Docker keeps the container. A stopped container starts again with its disk, and the driver starts the display again.

Warning

A kept desktop holds what the user did in it, like signed-in browser sessions. Delete the sandbox when the user deletes the thread.

Restore files into a new sandbox

Give sandboxComputer() a persist.snapshots option. The driver saves the files of one folder, like the browser profile. When the sandbox of a thread is gone, the next new sandbox gets those files, so the browser is still signed in. This works with every provider, also when its disk does not last, like Cloudflare.

ts
import { createComputerUse } from '@tanstack/ai-computer-use'
import { sandboxComputer } from '@tanstack/ai-computer-sandbox'
import { memorySandboxSnapshots } from '@tanstack/ai-sandbox'
import { e2bSandbox } from '@tanstack/ai-sandbox-e2b'
import type { SandboxHandle } from '@tanstack/ai-sandbox'

async function run(sandbox: SandboxHandle, command: string) {
  const result = await sandbox.process.exec(command)
  if (result.exitCode !== 0) throw new Error(result.stderr)
}

const computerUse = createComputerUse({
  driver: sandboxComputer({
    provider: e2bSandbox({ template: 'my-desktop' }),
    persist: {
      snapshots: {
        store: await memorySandboxSnapshots(),
        root: '/home/user/browser-profile',
        // Skip the caches. They are large, and the browser makes them again.
        policy: { exclude: (path) => path.includes('Cache') },
        // The browser writes its profile all the time. Close it cleanly during
        // a save. A kill signal can stop it before it writes its cookies.
        before: (sandbox) => run(sandbox, 'stop-browser'),
        after: (sandbox) => run(sandbox, 'start-browser'),
      },
    },
  }),
})

With persist.snapshots, the driver works like this:

  • When a thread closes or goes idle, the driver runs before, saves the files under root, then runs after.
  • When the driver creates a new sandbox for a thread with a saved snapshot, it runs before, restores the files, then runs after.
  • A sandbox that the driver reconnects to with persist.reconnect keeps its own files. The driver restores into it only when its disk is empty, like a Cloudflare sandbox after a restart.
  • Each save becomes the newest checkpoint of the thread. Use these stores only for computers.

To save at a moment that you must keep, like right after a sign-in, call saveComputerSnapshot({ sandbox, threadId, snapshots }). snapshots takes the same options as persist.snapshots. The sign-in tutorial does this.

memorySandboxSnapshots() keeps the files in memory, so they are lost when the server stops. For a user who comes back the next day, build the stores with createSandboxSnapshots on your database.

Warning

A snapshot of a browser profile holds its session cookies. A person with the cookies can use the signed-in accounts. Keep the blob store secret, and encrypt it in production.

Do you already have a desktop sandbox? See Share one sandbox.

Your own computer

If the model must use the apps on the machine that runs chat(), use localComputer(). The model moves your real mouse, presses your real keys, and reads your main screen.

Warning

Watch the model while it uses your computer. It can do what you can do there: read files, send messages, and delete data. If you cannot watch it, use a sandbox or a VM.

Install the packages:

shell
npm i @tanstack/ai-computer-use @tanstack/ai-computer-local @jitsi/robotjs zod

@jitsi/robotjs has prebuilt binaries for Windows, macOS, and Linux. If pnpm asks to allow its install script, you can deny it.

Then do these steps:

  1. On macOS, open System Settings > Privacy & Security. Give the Accessibility and Screen Recording permissions to the app that runs Node, for example your terminal.
  2. On Linux, use an X11 session. Wayland is not supported.
  3. Give localComputer() to createComputerUse:
ts
import { createComputerUse } from '@tanstack/ai-computer-use'
import { localComputer } from '@tanstack/ai-computer-local'

const computerUse = createComputerUse({
  driver: localComputer(),
})

localComputer() takes these options. Both are optional:

  • display: the size of the screenshots that the model sees. The driver scales each screenshot down to this size, and scales each click up to the screen. The default is your screen size, scaled down to fit in 1280 x 800. Keep the aspect ratio of your screen.
  • robot: the object that moves the mouse and presses the keys. The default is @jitsi/robotjs. A test can give a fake with the same methods.

Your computer is one machine:

  • Every thread shares it, so it serves one conversation at a time.
  • The model sees only the main screen.
  • There is no navigate tool. The model opens the browser on your desktop.
  • There are no back and forward mouse buttons.

Use it in chat

With every driver, createComputerUse gives tools, middleware, and interrupt. Give them to chat() with the thread id:

ts
import { chat } from '@tanstack/ai'
import type { ModelMessage } from '@tanstack/ai'
import { anthropicText } from '@tanstack/ai-anthropic'
import { createComputerUse } from '@tanstack/ai-computer-use'
import { sandboxComputer } from '@tanstack/ai-computer-sandbox'
import { e2bSandbox } from '@tanstack/ai-sandbox-e2b'

declare const messages: Array<ModelMessage>
declare const threadId: string

const computerUse = createComputerUse({
  driver: sandboxComputer({
    provider: e2bSandbox({ template: 'desktop', timeoutMs: 60 * 60_000 }),
  }),
})

const stream = chat({
  adapter: anthropicText('claude-opus-5-5'),
  messages,
  threadId,
  tools: computerUse.tools,
  middleware: [computerUse.middleware],
  interrupts: [computerUse.interrupt],
})

The model now uses a browser page or a full Linux desktop for each thread, or your own computer. For a machine that no driver controls, write a custom driver.

Advanced

Share one page

You already have a Playwright page, for example in a test. Give it to createPlaywrightComputerDriver. Use the Playwright package that your app already has.

ts
import { createComputerUse } from '@tanstack/ai-computer-use'
import { createPlaywrightComputerDriver } from '@tanstack/ai-computer-playwright'
import type { PlaywrightPageLike } from '@tanstack/ai-computer-playwright'

// Your Playwright page. It must have a viewport size.
declare const page: PlaywrightPageLike

const computerUse = createComputerUse({
  driver: createPlaywrightComputerDriver({ page }),
})
  • Every thread shares this page, so it serves one conversation at a time.
  • The page must have a viewport size. If it has none, the driver throws an error.
  • The driver reads the viewport size one time, when you create it.
  • A page from playwright, playwright-core, or @playwright/test works.

Your app owns this page. createComputerUse never closes it.

Share one sandbox

You already have a desktop sandbox. Give it to createSandboxComputerDriver:

ts
import { createComputerUse } from '@tanstack/ai-computer-use'
import { createSandboxComputerDriver } from '@tanstack/ai-computer-sandbox'
import { e2bSandbox } from '@tanstack/ai-sandbox-e2b'

const sandbox = await e2bSandbox({ template: 'desktop' }).create({})

const computerUse = createComputerUse({
  driver: await createSandboxComputerDriver({ sandbox }),
})
  • Every thread shares this sandbox, so it serves one conversation at a time.
  • createSandboxComputerDriver is async. It starts the desktop if needed, and reads the screen size one time.

Your app owns this sandbox. createComputerUse never destroys it. When you are done, call await sandbox.destroy().

zoom and cursor_position on Anthropic

The Anthropic adapter turns on zoom and cursor_position only on the computer tool versions that have those flags:

  • computer_toolset_20260801 turns on both when the driver has the method.
  • computer_20251124 turns on zoom only, and only when the driver has zoom.
  • computer_20250124 turns on neither.