Let an AI model update state inside a browser page

Open the live scene, ask a model to recolour or move an object, and follow the tool call back into the browser. The page retains its rendered state and validation rules while A11 carries typed calls between the model backend and the browser.

Enter make blob 2 red and move it up below. The right pane shows describe_scene, set_color and shift_position calls as the page handles them.

The complete implementation is js/demo/browser_tools.ts. The steps below use code from that file.

Choose state that belongs in the page#

This example keeps five rendered blobs in a browser-owned Scene. A model can inspect their positions, recolour them and move them, but it does not receive the Three.js objects or control the renderer directly.

The same division applies to an editor selection, an unsaved document or a canvas. The page owns the state and provides a small set of operations that preserve its rules.

The demo exposes three operations:

Describe each tool as an action#

Define one port for each argument the model supplies. Descriptions explain the operation and each input; the output records what changed.

const SET_COLOR_SCHEMA = new ActionSchema({
  name: 'set_color',
  description: 'Recolour blobs: the i-th id is given the i-th colour.',
  inputs: {
    ids: new ActionPortSchema({
      name: 'ids',
      type: 'application/json',
      required: true,
      description: 'Which blobs to recolour.',
    }),
    colors: new ActionPortSchema({
      name: 'colors',
      type: 'text/plain',
      required: true,
      description: 'One `#rrggbb` per id, in the same order.',
    }),
  },
  outputs: {
    recoloured: new ActionPortSchema({
      name: 'recoloured',
      type: 'application/json',
      unary: true,
      required: true,
      description: 'How many blobs changed colour.',
    }),
  },
});

The TypeScript port declares a MIME type. Supply the value shapes used by the model tool definition as Zod schemas:

const PORT_SCHEMAS = {
  set_color: {
    ids: z.number().int(),
    colors: z.string(),
  },
  shift_position: {
    ids: z.number().int(),
    dx: z.number(),
    dy: z.number(),
    dz: z.number(),
  },
} as const;

This keeps the A11 action interface and the model-facing argument contract aligned. The backend derives its tool definitions from the registered actions plus these value shapes.

Register a handler around the scene#

Create the registry beside the state it serves. The set_color handler reads all arguments, resolves the requested blobs, validates every colour and then changes the scene.

const registry = new ActionRegistry();

need(
  registry.register(
    SET_COLOR_SCHEMA.name,
    SET_COLOR_SCHEMA,
    async (action): Promise<Status> => {
      try {
        const [ids, colors] = await Promise.all([
          readAll(action, 'ids'),
          readAll(action, 'colors'),
        ]);
        const blobs = blobsFor(scene, ids);
        if (isStatus(blobs)) {
          return await refuse(action, blobs, onLog);
        }

        const wanted: string[] = [];
        for (const [index, blob] of blobs.entries()) {
          const color = colorFor(
            colors[Math.min(index, colors.length - 1)],
            blob.id,
          );
          if (isStatus(color)) {
            return await refuse(action, color, onLog);
          }
          wanted.push(color);
        }

        blobs.forEach((blob, index) => {
          scene.recolour(blob, wanted[index]!);
        });
        const result = need(await action.getOutput('recoloured'));
        need(await result.finalize(blobs.length));
        await narrate(
          action,
          `Recoloured ${blobs.length} blob(s): ${blobs
            .map((blob) => blob.id)
            .join(', ')}.`,
          onLog,
        );
        return okStatus();
      } catch (error) {
        return statusFromUnknown(error, 'set_color failed.');
      }
    },
  ),
);

The complete source includes blobsFor and colorFor. Invalid IDs and colours return INVALID_ARGUMENT to the model. The model can correct the arguments in another call, and the page remains unchanged until every argument passes validation.

Keep activity separate from the tool result#

The numeric recoloured output is the tool result consumed by the model. The page also records a sentence for the person watching:

async function narrate(
  action: Action,
  text: string,
  onLog: (text: string) => void,
): Promise<void> {
  onLog(text);
  need(await action.log(text));
}

action.log() does not need an output port. The backend associates the log with the tool call and keeps it out of the model's structured result.

Attach the page registry before connecting#

Bind the registry to the A11 session before attaching its WebSocket stream. An incoming call then reaches a registered handler as soon as the session begins receiving messages.

const session = need(Session.create({ actionRegistry: registry }));
const stream = need(WebSocketWireStream.connect(serverUrl));
need(await session.addStream(stream, StreamMode.START));

The backend discovers the page actions through this session. Each discovered schema belongs to the current connection because its handlers operate on state in this specific page.

Allow the model to call the page actions#

The application sends an ordinary model turn and names the actions available for that request:

need(
  call.setHeader(
    LlmHeaders.ALLOWED_LLM_ACTIONS,
    'describe_scene,set_color,shift_position',
  ),
);

When the model selects set_color, the backend dispatches the call through the same WebSocket. The browser handler validates the arguments, changes the scene and returns the result. describe_scene gives the model authoritative world coordinates before a mutation; rendered pixels remain presentation data.

Adapt the pattern to an existing interface#

Start with one operation that already has a clear owner:

• An editor action receives a range and replacement text, then uses the editor's own transaction API.
• A canvas action receives object IDs and bounded coordinates, then applies the existing selection and snapping rules.
• A document action receives a validated patch, then uses the page's current undo and persistence path.

Keep credentials and backend-owned tools on the server. Browser-hosted actions fit capabilities whose authoritative state already lives in the page.

Run and inspect the complete example#

Use the deployed scene to watch the round trip without installing anything. Try give the blobs a warm palette, then inspect the activity log beside the scene.

Read the complete TypeScript source to follow the scene implementation, argument validators, registry connection and model turn in one file. The TypeScript model guide explains provider selection, history and streamed model text. The agent debugging guide continues with nested calls, failures and cancellation in Studio.