Harnessing the agents

Hey, it’s Max, tldraw product engineer, here.

We recently launched tldraw offline, a local whiteboard that you and your agents can work in together. It works really well! However, if you’ve ever used a coding agent for any kind of visual or design work, you’ll know that even the most advanced ones are quite bad at it. So how did we get coding agents to understand how to work in 2D space, instead of a linear space like chat?

The answer is a set of tools, systems, and opinions that make up something we call the spatial harness, and its story traces back through years of tldraw projects.

The spatial harness

First, a definition. The spatial harness isn’t exactly a harness, at least not in the same way Pi or Claude Code is. It’s shorthand we, at tldraw, use for the systems we have in place that let an agent understand and act in the 2D space of the canvas.

What it's actually composed of changes depending on the iteration. Generally, it includes the ways of transforming the representation of the canvas into a form that is easy for the agent to understand and access, tools it can use to edit the canvas, and any number of other systems that are built around the agent to help it, such as a canvas linting system we built to warn the agent when it has, for example, made two text shapes overlap.

Some background

The tldraw SDK was developed to make it possible to create canvas apps and experiences without having to worry about building the canvas itself, a notoriously tricky design and engineering problem. To that end, we’ve long been interested in what it would look like to interact with AI on the canvas, a fundamentally different environment and interface than chat.

The spatial harness in tldraw offline lets agents write code to edit the canvas (drawing inspiration from CodeAct and Code Mode) instead of calling pre-defined tools, and puts most of the canvas context behind tool calls. This allows the agent to dynamically pull in context as needed, instead of overloading the model with potentially unimportant tokens up front (this approach was inspired by RLM).

This is incredibly important when paired with our code mode implementation, as most of the tldraw SDK documentation needs to be available to the agent. The spatial harness is the product of many projects aimed at this research goal, and we couldn’t have built it without everything we learned along the way. From here on out, I'll explore the projects we experimented with and what we learned from them.

If you stop reading here, though, here are two of the most interesting things to shake out of this research:

1. The canvas makes it so much easier to manage multiple agents working on one document at once, something that’s incredibly difficult to do from chat. You can communicate agent state through animations or sprites, and you can see “where” in the document they’re working just from a glance.
2. Agents act differently in the canvas. When you ask an agent who lives on the canvas to explain something to you, they’ll draw something visually to aid it. They organize things hierarchically the same way you might. Are they smarter in canvas? It’s hard to tell, but they act in a totally different way than they do in chat.

All this to say, using agents on the canvas is a totally different experience than in chat. Agents love to take their space!

Teaching the agents

Teach was our first experiment at getting an agent to understand and edit the canvas given explicit human instruction (there were also even earlier experiments with implicit human instruction, such as autocomplete for canvas). You type a single prompt — say, “blow out the candle” — and the agent does exactly one thing in response: it looks at the canvas, decides what “blowing out a candle” would mean as a sequence of shape edits, and executes it.

As simple as it may seem, it became the foundation for how we thought about agents on the canvas. First, we had to ask what it means to give a language model a place to look and a place to act.

Answering that question is what an “agent harness” is for: the scaffolding around the model that turns “predict the next token” into “do a task.” Teach is the simplest possible version of that scaffolding, built for a single spatial task.

Agent starter kit

From there, we built the agent starter kit, which wraps the Teach engine in an agentic loop. Instead of a single-shot response to your prompt, an agent can plan its work, navigate the canvas, set a goal, review its work, and loop until it decides the goal is complete.

Like a coding agent, it can also adjust mid-workflow to changing circumstances. Maybe something it was working on has been deleted, and it needs to recreate it, or maybe it needs to ask you for clarification about which diagram you asked it to edit.

We also discovered that, in the canvas, the context the agent needs to understand and act in its environment is quite different from what a coding agent needs. For every prompt we send to the agent, including when it starts a new turn during its loop, we tell it where your viewport bounds are. We also tell it the coordinates of its viewport bounds, a screenshot of those bounds, and the shape data for all shapes contained within them. We also send information about all shapes on the canvas, but to avoid polluting the agent's context, we send less detailed information about shapes outside its viewport. It's analogous to giving a coding agent an overview of a repo instead of giving it all of the repo's code in every prompt.

This is just one example of a system we developed in order to bring an agent into the canvas (there are many more, and you can read about them here if you're interested).

Fairies at work

With Fairies, we wanted to take the harness further. Instead of an invisible agent doing its thing, what if it was embodied and localized to a specific "place" inside the canvas, and what if more than one agent worked on the same canvas at once?

Each fairy is its own agent, with its own goals, but they share space and can collaborate—dividing up work, reacting to what another fairy just did, and signalling status both to each other and to the user.

Every action the fairies take is one of many discrete actions we decided on ourselves and implemented manually, and most of them involve reading and editing the canvas or collaborating with each other. They can't write code or interact with the world outside the canvas. Their process runs in the browser, and they don't have access to any kind of filesystem, nor do we give them any tools to search the internet or access external data.

This was partially a scope decision; we launched fairies as a limited-time addition to tldraw.com for the month of December. But it was also a product decision. We needed to figure out what multi-agent collaboration looked like on the canvas, and continue to develop that system, without worrying about anything outside the canvas.

Unfortunately, this left fairies unable to do "real work." Read an email? Move a real file? Trigger something on your desktop? Not possible — a fairy’s entire world is the canvas.

Part of the issue was our custom harness. While we could add tools for accessing specific third-party APIs, we didn’t have good tools for web search; it wasn’t set up to work with MCP, and because it was running in the browser, it didn’t have access to a file system or scripting capabilities, which also made it less suitable for super long-running tasks. A bespoke harness that couldn’t work without the canvas, while incredibly powerful, turned out to have some difficult limitations.

Taking Fairies offline

We found a way to address this in tldraw offline, a local app that works on local files. Your local coding agents can interface with it, meaning we could outsource all of the harness and agentic loop logic to systems purpose-built for it, and we could just focus on making the canvas understandable and work-on-able for the agent. This informed a lot of our design decisions moving forward.

Because your entry point to the canvas is now a local coding agent, it can bring whatever tools, context, and/or MCPs you've decided to give it to complete its task. It can read a codebase that lives on your machine, then diagram some behavior on the canvas. Or, write a script that gets your emails, and then creates a custom tldraw shape in order to represent those emails on the canvas (we ship an agent skill with the tldraw offline that tells it how to do these things).

Fairies edited the canvas, as well as did everything else, by outputting discrete, declarative actions (createShape, updateShape, etc), and our first iterations of the offline harness did this as well. While building, we realized that this approach was severely limiting what it could do. At the same time, we had been seeing research and examples in the wild indicating that allowing the agent to write code to describe what it wanted to do instead of performing discrete actions might be the way to go, both for decreased token use and latency, but also for allowing it to reason and plan through the writing of code, which it turns out agents are quite good at!

We also realized that we have the perfect target for any code the agent writes: the live tldraw Editor instance running in the canvas. The Editor has methods for doing anything one might need to do in tldraw, from creating and editing shapes to modifying the user selection, changing the user's view, or even registering live side effects, allowing for the agent to create custom "UI" and interactions out of tldraw shapes.

Agents need space.

We still use all the lessons we learned from fairies, the agent starter kit, and teach, except we now allow the agent to proactively get the context it needs from the canvas instead of it all in the context up front.

Most importantly, it lets agents do whatever they want in tldraw without requiring us, the humans, to foresee and build specific tools for everything they might want to do. Frequently in testing tldraw offline, we were surprised by the creative methods (and sometimes strange workarounds) the agents would use in order to carry out our requests, and throughout the process we tried to yes-and the agents by building tools to make it easier to do the things we saw them already trying to do.

Executing arbitrary code is obviously dangerous, which meant we had to think deeply about what we were willing to trade off for security. Read about how that process fed into offline.

The canvas gives agents the space they need to interact with each other and with the user. The harness lets agents understand and work on the canvas while keeping them situated. The same features that make a canvas great for people collaborating—cursors, selections, seeing where everyone’s attention is—also make it great for understanding what your agents are doing.