Agents can’t point

Hi, Tim Gilboy, tldraw PM here.

Claude - 3 minutes ago
The "Build with the tldraw SDK" developer banner is the only text callout a brand-new user sees, and its dashed border reads as unfinished UI. Consider deferring it until after first interaction, since most first-time visitors are end users, not developers.

Tim - 2 minutes ago
@Claude could we change the logic so this only shows for logged in users?

Claude - 2 minutes ago ·
Done — I flipped the check in TlaAnonDotDevLink.tsx so the banner only renders when isSignedIn is true. I'll rerender the page and take a new screenshot now

That’s an actual conversation I had with Claude last week to try to make some improvements to tldraw.com for new users.

While the content isn’t particularly interesting, where the conversation took place is. This conversation was all within one comment thread about a section of our footer.

This conversation was all within one comment thread about a section of our footer. It didn’t live in a sidebar or within my terminal - it was right there on the canvas. At no point did I have to send Claude a screenshot or explain what “this” meant. I had the agent open up tldraw.com in a headless browser, capture it, drop the image on the canvas, and leave comments wherever it had feedback.

I talk by pointing

Models and agents have changed enormously since ChatGPT shipped in 2022, but the interface has barely moved. You’re typing in a text box, a sidebar, a terminal - almost always something detached from what you’re actually working on.

While chat has its place, it’s bad at deictics; this, here, there. Things we say to each other all the time, but an agent has no visual context for.

Try it with an agent. If you want it to change that shape, and your options are to describe it ("the blue rectangle top left, no, the other one"), to hope the harness passes the selection through, or to send the whole document and trust the model to work it out. The agent can't point, and it can’t understand what you’re trying to point to for it.

Comments > Shapes

Historically (in things like fairies and our agent starter kit), we’ve had agents able to leave feedback and communicate on the canvas by writing, adding notes, adding shapes, etc. While these make for some good demos, they tend to run into the same problem; they’re anonymous.

The moment there's an agent and a person (or loads of people) on the same board, you can't tell who said what. It lives in the canvas, so twenty notes is twenty things to delete before you can export. It isn't a thread, so there's nowhere to answer. It all just accumulates and clutters up over time.

Jess wrote about why a comment isn't a shape when we shipped commenting in version 5.3: it's its own record, sitting parallel to the canvas and pointing into it. That design was for people collaborating. But it turns out it works pretty damn well for agents too.

Comments aren’t built for agents, but they manage

Nothing in that workflow with Claude was built for an agent.

It used the same comment tool we ship as the regular bit of the SDK. I replied with a simple @mention, the same way I'd mention a colleague (I just added Claude Code to our roster). It answered in the thread. The pin held onto the banner the way any pin holds onto any shape.

And it made the whole interaction way nicer. The agent was able to give dozens of pieces of feedback on the same screenshot. I could easily understand what each one was referencing and reply to them independently. Or realize I disagreed and just resolve them and clean up the canvas.

A motley crew of agents

Screenshots and wireframes felt like the obvious way to use comments to drive agents. But I wanted to see what it could look like to explore something more complex and a little sillier.

So I built a pirate game where you're the captain of a ship and each of your crew members is an agent, each with a different job. The catch is you can’t take any action as the captain - you can just give your crew orders through comments.

It turned out that for coordinating multiple agents, comments are a really nice interaction surface. You can easily have multiple threads going at once; they can interact and mention each other, and you can easily bring in different contextual information from the pins.

You (probably) aren’t often going to want to drop a pin on a rival ship and tell your helmsman to set sail to intercept them, but if you’re working with multiple agents in your browser and need to coordinate multiple tasks, comments are a surprisingly nice surface for it.

Try it out

Comments shipped in version 5.3 of the SDK, as a new package @tldraw/commenting. Take the <CanvasComments> overlay whole, or build your own surface from the hooks and components underneath, whether that’s a review panel, an agent inbox, or a crew of pirates.