AI integrations

You can use language models to read the tldraw canvas and to create shapes on it. This page covers three patterns for doing that: using the canvas as an output surface for generated content, building visual workflows where AI models are nodes in a graph, and giving an agent direct control of the editor. It also covers exporting canvas content so a model can read it.

If you're feeding the docs themselves to a model, see LLM documentation. To turn model-generated diagram text into shapes, see Mermaid diagrams. To let an agent simulate user input instead of calling editor methods, see Driving the editor.

Canvas as output

The simplest AI integration uses the canvas as a surface for displaying generated content. When an AI model generates an image, website preview, or other visual output, you can place it on the canvas as a shape.

Embedding generated content

Use the EmbedShapeUtil to display websites, or create custom shapes to render generated images and HTML content:

editor.createShape({
	type: 'embed',
	x: 100,
	y: 100,
	props: {
		url: 'https://generated-preview.example.com/abc123',
		w: 800,
		h: 600,
	},
})

Embeds from hosts that aren't in the util's embed definitions render in a restricted sandbox. To allow a host of your own, use EmbedShapeUtil.configure({ embedDefinitions }). For content you generate yourself, a custom shape (below) is usually the better fit.

For generated images, use the AssetRecordType to create an asset from a blob or URL, then create an image shape:

const asset = AssetRecordType.create({
	id: AssetRecordType.createId(),
	type: 'image',
	props: {
		src: generatedImageUrl,
		w: 512,
		h: 512,
		mimeType: 'image/png',
		name: 'generated-image.png',
		isAnimated: false,
	},
})

editor.createAssets([asset])
editor.createShape({
	type: 'image',
	x: 100,
	y: 100,
	props: {
		assetId: asset.id,
		w: 512,
		h: 512,
	},
})

Custom preview shapes

For richer AI output, create a custom shape that renders generated content. This approach works well for live HTML previews, interactive prototypes, or any content that needs special rendering:

import { HTMLContainer, ShapeUtil } from 'tldraw'

// Abbreviated: a full ShapeUtil also needs getDefaultProps, getGeometry, and
// getIndicatorPath. See the Shapes docs for a complete custom shape.
class PreviewShapeUtil extends ShapeUtil<PreviewShape> {
	static override type = 'preview' as const

	component(shape: PreviewShape) {
		return (
			<HTMLContainer>
				<iframe
					srcDoc={shape.props.html}
					sandbox="allow-scripts"
					style={{ width: '100%', height: '100%', border: 'none' }}
				/>
			</HTMLContainer>
		)
	}
}

This pattern is useful for "make real" style applications where users sketch a UI and an AI model generates working code to preview alongside the original drawing. See Shapes for how to write the rest of the util.

Visual workflows

The tldraw binding system enables node-based visual programming where AI models can be part of a larger workflow. Shapes represent operations or data sources, and bindings connect them to form processing pipelines.

Workflow architecture

In a visual workflow, each node is a custom shape with input and output ports. Connections between nodes are bindings that track relationships as shapes move. When data flows through the system, each node processes its inputs and produces outputs for downstream nodes.

See the Workflow starter kit for a complete implementation of this pattern. The starter kit includes:

• Custom node shapes with configurable ports
• A binding system for smart connections that update as nodes move
• An execution engine that resolves dependencies and runs nodes in order
• Tools for creating and managing connections

Adding AI to workflows

To add AI capabilities to a workflow, create node types that call AI models:

// Abbreviated: a real NodeDefinition also declares a validator, ports, a default
// value, and a body height. See the workflow starter kit for the full interface.
class LLMNode extends NodeDefinition<LLMNodeData> {
	static type = 'llm'

	async execute(shape, node, inputs) {
		const response = await fetch('/api/generate', {
			method: 'POST',
			body: JSON.stringify({ prompt: inputs.prompt }),
		})
		const result = await response.json()
		return { output: result.text }
	}
}

Workflow systems let users compose AI operations visually: connect a prompt source to a model, route its output to other steps, and build a pipeline without writing code.

AI agents

For full canvas control, you can give AI models direct access to read and manipulate shapes. An agent can observe what's on the canvas, understand spatial relationships, and create or modify shapes to accomplish tasks.

Agent architecture

The Agent starter kit is a complete implementation. The agent gathers context from screenshots and structured shape data, applies the model's responses through a set of typed actions, and streams results onto the canvas as they arrive. Chat history carries across prompts.

Using the agent programmatically

The agent exposes a simple API for triggering canvas operations:

// Inside a component wrapped by the starter kit's TldrawAgentAppProvider
const agent = useAgent()

// Simple prompt
agent.prompt('Draw a flowchart showing user authentication')

// With additional context
agent.prompt({
	message: 'Add labels to these shapes',
	bounds: { x: 0, y: 0, w: 500, h: 400 },
})

How agents see the canvas

The agent builds context from multiple sources:

• A screenshot of the current viewport
• Simplified representations of shapes within view
• Information about shape clusters outside the viewport
• The user's current selection and recent actions
• Conversation history from the session

How agents manipulate the canvas

Agents perform operations through typed action schemas. Each action has a defined structure, and the agent system validates, sanitizes, and applies actions to the editor:

// Abbreviated from CreateActionUtil in the agent starter kit. The model describes
// shapes in a simplified format that the util converts to a real shape record.
class CreateActionUtil extends AgentActionUtil<CreateAction> {
	override applyAction(action: Streaming<CreateAction>, helpers: AgentHelpers) {
		const { shape } = action
		if (!shape || !shape._type) return

		// Translate from the model's coordinate space back to the page
		const shapePartial = helpers.removeOffsetFromShapePartial(shape)
		const result = convertPartialFocusedShapeToTldrawShape(this.editor, shapePartial, {
			defaultShape: getDefaultShape(shape._type, action.complete),
			complete: action.complete,
		})
		if (!result.shape) return

		this.editor.createShape(result.shape)
	}
}

The sanitization layer handles common LLM mistakes. It corrects shape IDs that don't exist, ensures new IDs are unique, and normalizes coordinates.

Reading canvas content

For applications where AI reads but doesn't modify the canvas, you can export canvas content for analysis.

Screenshots

Use Editor.getSvgString or Editor.toImage to export the current view or specific regions:

const svg = await editor.getSvgString(editor.getCurrentPageShapes())
const { blob } = await editor.toImage(editor.getCurrentPageShapes(), { format: 'png' })

Structured data

Access shape data directly from the store for text extraction or structured analysis. Use ShapeUtil.getText to extract a shape's text content as a plain string. It returns undefined for shapes with no text:

const shapes = editor.getCurrentPageShapes()

const textContent = shapes.map((shape) => ({
	id: shape.id,
	type: shape.type,
	text: editor.getShapeUtil(shape).getText(shape),
	bounds: editor.getShapePageBounds(shape),
}))

Sending both to the model works best: the image shows spatial relationships and styling, and the structured data gives exact text and positions.

Starter kits

We provide several starter kits for AI integrations:

Related

• LLM documentation — Feeding the tldraw docs to a model
• Mermaid diagrams — Turning diagram text into shapes
• Driving the editor — Simulating user input programmatically
• Shapes — Creating custom shapes for AI-generated content
• Bindings — Connecting shapes for workflow systems
• Image export — Exporting canvas content for AI analysis