An experiment in letting an LLM drive Blender , a chat panel, a Python agent service, 700 viewport screenshots, and a lot of lessons about agent architecture.
Over a couple of weeks in March I built an experimental “Cursor for Blender“: a chat panel inside Blender where you type something like “create a wooden motorboat like this reference image”, attach a photo, and an AI agent writes and runs the Blender Python to build it, then looks at the viewport, judges its own work, and iterates.
This post is the honest version of how that went: the architecture, the three major rewrites, the results and what I’d do differently.
The setup
The system has two halves:
- A Blender add-on: a chat UI in the 3D View sidebar, a scene extractor (objects, transforms, materials, relations), viewport capture, and a small execution layer that runs generated code in one undo group.
- A local agent service: a Python WebSocket server that talks to an LLM (Claude, or any OpenAI-compatible endpoint). It receives the user message, the scene snapshot, and a viewport screenshot; it returns plans and
bpyPython code.

The key feedback trick: every time code runs, the add-on captures the viewport and sends it back to the agent. The agent literally sees what it built. Over eight days of experiments the agent looked at the scene 697 times, every one of those screenshots is saved in a debug folder, which is why I can show you exactly what it saw below.
Iteration 1: tool calls (too rigid)
The first working version didn’t generate code at all. The LLM returned a JSON plan of whitelisted tool steps : create_primitive, move_object, add_light, set_material_slot, which the add-on validated and executed.
This was safe and predictable, and it could handle “add a sun light” or “move the cube up”. It could not build a boat. A dozen primitive-manipulation tools is nowhere near expressive enough for real modeling, and every new capability meant hand-writing another tool. The ceiling was obvious within a day.
Iteration 2: let it write Python, let it see
The second version gave the model two big upgrades:
- Python mode : instead of tool steps, the LLM writes a full
bpyscript and the add-on executes it. - Vision : the viewport screenshot and user reference images go into every LLM call (this is when I switched to Claude for multimodal).
This is where it started actually modeling. Given this reference image:

…the first attempts produced things like this:

Recognizably a boat. Not recognizably the boat. And single-shot generation had a fundamental problem: one giant script either mostly worked or failed halfway through, leaving debris in the scene the model didn’t know about.
Iteration 3: the “agentic” pipeline
So I built the thing everyone builds next: a multi-stage pipeline.
The final pipeline looked like this:
user message + reference images + viewport -> intent extraction (LLM call 1) -> image analysis (LLM call 2) -> prompt refinement (LLM call 3) -> detailed plan, up to 12 steps (LLM call 4) -> per-step code generation (LLM call 5..N) -> execute step in Blender -> screenshot -> strict quality gate (another LLM call) -> satisfied? next step : refine / repair
Some genuinely good ideas landed here:
- Incremental code generation. Code for step N+1 is only generated after step N has executed, using a fresh screenshot and scene snapshot — so each step sees reality, not the plan’s assumptions.
- Error self-repair. When
bpycode threw, the error, the failed code, and the live scene state went back to the model for a corrected version. Failed errors were also appended to a persistent log that got injected into future prompts, so the agent stopped repeating known mistakes. - Version-aware context. The add-on shipped the actual
bpyAPI names and shader node types from the running Blender to the agent.
That last one mattered more than expected. My favorite failure: the agent kept using ShaderNodeTexMusgrave for terrain materials, a node that Blender removed in 4.1. You can see the error caught live in the panel here, mid-castle-build:

The results
The boat, over successive sessions, went from the crude placeholder above, through an over-stretched hull with furniture-like white blocks:

…to something with an actual hull shape, windshield, and wood materials:

Here’s the step-by-step pipeline at work — the panel on the right shows the plan (“Step 3 of 9”), with code being generated for the next step from a fresh top-down look at the scene:

The castle was the more ambitious test: “Create a model like the Hogwarts castle. Use the reference image.” First pass, blocky:

After iterations of “add details” it became a genuinely respectable clay model, towers, spires, courtyard walls, a rocky base:

What didn’t work
I want to be honest about the failure modes, because they’re more instructive than the successes.
The quality gate churned. I gave the verifier a strict prompt: results must be “detailed and real-life accurate”, with up to 6 refinement attempts per step. The session logs are full of Step 0: not satisfying, advancing to step 1/8, eight steps in a row where the judge never once said yes. A judge that demands photorealism from procedurally-generated primitives always says no, so its signal carried almost no information. Some sessions ended in failed after 3 attempts with the scene half-modified.
The pipeline lost context at every seam. Each of those 5+ LLM calls per turn was stateless, connected only by truncated text summaries of what earlier calls decided. The step-code generator couldn’t see the code from previous steps, only their titles, so step 3 had to guess what step 2 named its objects. Repairs happened on top of half-executed scripts, so the scene accumulated debris from failed attempts.
JSON was the wrong transport for code. Returning Python inside a JSON string field caused endless escaping bugs (a backslash-n arriving as a literal two-character sequence instead of a newline), which I patched with heuristics in two places. The codebase grew five separate “strip the markdown fence and parse JSON” functions.
Fail-open defaults lied to me. When the quality check returned malformed JSON, the code treated it as “satisfied” to avoid infinite loops — meaning a garbled response silently reported success. Debugging “why did it stop early” took longer than any modeling problem.
And costs multiplied. Every stage re-uploaded the viewport plus up to 8 reference images. One user message could mean five vision calls before the first line of code ran.
What I learned
- A pipeline is not an agent. Every hardcoded stage, intent extraction, prompt refinement, planning , replaces the model’s live reasoning with a frozen, lossy approximation of it. The stages I added to make the system smarter mostly made it forgetful.
- The feedback loop is the product. The single best design decision was making the agent look at the viewport after every change. The screenshots-plus-error-log loop is what took the castle from blocks to spires. Everything else was overhead around that loop.
- Judges need a relative bar, not an absolute one. “Does this satisfy what the user asked for” works. “Is this photorealistic” makes a judge that always fails you.
- Give the model ground truth, not vibes. A structured scene diff (what objects appeared, moved, changed material) after each execution would have been cheaper and more reliable than a vision call. That’s the top of my list for the rebuild.
- Version-pin the model’s world. LLMs know a Blender from their training data. Shipping the live API surface (and errors) into context fixed a whole class of failures.
What’s next
The rebuild is a much smaller system: one conversation, one tool-calling loop, run_python(code), screenshot(), inspect_scene(query), done(), with full history, so planning, verification and repair are just turns in a conversation instead of separate subsystems. The Blender add-on shrinks to a thin executor. Less code, more model.
The experiment version, for all its flaws, built me a castle. I’m curious what the simpler one can do.
Built with Blender 4.x, Python, WebSockets, and Claude. The screenshots in this post are the actual images the agent received during its runs, its point of view, unedited.
















Machine learning is a big trend in the industry now. The performance of these algorithms are improving every day and coming in par with human skills.








