Code as Canvas: How AI Coding Agents Generate Images Through Executable Scripts and Rendering Engines
In the last few years, most conversations about AI image generation have centered on diffusion models and their ability to turn a text prompt into a photorealistic or stylized picture in seconds. Tools built on these systems produce stunning results, but their native representation is typically raster imagery rather than an explicitly structured scene. Although modern models increasingly support masking, inpainting, reference-based editing, and iterative image edits, reliably making exact structural changes-such as enforcing object counts, coordinates, typography, or reusable hierarchy-can remain difficult.
Tools built on these systems produce stunning results, but their native representation is typically raster imagery rather than an explicitly structured scene. Although modern models increasingly support masking, inpainting, reference-based editing, and iterative image edits, reliably making exact structural changes-such as enforcing object counts, coordinates, typography, or reusable hierarchy-can remain difficult.
A parallel and increasingly powerful approach has been emerging. Instead of asking the model to paint the final pixels directly, researchers and practitioners ask large language models and coding agents to write the program that produces the image. That program might be SVG markup, HTML and CSS, a Python script using Matplotlib or a canvas library, a Three.js scene, a p5.js sketch, a Blender Python script, TikZ code for a scientific diagram, or even a shader.
A deterministic rendering engine then turns the code into pixels. Because the intermediate representation is editable source code, an agent (or a human) can inspect the result, diagnose problems, revise specific lines, and re-render. The process becomes a closed loop of Code → Render → Inspect → Revise.
This is what people mean by code-native AI image generation. It reframes visual creation as a programming task and leverages the strengths that modern language models already possess: writing correct, structured code and reasoning about spatial and logical relationships. The approach is especially powerful when the agent can see the rendered output (via a vision-language model) and iterate automatically.
The fundamental distinction: pixels versus programs
Pixel-native systems operate in a continuous latent space. They are excellent at atmosphere, texture, lighting, and photorealism. They struggle with discrete constraints-exactly three apples arranged left-to-right, text that must be legible and correctly spelled, geometric precision, or hierarchical structure that must survive later editing.
Code-native systems invert the priority. The model produces an explicit symbolic description. The rendering engine deterministically executes the generated description. When the code correctly expresses the intended constraints, object counts, coordinates, layering, text content, and proportions can therefore be enforced explicitly rather than left to stochastic image sampling. The trade-off is that pure code is weaker at high-frequency detail and photorealistic materials. The most interesting recent systems therefore hybridize the two: the agent uses code to establish the structural skeleton (the “sketch”), then hands the result to a diffusion or image model for texturing and realism (the “coloring”).
This hybrid pattern appears repeatedly. An agent writes SVG or HTML that locks composition and typography; an image model then fills in surfaces and lighting while respecting the underlying layout. The code remains the editable source of truth.
Why iteration becomes possible
The decisive advantage is the feedback loop. In a prompt-only diffusion pipeline, a new attempt is often another stochastic generation, and even image-editing pipelines generally provide less explicit access to the underlying object hierarchy than programmatic representations do. Feedback is global and hard to localize: “the composition feels off” does not map cleanly to a single corrective action.
In a code-native pipeline the mapping is local and actionable. If two shapes overlap incorrectly, the agent can change the coordinates or z-order in the SVG. If text overflows a box, it adjusts the font size or container dimensions in CSS. If a physical simulation looks wrong, it revises the parameters of the numerical code that generated the geometry.
A vision-language model can look at the rendered image, describe the discrepancies in natural language, and the coding agent turns those observations into targeted source edits. The same loop can run for multiple cycles until the visual result satisfies the original intent.
This is exactly the workflow that coding agents already use for software: write, execute, read the error or test output, patch, repeat. Extending the same pattern to visual output simply requires a renderer that can turn the generated program into an image the agent can inspect.
Concrete rendering engines and the languages that feed them
The practical power of the approach depends on mature, widely available renderers.
SVG is the most accessible entry point. Browsers, design tools, and many libraries can render SVG instantly. Large language models have become remarkably good at producing valid SVG. Independent evaluations and informal benchmarks (including the well-known “pelican riding a bicycle” test popularized by Simon Willison) show steady improvement across frontier models.
Gemini, Claude, and GPT-family models routinely generate coherent icons, simple illustrations, diagrams, and even animated SVGs. Because the output is XML text, the agent can read its own previous code, modify individual paths or groups, and re-render without starting from scratch. Dedicated systems such as StarVector and LLM4SVG train or adapt language/multimodal models specifically for SVG understanding and generation, while hybrid systems such as Chat2SVG combine LLM-generated SVG structure with diffusion guidance and optimization.
HTML and CSS (or React components) turn the browser itself into the rendering engine. An agent can generate a complete visual layout-infographics, posters, UI mockups, process diagrams-and the browser paints it. Live preview environments (Claude Artifacts, various canvas tools, or simple local servers) close the loop so the agent sees the result immediately. Designers and developers have demonstrated end-to-end pipelines in which an AI writes the markup, a headless browser or Playwright captures the screenshot, and the agent refines the code on the basis of visual feedback.
Python ecosystems provide another rich set of renderers. Matplotlib, Seaborn, Plotly, Pillow, and Cairo can produce charts, scientific diagrams, and generative art. Multi-agent systems such as MatPlotAgent, PlotGen, and various chart-generation pipelines decompose the task: one agent plans the visualization, another writes the code, a third executes it and captures errors or visual feedback, and a fourth revises. The same pattern appears in scientific domains where TikZ or CircuitikZ code is generated, compiled with LaTeX, inspected, and corrected until the diagram is publication-ready.
Creative-coding environments such as p5.js and Processing have a long history in generative art. Tools like Spellburst demonstrated early on how an LLM can translate high-level semantic prompts (“stained-glass bouquet of roses”) into executable sketches, then support branching, parameter tweaking via natural-language-driven sliders, and semantic merging of different versions. The artist stays in a continuous exploratory loop rather than wrestling with syntax.
Three-dimensional and simulation-oriented renderers expand the possibilities further. Three.js lets agents build interactive web-based 3D scenes. Blender’s Python API, used by systems such as VIGA, turns the entire modeling and rendering pipeline into a programmable environment. The agent can place geometry, set materials and lights, render from multiple viewpoints, inspect the results with a vision model, and revise the script. Procedural 3D modeling via code is now being systematically benchmarked; the advantage again is that the output is an editable, engine-ready asset rather than a single view.
Shaders, Lottie JSON, and other specialized formats follow the same logic: the model emits a structured description that a specialized player or GPU pipeline executes.
Agentic systems that close the loop
The most advanced demonstrations treat image generation as a multi-stage agentic process rather than a single inference call.
GenClaw, for example, explicitly mirrors a human artist’s workflow. In the conceptualize stage the agent gathers knowledge through search and reasoning. In the sketch stage it writes executable code-SVG for layout, HTML for text-heavy compositions, Three.js or Python for physical constraints-producing a precise intermediate canvas.
In the color stage an image-generation model adds texture and photorealism while respecting the structural sketch. Errors can be traced to a specific stage and corrected without discarding the entire effort. Evaluations on complex scene instruction following, long-text rendering, image editing, and knowledge-intensive generation show advantages over several conventional generators and prompt-rewriting agents.
Similar staged or multi-agent designs appear in scientific visualization, UI generation, and 3D asset creation. A critic vision-language model looks at the rendered output, a coding agent proposes revisions, a judge selects the best candidate, and the cycle repeats. Because the intermediate representation is code, the revisions remain precise and cumulative.
Even simple interactive tools illustrate the principle. A user (or an agent) asks for an SVG icon; the model emits the markup; a browser or design tool renders it; the user requests a change in natural language; the model edits the previous code rather than regenerating from scratch. The same pattern scales to complex scenes when the agent maintains memory of the current source file.
Advantages that matter in practice
Editability is the most obvious benefit. The output is not a flat bitmap; it is a hierarchical, versionable artifact that can be handed to a designer, imported into Figma or Illustrator, integrated into a web application, or further refined by another agent.
Precision and reliability follow. Exact object counts, spatial relations, text content, and geometric constraints become enforceable. This is decisive for diagrams, technical illustrations, data visualizations, logos, and UI components.
Reusability and composition become natural. An agent can generate modular components, combine them, or parameterize them. The same SVG can be styled differently or animated later.
Transparency and debuggability improve. When something goes wrong, the failure can often be localized to a particular block of code rather than remaining an opaque sampling failure.
Integration with existing software stacks is straightforward. The generated code can be committed to a repository, tested, linted, and deployed like any other program.
Remaining challenges
Code generation is not yet perfect. Models still occasionally emit invalid syntax, overly complex path data, or layouts that only work at a specific size. Complex scenes with many overlapping elements or intricate physical interactions remain difficult. Pure code struggles with the rich textures and lighting that diffusion models handle easily, which is why hybrid pipelines are attractive.
Efficiency is another consideration. Writing and iteratively refining code, plus running a renderer and a vision critic, costs more compute and latency than a single diffusion pass. For pure exploratory mood-board work, pixel-native tools remain faster.
Safety and evaluation are still maturing. Because the output is executable, care must be taken with code execution environments. Standardized benchmarks for code-native visual generation are emerging but are not yet as mature as those for text-to-image models.
Looking forward
The trajectory is clear. As language models improve at code generation and as vision-language models become better critics, the Code → Render → Inspect → Revise loop will become more autonomous and more reliable. Hybrid systems that use code for structure and generative models for appearance will grow more seamless. Domain-specific languages and intermediate representations tailored for AI (cleaner SVG dialects, model-friendly animation formats, higher-level scene graphs) will reduce friction.
In design, engineering, scientific communication, education, and generative art, the ability to treat images as programs rather than opaque bitmaps opens new workflows. An AI coding agent can maintain a living visual document, update it when requirements change, and guarantee consistency across scales and media. The human remains in the loop for high-level direction and taste, while the mechanical iteration is handled by the machine.
Code-native generation does not replace pixel-native generation; it complements it. For many of the tasks where structure, editability, and reliability matter most, writing the program that draws the picture is proving to be the more powerful and more controllable path. The rendering engine becomes the shared canvas, the code becomes the editable sketch, and the AI agent becomes a collaborator that can keep improving the work until it is right.
Sources and further reading
- Yoko Li, “The Next Frontier of Visual AI Is Code,” a16z, June 2026. a16z.news
- Junyan Ye et al., “GenClaw: Code-Driven Agentic Image Generation,” arXiv:2605.30248, 2026. arxiv.org and arxiv.org
- StarVector project and paper: joanrod/star-vector and related arXiv works on multimodal SVG generation.
- LLM4SVG: “Empowering LLMs to Understand and Generate Complex Vector Graphics.” ximinng.github.io and arXiv.
- Chat2SVG: Vector Graphics Generation with Large Language Models and Image Diffusion Models, arXiv:2411.16602.
- Spellburst: A Node-based Interface for Exploratory Creative Coding with Natural Language Prompts (Stanford / Replit). spellburstllm.github.io and arXiv:2308.03921.
- Simon Willison’s ongoing SVG generation experiments and “pelican riding a bicycle” benchmark. simonw/pelican-bicycle
- Yupp SVG AI Leaderboard.
- Various agentic chart and scientific diagram systems (MatPlotAgent, PlotGen, TikZ/CircuitikZ agents, etc.) appearing in recent arXiv literature on multi-agent visualization.
- YouTube demonstrations and tutorials: Claude AI SVG generation lessons, HTML/CSS graphics pipelines with AI assistance, MCP-based SVG tools, and agentic infographic pipelines (search terms such as “Claude generate SVG,” “AI HTML CSS graphics,” “Claude Code SVG,” and related titles yield practical walkthroughs of live rendering and iteration).
These sources, together with the broader literature on text-to-SVG, frontend code generation with visual feedback, and procedural 3D modeling via code, form the foundation of the current code-native paradigm.
1 point
Mentions
Code as Canvas: How AI Coding Agents Generate Images Through Executable Scripts and Rendering Engines In the last few years, most conversations about AI image generation have centered on diffusion models and their ability to turn a text prompt into a photorealistic or stylized picture in seconds. Tools built on these systems produce stunning results, but their native representation is typically raster imagery rather than an explicitly structured scene. Although modern models increasingly support masking, inpainting, reference-based editing, and iterative image edits, reliably making exact structural changes-such as enforcing object counts, coordinates, typography, or reusable hierarchy-can remain difficult. Tools built on these systems produce stunning results, but their native representation is typically raster imagery rather than an explicitly structured scene. Although modern models increasingly support masking, inpainting, reference-based editing, and iterative image edits, reliably making exact structural changes-such as enforcing object counts, coordinates, typography, or reusable hierarchy-can remain difficult. A parallel and increasingly powerful approach has been emerging. Instead of asking the model to paint the final pixels directly, researchers and practitioners ask large language models and coding agents to write the program that produces the image. That program might be SVG markup, HTML and CSS, a Python script using Matplotlib or a canvas library, a Three.js scene, a p5.js sketch, a Blender Python script, TikZ code for a scientific diagram, or even a shader. A deterministic rendering engine then turns the code into pixels. Because the intermediate representation is editable source code, an agent (or a human) can inspect the result, diagnose problems, revise specific lines, and re-render. The process becomes a closed loop of Code → Render → Inspect → Revise. This is what people mean by code-native AI image generation. It reframes visual creation as a programming task and leverages the strengths that modern language models already possess: writing correct, structured code and reasoning about spatial and logical relationships. The approach is especially powerful when the agent can see the rendered output (via a vision-language model) and iterate automatically. The fundamental distinction: pixels versus programs Pixel-native systems operate in a continuous latent space. They are excellent at atmosphere, texture, lighting, and photorealism. They struggle with discrete constraints-exactly three apples arranged left-to-right, text that must be legible and correctly spelled, geometric precision, or hierarchical structure that must survive later editing. Code-native systems invert the priority. The model produces an explicit symbolic description. The rendering engine deterministically executes the generated description. When the code correctly expresses the intended constraints, object counts, coordinates, layering, text content, and proportions can therefore be enforced explicitly rather than left to stochastic image sampling. The trade-off is that pure code is weaker at high-frequency detail and photorealistic materials. The most interesting recent systems therefore hybridize the two: the agent uses code to establish the structural skeleton (the “sketch”), then hands the result to a diffusion or image model for texturing and realism (the “coloring”). This hybrid pattern appears repeatedly. An agent writes SVG or HTML that locks composition and typography; an image model then fills in surfaces and lighting while respecting the underlying layout. The code remains the editable source of truth. Why iteration becomes possible The decisive advantage is the feedback loop. In a prompt-only diffusion pipeline, a new attempt is often another stochastic generation, and even image-editing pipelines generally provide less explicit access to the underlying object hierarchy than programmatic representations do. Feedback is global and hard to localize: “the composition feels off” does not map cleanly to a single corrective action. In a code-native pipeline the mapping is local and actionable. If two shapes overlap incorrectly, the agent can change the coordinates or z-order in the SVG. If text overflows a box, it adjusts the font size or container dimensions in CSS. If a physical simulation looks wrong, it revises the parameters of the numerical code that generated the geometry. A vision-language model can look at the rendered image, describe the discrepancies in natural language, and the coding agent turns those observations into targeted source edits. The same loop can run for multiple cycles until the visual result satisfies the original intent. This is exactly the workflow that coding agents already use for software: write, execute, read the error or test output, patch, repeat. Extending the same pattern to visual output simply requires a renderer that can turn the generated program into an image the agent can inspect. Concrete rendering engines and the languages that feed them The practical power of the approach depends on mature, widely available renderers. SVG is the most accessible entry point. Browsers, design tools, and many libraries can render SVG instantly. Large language models have become remarkably good at producing valid SVG. Independent evaluations and informal benchmarks (including the well-known “pelican riding a bicycle” test popularized by Simon Willison) show steady improvement across frontier models. Gemini, Claude, and GPT-family models routinely generate coherent icons, simple illustrations, diagrams, and even animated SVGs. Because the output is XML text, the agent can read its own previous code, modify individual paths or groups, and re-render without starting from scratch. Dedicated systems such as StarVector and LLM4SVG train or adapt language/multimodal models specifically for SVG understanding and generation, while hybrid systems such as Chat2SVG combine LLM-generated SVG structure with diffusion guidance and optimization. HTML and CSS (or React components) turn the browser itself into the rendering engine. An agent can generate a complete visual layout-infographics, posters, UI mockups, process diagrams-and the browser paints it. Live preview environments (Claude Artifacts, various canvas tools, or simple local servers) close the loop so the agent sees the result immediately. Designers and developers have demonstrated end-to-end pipelines in which an AI writes the markup, a headless browser or Playwright captures the screenshot, and the agent refines the code on the basis of visual feedback. Python ecosystems provide another rich set of renderers. Matplotlib, Seaborn, Plotly, Pillow, and Cairo can produce charts, scientific diagrams, and generative art. Multi-agent systems such as MatPlotAgent, PlotGen, and various chart-generation pipelines decompose the task: one agent plans the visualization, another writes the code, a third executes it and captures errors or visual feedback, and a fourth revises. The same pattern appears in scientific domains where TikZ or CircuitikZ code is generated, compiled with LaTeX, inspected, and corrected until the diagram is publication-ready. Creative-coding environments such as p5.js and Processing have a long history in generative art. Tools like Spellburst demonstrated early on how an LLM can translate high-level semantic prompts (“stained-glass bouquet of roses”) into executable sketches, then support branching, parameter tweaking via natural-language-driven sliders, and semantic merging of different versions. The artist stays in a continuous exploratory loop rather than wrestling with syntax. Three-dimensional and simulation-oriented renderers expand the possibilities further. Three.js lets agents build interactive web-based 3D scenes. Blender’s Python API, used by systems such as VIGA, turns the entire modeling and rendering pipeline into a programmable environment. The agent can place geometry, set materials and lights, render from multiple viewpoints, inspect the results with a vision model, and revise the script. Procedural 3D modeling via code is now being systematically benchmarked; the advantage again is that the output is an editable, engine-ready asset rather than a single view. Shaders, Lottie JSON, and other specialized formats follow the same logic: the model emits a structured description that a specialized player or GPU pipeline executes. Agentic systems that close the loop The most advanced demonstrations treat image generation as a multi-stage agentic process rather than a single inference call. GenClaw, for example, explicitly mirrors a human artist’s workflow. In the conceptualize stage the agent gathers knowledge through search and reasoning. In the sketch stage it writes executable code-SVG for layout, HTML for text-heavy compositions, Three.js or Python for physical constraints-producing a precise intermediate canvas. In the color stage an image-generation model adds texture and photorealism while respecting the structural sketch. Errors can be traced to a specific stage and corrected without discarding the entire effort. Evaluations on complex scene instruction following, long-text rendering, image editing, and knowledge-intensive generation show advantages over several conventional generators and prompt-rewriting agents. Similar staged or multi-agent designs appear in scientific visualization, UI generation, and 3D asset creation. A critic vision-language model looks at the rendered output, a coding agent proposes revisions, a judge selects the best candidate, and the cycle repeats. Because the intermediate representation is code, the revisions remain precise and cumulative. Even simple interactive tools illustrate the principle. A user (or an agent) asks for an SVG icon; the model emits the markup; a browser or design tool renders it; the user requests a change in natural language; the model edits the previous code rather than regenerating from scratch. The same pattern scales to complex scenes when the agent maintains memory of the current source file. Advantages that matter in practice Editability is the most obvious benefit. The output is not a flat bitmap; it is a hierarchical, versionable artifact that can be handed to a designer, imported into Figma or Illustrator, integrated into a web application, or further refined by another agent. Precision and reliability follow. Exact object counts, spatial relations, text content, and geometric constraints become enforceable. This is decisive for diagrams, technical illustrations, data visualizations, logos, and UI components. Reusability and composition become natural. An agent can generate modular components, combine them, or parameterize them. The same SVG can be styled differently or animated later. Transparency and debuggability improve. When something goes wrong, the failure can often be localized to a particular block of code rather than remaining an opaque sampling failure. Integration with existing software stacks is straightforward. The generated code can be committed to a repository, tested, linted, and deployed like any other program. Remaining challenges Code generation is not yet perfect. Models still occasionally emit invalid syntax, overly complex path data, or layouts that only work at a specific size. Complex scenes with many overlapping elements or intricate physical interactions remain difficult. Pure code struggles with the rich textures and lighting that diffusion models handle easily, which is why hybrid pipelines are attractive. Efficiency is another consideration. Writing and iteratively refining code, plus running a renderer and a vision critic, costs more compute and latency than a single diffusion pass. For pure exploratory mood-board work, pixel-native tools remain faster. Safety and evaluation are still maturing. Because the output is executable, care must be taken with code execution environments. Standardized benchmarks for code-native visual generation are emerging but are not yet as mature as those for text-to-image models. Looking forward The trajectory is clear. As language models improve at code generation and as vision-language models become better critics, the Code → Render → Inspect → Revise loop will become more autonomous and more reliable. Hybrid systems that use code for structure and generative models for appearance will grow more seamless. Domain-specific languages and intermediate representations tailored for AI (cleaner SVG dialects, model-friendly animation formats, higher-level scene graphs) will reduce friction. In design, engineering, scientific communication, education, and generative art, the ability to treat images as programs rather than opaque bitmaps opens new workflows. An AI coding agent can maintain a living visual document, update it when requirements change, and guarantee consistency across scales and media. The human remains in the loop for high-level direction and taste, while the mechanical iteration is handled by the machine. Code-native generation does not replace pixel-native generation; it complements it. For many of the tasks where structure, editability, and reliability matter most, writing the program that draws the picture is proving to be the more powerful and more controllable path. The rendering engine becomes the shared canvas, the code becomes the editable sketch, and the AI agent becomes a collaborator that can keep improving the work until it is right. Sources and further reading - Yoko Li, “The Next Frontier of Visual AI Is Code,” a16z, June 2026. a16z.news - Junyan Ye et al., “GenClaw: Code-Driven Agentic Image Generation,” arXiv:2605.30248, 2026. arxiv.org and arxiv.org - StarVector project and paper: joanrod/star-vector and related arXiv works on multimodal SVG generation. - LLM4SVG: “Empowering LLMs to Understand and Generate Complex Vector Graphics.” ximinng.github.io and arXiv. - Chat2SVG: Vector Graphics Generation with Large Language Models and Image Diffusion Models, arXiv:2411.16602. - Spellburst: A Node-based Interface for Exploratory Creative Coding with Natural Language Prompts (Stanford / Replit). spellburstllm.github.io and arXiv:2308.03921. - Simon Willison’s ongoing SVG generation experiments and “pelican riding a bicycle” benchmark. simonw/pelican-bicycle - Yupp SVG AI Leaderboard. - Various agentic chart and scientific diagram systems (MatPlotAgent, PlotGen, TikZ/CircuitikZ agents, etc.) appearing in recent arXiv literature on multi-agent visualization. - YouTube demonstrations and tutorials: Claude AI SVG generation lessons, HTML/CSS graphics pipelines with AI assistance, MCP-based SVG tools, and agentic infographic pipelines (search terms such as “Claude generate SVG,” “AI HTML CSS graphics,” “Claude Code SVG,” and related titles yield practical walkthroughs of live rendering and iteration). These sources, together with the broader literature on text-to-SVG, frontend code generation with visual feedback, and procedural 3D modeling via code, form the foundation of the current code-native paradigm.