The Browser Beyond Pages

Almost all the attention in software is currently focused on large language models. Every few weeks, a new model is released and the discussion starts again. But another important change is happening at the same time: browsers are becoming capable of running much more complex graphical software.

To understand why this matters, we need to go back to one of the oldest architectural boundaries on the web: the boundary between the DOM and Canvas.

Two Different Models for Building User Interfaces

When a browser displays a normal web page, it converts the HTML into a structure called the DOM. Every button, piece of text, image, and form input becomes a node in this tree.

This model gives us several important capabilities. The browser understands the structure of the page. It makes text selectable, manages user input, and provides the information required by screen readers, search engines, and browser extensions.

But the DOM is not the right model for every type of software.

When building a graphical editor, an interactive map, a game, or an application with thousands of moving objects, representing every object as a separate DOM element can become limiting. This is especially true when many of those elements need to change on every frame or when their positions depend on layout calculations. Continuous changes can cause the browser to repeat style calculations, layout, painting, and compositing work.

This does not mean that the DOM is inherently slow. Thousands of static elements may work without any issue, while a much smaller number of elements with constant updates can become expensive.

This is where Canvas comes in. Canvas gives the application a drawing surface. Instead of creating a DOM element for every shape, we use JavaScript to draw lines, images, text, and shapes directly onto that surface.

For example, drawing a circle with the Canvas 2D API looks roughly like this:

const canvas = document.querySelector('canvas')
const context = canvas.getContext('2d')
 
context.beginPath()
context.arc(150, 75, 50, 0, Math.PI * 2)
context.fillStyle = '#3498db'
context.fill()

That freedom comes with a cost. Canvas has no built-in concept of objects, selection, resize handles, layers, event delegation, or history. If we want users to select, move, or resize a circle, we need to build much of that system ourselves.

I experienced this problem directly. It was one of the main reasons I built Pikaso, an open-source library that turns the low-level Canvas API into a more manageable, object-based model. In Pikaso, the same circle can be created like this:

pikaso.shapes.circle.insert({
  radius: 100,
  x: 100,
  y: 150,
  fill: '#3498db',
})

Libraries such as Pikaso and Konva solve part of the Canvas problem. They build an object model on top of its pixel-based drawing surface and provide capabilities such as selection, transformations, events, and serialization. But a more fundamental problem remains.

The Browser Does Not Understand the Objects Inside Canvas

Our application may know that a Canvas contains a button, a text field, a chart, and several independent shapes. But from the document's point of view, those objects are not separate DOM elements. The browser does not understand the application's internal object model.

As a result, text drawn inside Canvas is not normally selectable like HTML text. Screen readers cannot inspect the application's internal scene graph, and browser features such as find-in-page, autofill, and standard form controls cannot directly interact with those objects.

For years, graphical web applications have had to choose between two different worlds. With the DOM, much of the browser’s built-in behavior comes for free. With Canvas, we give up some of that in exchange for more control over rendering.

An experimental proposal called HTML-in-Canvas is trying to reduce the distance between them.

Bringing HTML into Canvas

Until now, the common solution has been to place HTML elements on top of Canvas, rather than bringing them into its rendering process. For example, a design tool may render its main scene using Canvas, but use regular HTML elements for inputs, menus, and tooltips. These elements are positioned over the Canvas using CSS.

This works for simple interfaces. It becomes much harder when the scene moves, zooms, rotates, or uses perspective. The application needs to keep the position of every HTML element synchronized with the objects inside Canvas. If we want to apply a shader to the interface or make it part of a WebGL or WebGPU scene, this approach quickly becomes limiting.

HTML-in-Canvas takes a different approach. In the current WICG proposal, a <canvas> can opt its HTML children into layout and hit testing with the layoutsubtree attribute:

<canvas id="canvas" layoutsubtree>
  <article id="card">
    <h2>Modern apartment in Lisbon</h2>
    <p>Three bedrooms · 148 m²</p>
    <button>View property</button>
  </article>
</canvas>

The browser still calculates the layout of this element as normal HTML. The application can then draw a rendered snapshot of it into the Canvas:

const canvas = document.querySelector('#canvas')
const card = document.querySelector('#card')
const context = canvas.getContext('2d')
 
canvas.onpaint = () => {
  context.reset()
 
  const transform = context.drawElementImage(card, 40, 40)
  card.style.transform = transform.toString()
}

Calling drawElementImage() draws a snapshot of the HTML element into the Canvas and returns a transform. The application applies that transform to the original element so its DOM location stays aligned with where it was drawn. This alignment matters for hit testing, interaction, and accessibility.

The pixels inside Canvas do not suddenly become real buttons or inputs. The original HTML element still exists in the DOM, and the browser can continue using it to provide normal web capabilities.

The API also adds a paint event to the canvas. When the rendering of one of its HTML children changes, this event runs and gives the application an opportunity to redraw the scene. For updates that need another paint even when no child changed, the proposal adds canvas.requestPaint(). So this is not simply a way to take a one-time screenshot of an HTML element. The goal is to maintain a relationship between the DOM version of the element and its graphical representation inside Canvas.

From Canvas 2D to WebGPU

The importance of this proposal is not limited to Canvas 2D. The current API design suggests similar paths for WebGL and WebGPU. This means rendered HTML could become a texture inside a graphics pipeline. For WebGL, the proposal adds texElementImage2D(). For WebGPU, it adds copyElementImageToTexture() to GPUQueue.

This would allow an interface built with HTML and CSS to appear on the surface of a 3D object or be affected by shaders and graphical effects. For example, a settings panel could be built with regular HTML while being displayed as part of a 3D scene. Developers would not need to rebuild text rendering, layout, and input controls inside a custom graphics engine.

The browser could continue handling the things it already understands well:

• layout and typography
• right-to-left and left-to-right languages
• form controls
• keyboard navigation
• accessibility
• text selection
• features such as autofill and find-in-page

Canvas, WebGL, and WebGPU would still provide greater control over the final composition:

• free movement, rotation, and scaling
• shaders
• placing interfaces in three-dimensional space
• combining UI with images, video, and graphical objects
• managing many objects inside a graphics pipeline

HTML-in-Canvas is not trying to replace the DOM with Canvas. It is trying to reduce the cost of choosing between these two rendering models.

Which Products Could Benefit from This?

For regular websites, articles, online stores, forms, and most dashboards, the DOM remains the simpler and more appropriate choice. Using Canvas without a real need would only increase complexity.

But some applications genuinely need to combine browser semantics with the control of a graphics pipeline:

• graphical editors
• 3D design tools
• video editors
• maps and spatial environments
• product configurators
• web games
• simulations and educational tools
• interface design tools

In a video editor, for example, the scene preview might be rendered with Canvas or WebGPU, while some of the objects inside it are real HTML forms, text, or components. In a 3D design tool, an HTML panel could be placed inside the scene and transformed or affected by a shader while maintaining its connection to the DOM.

Of course, this does not mean that all the difficult parts have been solved. Interaction with inputs under complex transformations, focus management, scrolling, performance in large scenes, and security restrictions are still active design and experimentation problems.

This Is Not a Production-Ready Standard Yet

HTML-in-Canvas is still experimental. The HTML-in-Canvas Origin Trial runs in Chrome 148 through 150. It should not be treated as a stable standard or as a feature ready for broad production use.

There are still several important questions:

• How expensive will HTML snapshots and repaints be?
• How will it perform in large scenes?
• How will interaction behave under complex transformations?
• What security restrictions will be required for cross-origin content?
• How will developer tools debug these two representations?
• Will Firefox and Safari agree with this model?
• What abstractions will frameworks build on top of it?

The current proposal also has specific limitations. For example, the element must have been a direct child of the Canvas during the latest rendering update, and it must generate an actual box. Details like these may change as the proposal develops.

The important question is therefore not whether this exact API, with this exact name and shape, will succeed. What matters is the problem it is trying to solve.

What Is the Browser Becoming?

In the early years of the web, the browser was mainly a tool for displaying documents. JavaScript turned it into an application runtime. WebGL and WebGPU expanded its graphical capabilities. WebAssembly made it possible to move more complex software and engines to the web. APIs such as WebCodecs, File System Access, and WebXR have given web applications access to more parts of the underlying system.

HTML-in-Canvas can be seen as another step in the same direction. Web applications are becoming more graphical, interactive, and complex. The hard boundary between document layout and programmable graphics no longer fits the requirements of every application we want to build.

The browser has already evolved from a document viewer into an application runtime. Now the gap between its document and graphics models is starting to get smaller.

HTML-in-Canvas may become a standard in its current form. It may change significantly, or it may eventually be replaced by a different API. But the need for a better connection between the DOM and programmable graphics will not disappear.

Where AI Fits

AI is reducing the cost of producing code, content, designs, animations, and interfaces. But everything a model generates still needs somewhere to run. A model may be able to generate an interface for a specific user goal, but the browser still needs to display it in a way that is fast, interactive, and accessible.

If that experience includes visualization, video, three-dimensional environments, or heavy graphical work, a traditional DOM interface may not be enough for every part of it. In these cases, HTML-in-Canvas could become one useful primitive, not a complete solution.

HTML could be used for the semantic and interactive parts. WebGPU could render the visualization or 3D environment. APIs such as HTML-in-Canvas could help compose those two worlds into a single experience.

AI may make more of these applications practical to build. But whether they can live comfortably on the web still depends on what the browser can actually run.

The browser is no longer just a place for displaying pages. It is becoming a serious runtime for graphical and interactive software.