HTML-in-Canvas

We propose new HTML Canvas APIs for rendering HTML content into the canvas for Canvas 2D and WebGL.

Status

This proposal is a subset of a previous proposal covering APIs to allow live HTML elements.

Motivation

There is no web API to easily render complex layouts of text and other content into a . As a result, -based content suffers in accessibilty, internationalization, performance and quality.

Use cases

• Styled, Laid Out Content in Canvas. There’s a strong need for better styled text support in Canvas. Examples include chart components (legend, axes, etc.), rich content boxes in creative tools, and in-game menus.
• Accessibility Improvements. There is currently no guarantee that the canvas fallback content currently used for accessibility always matches the rendered content, and such fallback content can be hard to generate. With this API, elements drawn into the canvas bitmap will match their corresponding canvas fallback.
• Composing HTML Elements with Shaders. A limited set of CSS shaders, such as filter effects, are already available, but there is a desire to use general WebGL shaders with HTML.
• HTML Rendering in a 3D Context. 3D aspects of sites and games need to render rich 2D content into surfaces within a 3D scene.

Proposed solution: layoutsubtree, drawElement, texElement2D and setHitTestRegions

• the layoutsubtree attribute on a element allows its descendant elements to have layout (*), and causes the direct children of the to have a stacking context and become a containing block for all descendants. Descendant elements of the still do not paint or hit-test, and are not discovered by UA algorithms like find-in-page.
• The CanvasRenderingContext2D.drawElement(element, x, y, options) method renders element and its subtree into a 2D canvas at offset x and y, so long as element is a direct child of the . It has no effect if layoutsubtree is not specified on the . The options dictionary, if given, has a single option that preserves user privacy in the drawn content, allowing readback or use in WebGL.
• The WebGLRenderingContext.texElement2D(..., element) method renders element into a WebGL texture. It has no effect if layoutsubtree is not specified on the .
• The CanvasRenderingContext2D.setHitTestRegions([{element: ., rect: {x: x, y: y, width: ..., height: ...}, ...]) (and WebGLRenderingContext.setHitTestRegions(...)) API takes a list of elements and -relative rects indicating where the element paints relative to the backing buffer of the canvas. These rects are then used to redirect hit tests for mouse and touch events automatically from the element to the drawn element.

(*) Without layoutsubtree, geometry APIs such as getBoundingClientRect() on these elements return an empty rect. They do have computed styles, however, and are keyboard-focusable.

drawElement(element ...) takes the CTM (current transform matrix) of the canvas into consideration. The image drawn into the canvas is sized to element's border box size; element outsize that bounds (including ink and layout overflow) are clipped. The drawElement(element, x, y, dwidth, dheight) variant resizes the image of element's subtree to dwidth and dheight.

In an addition, a fireOnEveryPaint option is added to ResizeObserverOptions, allowing script to be notified whenever the drawn elements might have changed their DOM state and the canvas should be redrawn. The callback to the resize obsever will be called at resize obsever timing, which is after DOM style and layout, but before paint.

The same element may be drawn multiple times.

Once drawn, the resulting canvas image is static. Subsequent changes to the element will not be reflected in the canvas, so the element must be explicitly redrawn if an author wishes to see the changes.

The descendant elements of the are considered fallback content used to provide accessibility information. See Issue#11 for an ongoing discussion of accessibility concerns.

Offscreen canvas contexts and detached canvases are not supported because drawing DOM content when the canvas is not in the DOM poses technical challenges. See Issue#2 for further discussion.

NOTE: The current implementation of drawElement() and texElement2D does not taint the canvas and is not suitable for use outside of local demos. When using this feature in a DevTrial, take steps to avoid leaking private information. When the work for privacy preserving painting is complete the allowReadback option must be set to true when an untainted canvas is required. DevTrial users may wish to start using this option now to avoid disruption when tainting is enabled.

interface CanvasRenderingContext2D {

  ...

  dictionary Canvas2DDrawElementOption {
    boolean allowReadback = false;
  };

  [RaisesException]
  void drawElement(Element element, unrestricted double x, unrestricted double y,
                   optional Canvas2DDrawElementOption options = {});

  [RaisesException]
  void drawElement(Element element, unrestricted double x, unrestricted double y,
                   unrestricted double dwidth, unrestricted double dheight,
                   optional Canvas2DDrawElementOption options = {});

interface WebGLRenderingContext {

  ...

  [RaisesException]
    void texElement2D(GLenum target, GLint level, GLint internalformat,
                      GLenum format, GLenum type, Element element);

Demos

See here to see an example of how to use the API. It should render like the following (the blue rectangle indicates the bounds of the , and the black the element passed to

drawElement). It draws like this:

See here for an example of how to use the WebGL texElement2D API to populate a GL texture with HTML content.

The example should render an animated cube, like in the following shapshot. Note how the border box fills the entire face of the cube. To adjust that, modify the texture coordinates for rendering the cube and possibly adjust the texture wrap parameters. Or, wrap the content in a larger