On Highlighting Code

Posted on September 15, 2026 by @jexia_

(CLICK TO HIGHLIGHT ART – SEIZURE WARNING)

Today, we’re excited to announce the public release of a new library we call Highlights.

Highlights is a syntax highlighter written in WebAssembly Text, or WAT. It supports over 70 built-in languages, uses Zed-compatible themes, and produces HTML or themed tokens. It also handles streaming input and incremental edits, so applications can keep highlighting as their code changes.

Highlights generates HTML at up to 298× Shiki’s throughput across the large-file fixtures below.

Median throughput after warmup, relative to Shiki’s 1× baseline. Measured October 5, 2026 on an Apple M4 Pro (14 cores, 48 GiB RAM) running macOS 27.0.1, using Bun 1.4.0 and Shiki 4.4.1. These figures match the Highlights page. See the benchmark methodology. Results vary by input and environment.

Background

Syntax highlighting is surprisingly expensive.

Historically in Diffs we’ve used the excellent Shiki library and then relied on workers to help keep this work off the main thread, and caches to help avoid repeating it.

This has been fine, but particularly as we’ve invested more into our edit functionality, syntax highlighting has become more and more an unavoidable bottleneck for us.

How mainstream code highlighters work

In const greeting = "hello";, a highlighter identifies const as a keyword and "hello" as a string. A theme supplies their colors. The output can be tokens for a renderer or HTML with styled spans. Most of the work is in finding those tokens.

Regular expressions

regex is the death of performance. - @aidenybai

Shiki, Prism, Sugar High, and Rangi all use regular expressions to match tokens. Regex gives them a compact way to describe what a keyword, number, or quoted string looks like. In engines such as Prism, adding a language means defining those patterns and letting the same engine do the matching. That makes language support easier to build and extend.

The cost comes from how matching happens. Different rules can scan the same stretch of code, each looking for its own pattern. Complex expressions can also backtrack: when a match fails, the engine revisits characters and tries another path. A single regex can be fast, but applying a whole grammar means paying for successful matches, failed attempts, and repeated scans. Across large files or code highlighted after every edit, that work adds up.

Syntax trees

Tree-sitter takes another approach. It parses source code into a concrete syntax tree, which retains details such as punctuation, then uses queries to assign highlight names to nodes in that tree. A theme maps those names to styles.

That structure lets highlighting rules distinguish a function name from a parameter by its place in the code. Building, storing, and querying the tree takes time and memory, but the tree can also support features such as code navigation. After an edit, Tree-sitter can update it incrementally and reuse unchanged parts. A full-file benchmark measures a different workload from maintaining a tree while someone types.

A learned lexer on the GPU

Somewhat recently, Shu Ding introduced gpu-lexer from Vercel Labs, an experimental highlighter that uses a small learned model running on WebGPU. One model handles different languages without requiring a language argument or a separate grammar for each one.

The CPU first splits the source into simple parts, such as words, whitespace, and symbols. The GPU processes features from those parts and their context, then predicts one of nine visual classes.

The learned approach is interesting because it can attempt to highlight a language it has never been explicitly configured to support. Its quality depends on what the model learned and how well that transfers to the input. The model card explains that evaluation measures agreement with Shiki's labels, rather than semantic correctness.

The chart shows median times from three passes, with one warm-up in a fresh worker for each sample; initialization and DOM rendering are excluded. Highlights and Shiki use codeToTokens(), returning themed JavaScript tokens. gpu-lexer returns classified spans; Starry Night returns a HAST tree; Sugar High, Prism.js, and Highlight.js return HTML. The recorded samples include versions and run conditions. This measures time for those API paths on one large JavaScript input; it does not measure highlighting quality.

How @pierre/highlights works

The core pipeline is fairly small:

JavaScript handles the public API, input encoding, and theme preparation. The WebAssembly module scans the source and emits HTML or token records. There is no syntax tree to construct along the way. The architecture notes describe the boundaries in more detail.

Give each language a lexer

A lexer walks through source text and identifies meaningful ranges: a keyword, a number, a comment, a string. In Highlights, these lexers are written by hand in WAT and share scanners for common operations.

More complicated languages need more state. Embedded languages also need a way to hand control to another lexer. An HTML <script> region, for example, can be scanned by the JavaScript lexer within the bounds of that region, then return control to HTML.

This makes language support explicit. Adding a language means implementing and maintaining its lexer. Existing TextMate grammars cannot simply be loaded into Highlights, and the output will not always classify code exactly as Shiki or Tree-sitter would.

Scan more bytes at once

A lot of source code consists of stretches where nothing interesting happens. Inside a line comment, the lexer mostly needs to find the next line ending. Inside a quoted string, it can skip ahead until it encounters a quote, an escape, or another character that needs special handling.

Highlights uses WebAssembly SIMD for several of these scans. SIMD lets one instruction operate on multiple values at once; here, a scanner can load 16 bytes and compare all of them against the characters it is looking for. The comparisons produce a bit mask, which identifies the first matching position.

That lets the scanner advance through a long comment 16 bytes at a time. Short, common cases still matter: whitespace scanning has a scalar path for the small gaps between tokens, avoiding SIMD setup when there is little to scan.

Keyword lookup also uses tables prepared during the build. The lookup hashes selected bytes and the word length, then checks the candidate's exact bytes. Language keywords become data the lexer can look up directly.

Emit the output while scanning

For HTML, the emitter writes escaped text and styled spans directly into WebAssembly memory. JavaScript prepares the theme ahead of time; ordinary color themes become a compact table containing color and font settings for each token type.

Neighboring ranges with the same style can share a span, including the whitespace between them. Span opening tags are cached for reuse. These details reduce both the work of producing HTML and the amount of markup the browser receives.

The API exposes that output as UTF-8 bytes:

import { codeToHtml } from '@pierre/highlights';
import { pierreDark } from '@pierre/highlights/themes';

const bytes = codeToHtml('const greeting = "hello";', {
  lang: 'js',
  theme: pierreDark,
});

const html = new TextDecoder().decode(bytes);

Highlights accepts Zed-compatible theme objects. You can import a bundled theme, such as pierreDark or githubLight, from @pierre/highlights/themes, or supply your own.

Streaming and incremental editing

Highlighting a complete file quickly helps, but live interfaces also need to handle text that is still arriving or being edited.

StreamTokenizer accepts chunks of text or UTF-8 bytes and emits themed tokens for completed lines. It preserves lexer state between chunks, so a comment or string can continue into the next chunk without rescanning earlier lines. An incomplete final line stays buffered until a newline arrives or the caller finishes the stream with end().

import { StreamTokenizer } from '@pierre/highlights';
import { pierreDark } from '@pierre/highlights/themes';

const response = await fetch('https://example.com/source.ts');
const lines = response.body.pipeThrough(
  new StreamTokenizer({ lang: 'ts', theme: pierreDark })
);

for await (const tokens of lines) {
  console.log(tokens);
}

LiveTokenizer handles edits to existing text. It keeps the document, per-line token records, and saved lexer states in its own WebAssembly instance. After an edit, it tokenizes the affected lines and continues until the outgoing state matches the previously saved state at an unchanged boundary. Beyond that point, the existing results can be reused.

import { LiveTokenizer } from '@pierre/highlights';
import { pierreDark } from '@pierre/highlights/themes';

const live = new LiveTokenizer({
  code: 'let count = 1;\nconsole.log(count);',
  lang: 'js',
  theme: pierreDark,
});

live.applyEdits([{
  range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } },
  newText: '2',
}]);

console.log(live.getLineText(0)); // let count = 2;
console.log(live.getLineTokens(0).tokens);

live.dispose();

Why we chose WAT

WAT is the text representation of WebAssembly instructions. Writing it by hand means working directly with loads, stores, branches, and vector operations. Which isn't the most fun…

Rust and Zig can both compile to WebAssembly and provide control over memory and SIMD.

However, our preference for WAT is about the workflow: the source expresses the loads, comparisons, and branches we want to inspect and tune.

For example, the 16-byte comment scan is written directly as a vector load, comparisons, and a bit-mask check. With a program built mostly from scans like this, working in that instruction representation is useful. The choice also gives us more details to maintain ourselves.

WAT preprocessing

Writing every table and constant by hand would make the lexers harder to maintain. Our preprocessor adds a few forms to the source and expands them into standard WAT during the build.

Local imports bring shared code into one module, including each file once. Enums give token types numeric IDs, and named constants let scanners refer to memory addresses without repeating raw offsets.

(import "./langs/tsx.wat")

WABT assembles the expanded WAT into WebAssembly, then Binaryen optimizes it. These conveniences are resolved during the build, so the runtime still scans bytes with ordinary WebAssembly instructions.

Size control

The scanners, keyword tables, language dispatch, and emitters all live in one module. Shared operations can be reused across lexers, and the build can control how tables and constants are represented.

One small example is a vector filled with the same byte. A full vector constant takes more space to encode than loading one small integer and repeating it across the vector. The build rewrites these repeated-byte constants after optimization to recover those bytes.

The bundle comparison below records Highlights 0.0.3 with 67 languages, measured in September 2026. It includes the JavaScript runtime and embedded Wasm, uses Brotli compression, and excludes theme assets. These are measurements of that earlier release.

This chart measures minified browser bundles compressed with Brotli, including the JavaScript runtime and selected languages, and excluding themes. Highlights 0.0.3 included all public runtime exports and its embedded Wasm with 67 languages: 83,584 bytes, or 81.6 KiB. The chart retains the source's KB/MB labels, where KB means 1,024 bytes. Competitor figures come from the gpu-lexer size comparison, recorded September 9; Highlights was measured September 16. The size data records the coverage and build settings.

Memory control

The module uses WebAssembly's linear memory, a growable buffer addressed by byte offsets. Highlights assigns space for control data, tables, lexer state, input, and output. Strings can be encoded directly into the input region, and the emitter writes into an output region that grows as needed.

This lets the HTML path process token ranges without allocating a JavaScript object for every token. The live tokenizer has a separate heap for document text, line records, and saved states, which it can retain across edits.

The ownership rules are visible in the API. codeToHtml() returns bytes backed by reusable memory: decode them immediately, as in the example above, or copy them with .slice() before the next highlighting or tokenization call. Requesting themed JavaScript tokens creates objects, while consumers of raw records can defer that work.

The following comparison measures median peak process memory across five runs while highlighting a 517 KiB TypeScript file and generating HTML.

Measured October 5, 2026 in the same environment as the HTML generation chart, using five fresh processes per engine. RSS includes the runtime, compiled code, Wasm, and allocator capacity; it is not live heap or bundle size. See the memory benchmark methodology.

The cost of that control

Writing WAT also makes the project responsible for details a higher-level language would normally help manage: buffer bounds, scratch space, state layout, and the behavior of every lexer on incomplete input.

Runtime differences still matter. The build includes JavaScriptCore workarounds around SIMD calling conventions and certain integer expressions. These are covered by tests, but they are a reminder that working closer to WebAssembly does not remove the surrounding engine from the picture.

The design fits a program built around compact state and repeated scans over bytes. As the implementation grows, keeping those boundaries understandable is part of the maintenance cost.

Fin

Highlights makes the processing behind syntax highlighting small enough to use in places where code keeps arriving and changing. Its lexers scan bytes directly, its emitters build compact output, and its streaming and live APIs preserve work that can be reused.

There are trade-offs. Handwritten lexers require ongoing language work. TextMate grammars have a much broader ecosystem, and applications that need syntax trees benefit from having that structure available. Faster highlighting also leaves the browser's layout, painting, and DOM costs to deal with.

For applications that need highlighted HTML or tokens, Highlights offers a small engine with explicit control over how much work happens and when. You can install it with pnpm add @pierre/highlights, read the docs, or explore the implementation.