Seven rounds of whack-a-mole: sandboxing Go's text/template

Dispatch formats webhooks into Discord embeds, Slack Block Kit, and Telegram HTML using Go's text/template. Until recently, the template editor's live preview was a client-side reimplementation of that engine in TypeScript — and it drifted. No index support, no .Meta, whole platforms missing. Templates that previewed fine failed at delivery, and vice versa.

So we shipped the obvious fix: preview now renders server-side with the real delivery engine, real per-platform function maps and all. What you see in the editor is what your Discord channel gets.

The catch: that means executing tenant-supplied templates, synchronously, inside the API process — the same process that hosts our delivery workers. A runaway render doesn't just break someone's preview. It starves webhook delivery for everyone.

This post is about what it took to make that safe, and specifically about the architectural mistake we made seven times before fixing it.

The plan that didn't work

text/template is a real language: variables, conditionals, loops, function calls, scope. Our first plan treated it like one you could domesticate — keep the whole language, find the dangerous constructs, and bound each of them.

Every round of that plan found a sibling of the construct we'd just patched:

1. print concatenates unbounded output, so we capped it. The attacker's next move is html, js, or urlquery — same amplification, different name.
2. We bounded those. Next: {{range 2000000000}}. Since Go 1.22, text/template ranges over plain integers. No slice required.
3. We bounded literal integer ranges. Next: the same integer, but arriving through a variable, where a literal check can't see it.
4. We taught the walker about variables. Next: {{$n = ...}} — assignment, not declaration — which mutates an enclosing scope, so the value at a range's subject position no longer matches anything you concluded from reading in source order.
5. We handled assignments. Next: {{if ($n = ...)}} — the parser hangs the assignment flags on a nested pipe node, not the outer one we were inspecting.
6. We walked nested pipes. Next: {{with}}, which rebinds dot itself — a naming channel with no identifier attached, so there was no variable name left to taint.
7. We built a dot-tainting apparatus to track that too.

At that point the lesson was hard to miss: a bound only covers what someone thought of. The language has more channels than we had reviewers, and Go keeps adding constructs (see: ranging over integers). Enumerate-and-bound is a race you re-lose on every stdlib release.

The inversion

The durable fix inverted the default. Preview no longer permits the language minus known hazards; it accepts an explicit, named subset and refuses everything else — including constructs that don't exist yet.

// The full set of builtins a preview template may call.
var allowedPreviewBuiltins = map[string]bool{
	"index": true, "slice": true, "len": true, "printf": true, "urlquery": true,
	"eq": true, "ne": true, "lt": true, "le": true, "gt": true, "ge": true,
	"and": true, "or": true, "not": true,
}

And at the bottom of the AST walker:

// Anything the allowlist does not name is refused. This is the whole point of
// the inversion: a construct nobody considered — including one a future Go
// release adds — fails closed instead of executing unbounded.
return w.refuse(fmt.Sprintf("node:%T", node), errNodeNotAllowed)

Delivery still runs the full language — delivery templates are bounded by the pipeline's own budget and retry machinery. Only preview, the surface that executes on user keystrokes, is subset.

The satisfying part: the inversion deleted code. The entire dot-tainting apparatus from round seven went away, because {{with}}'s subject is now gated by the same predicate {{range}} uses. The rebound dot is bounded by construction, so nothing needs to track it.

What the inversion didn't fix

It would make a tidier post to stop here. But the allowlist didn't end the bug hunt — it changed what the bugs looked like. The sibling-construct family died; adversarial review after the inversion found a different genus: bugs in the allowlist's own enforcement. Four are worth pattern-matching against.

A guard that checks one position when the stdlib accepts three. printf stayed allowlisted, so format strings had their own guard, which refused explicit argument indexes like %[1]s — a one-byte re-print multiplier. But fmt parses an argument index in three places: after the flags, after the width, and after the precision. Our guard checked the first. %1[1]s — a single width digit — sailed through and restored the exact multiplier we'd refused, measured through the real HTTP route at 7.5 GB of peak heap on a request that returned HTTP 200.

An arm that never consults its base. The walker validated field chains by length, never looking at what the chain hung off. So {{print.name}} passed: print is refused as a bare name, but wrapped in a field access it's a different AST node, and our arm never recursed into it. A refused function, laundered through a dot. It wasn't exploitable that day — every launder target errors when called with zero arguments — but it was one zero-argument helper away from live.

An implicit bound nobody wrote down. The static work budget assumed every loop level iterates at most one container's worth of entries — 100, our truncation cap. But index applied to a string returns a uint8, and ranging an integer iterates that many times. So a one-character field could legally iterate up to 255 times per level, blowing past the budget's silent assumption and compounding when nested. We fixed it with an explicit per-level byte bound of 256 — the type's bound, deliberately not the 244 we measured, because the measured number was an artifact of JSON UTF-8 sanitization, and modeling observed behavior instead of type guarantees is exactly the reasoning that produced rounds one through seven.

A rollout flag that widened the blast radius of the thing it de-risked. We added a log-only mode for the allowlist — refusals log but don't fail — promising ourselves that resource bounds stayed enforced either way. They didn't. One refusal path returns early from the pipe walker; making it non-fatal skipped validation of everything after it in the pipe. Five parentheses were enough to smuggle a variable assignment past a refusal we'd documented as unconditional. The flag wasn't fixed; it was deleted. A switch that silently widens the blast radius of the thing it's meant to de-risk is worse than no switch.

The data is part of the language

The nastiest bypass never touched a template construct at all.

Go's template executor resolves methods on your data before anything else — before struct fields, before map keys, and entirely outside the function map you control. Funcs() never sees a method call. Our preview data root was the same type delivery uses, and it has a String() method that marshals the entire payload to indented JSON.

So {{if $.Payload.String}} — perfectly legal under the allowlist, since it's just a field chain — bought a full marshal of the payload per call. Our static checker charged it as one unit of work.

The measured worst case, staying inside every request cap we had (246 KB payload under the 256 KB limit, array exactly at the 100-entry truncation ceiling, 1,260-byte template): 400,000 marshals, six to fourteen minutes of one CPU core, and roughly 442 GB of allocation churn. The static budget said fine — 800,202 units charged out of 1,000,000. The render deadline fired at five seconds, but Go template execution can't be cancelled mid-flight, so the abandoned goroutine kept its concurrency slot for minutes. Eight such requests and preview is wedged for every tenant while the delivery workers fight a compacting heap for CPU.

Refusing method names would have been another enumeration — the approach that had already failed seven times. Instead, the fix applied the inversion to data: preview renders against a struct of two plain map[string]interface{} fields. Maps carry no methods. Nothing is callable by any syntax, present or future.

type PreviewInput struct {
	Payload map[string]interface{}
	Meta    map[string]interface{}
}

The accepted cost: a bare {{.Payload}} now prints Go map syntax in preview instead of delivery's indented JSON, and the guarded json helper exists for anyone who wants the pretty form. We'll take that trade.

Over-rejection is a bug of equal rank

A sandbox review naturally fixates on what gets through. But this feature exists to eliminate preview/delivery divergence — so a template that renders at delivery and is refused in preview is the same disease we set out to cure, wearing a different coat.

Adversarial review found a whole class of these too. Declarations were gated to {{range}} headers, wrongly refusing {{with $pr := .Payload.pull_request}} — even though both keywords gate their subject first, so a header declaration is bound to something already vetted. And a range's else branch was billed per-iteration, though the engine runs it exactly once when the collection is empty — inflating a legitimately two-level template's cost by ~50× and refusing it as too deep.

Each over-rejection got fixed by making the rule express the actual hazard instead of a convenient approximation. If your sandbox review only files escapes, half the bug list is missing.

What we took away

• Write the repro before the fix, and run it. Twice, the proof a bypass was real was the test suite hanging. Several plausible-sounding reports were refuted the same way. And a clean review round is weak evidence: the {{with}}-dot hole arrived two rounds after "nothing further to raise."
• Prefer the structural bound to the enumerated one. The allowlist beat seven denylists; the method-free data root beat any list of method names we'd have maintained.
• Model the type bound, not the observed one. 256 because a byte is a byte, not 244 because that's what the harness happened to measure.
• Say the residual risk out loud. The shipped code documents what an authenticated user can still burn within the subset and why we accept it, rather than pretending the number is zero. Sandboxes have budgets, not force fields.

Live preview is in the template editor now — same engine as delivery, per-platform tabs, renders as you type. The sandbox is one Go file with all of this reasoning inline as comments, which is where it'll do the next person the most good.