All posts

Delivered, with zero delivery attempts: when a rename repoints your RLS policies

Evan6 min read

postgresrlsdebugging

The events list said the webhook was delivered. One green check, one attempt, a response time. Clicking into that same event showed an empty delivery-attempts table — no attempt, no status code, nothing. Same event, same user, same page load, two different answers.

Both numbers came out of the same database. Only one of them came through row-level security.

Two good theories, both wrong

The delivery-attempts table has a composite foreign key into events — the event id plus its creation timestamp — so the first theory was the obvious one: the client's embedded query was joining on a shape the database wouldn't honor, and quietly returning an empty set. That's a real failure mode, and it fit the symptom exactly. It also turned out to be wrong: querying the rows directly returned them.

The second theory was project access. If the viewer had somehow lost membership, their reads would come back empty and nothing would error. Also wrong, and refutable without running anything: the event itself rendered. Event visibility and delivery-attempt visibility hang off the same access check for the same user. If access were gone, the page would have been empty, not half-empty.

Both theories died on evidence rather than argument, which is the only reason the third one got looked at. The thing that actually found it was reading the policies themselves — not the table definitions, the policy bodies:

select tablename, policyname, qual from pg_policies
 where schemaname = 'public';

One of those bodies was selecting from a table nobody had mentioned in weeks.

Policies bind to a table, not to a name

PostgreSQL parses a policy expression when you create it and stores the parsed tree, with every table reference resolved to an OID. A rename doesn't change a table's OID. So a policy that referenced events keeps referencing that same table after it's renamed — silently, with no error, no warning, and no indication in the policy's own definition that anything moved. This is the same behavior views have, and it's documented and intended.

It's only surprising when you deliberately hand the old name to a different table. Which is exactly what partitioning an existing table looks like:

  1. Rename the original table out of the way, to something like events_archive.
  2. Create a new partitioned table under the original name, events.
  3. Re-attach the original as a partition, so the historical rows stay reachable.

Every policy on the events table, we recreated as part of that swap. Two policies that merely referenced events from inside their bodies, we didn't think about — one governing which delivery attempts you can read, one authorizing the realtime channel that streams delivery updates into an open event page.

Both of those policies were still pointing at the archive. Their EXISTS (SELECT 1 FROM events ...) check now ran against a table that, by definition, contained no new events. So every event created after the swap failed the check, and every delivery attempt belonging to it became invisible.

The rows were there the entire time.

Why it hid so well

Three things kept this from looking like what it was.

The schema was perfect. Table definitions, foreign keys, indexes, the partition attachment — all correct. Nothing you'd see in \d delivery_attempts or in a migration diff was wrong, because nothing about the schema was wrong. The damage lived in a stored parse tree.

Half the UI bypassed RLS. The events list reads through a SECURITY DEFINER function, which runs with the definer's rights and skips row-level security entirely. The detail page reads the table directly, through RLS. So the list was right and the detail was wrong, and the contradiction between them read like a frontend bug — a stale cache, a bad join, a race. It was neither. It was two different code paths giving honest answers about two different tables.

It broke for everyone, including the owner. Permission bugs usually spare the owner, so "this looks like a permissions problem" was a theory that kept getting discarded on the grounds that the owner couldn't see the rows either. Broken-for-everyone reads as broken, not as access control.

One thing worth stating plainly, because "RLS bug" reads alarming: this failed closed. The policy became strictly more restrictive — it hid rows from people entitled to see them. At no point did it show anyone another tenant's data. If this class of mistake is going to happen, that's the direction you want it to fail, and it's worth choosing schema changes that keep it failing that way.

The fix, and a bonus

The repair is just recreating each policy so its body binds to the current table. Two details were worth keeping beyond the fix itself.

Schema-qualify everything inside a policy body. Policies execute under the invoker's search_path, so an unqualified name is a name you don't fully control. Write public.events, not events.

Pin the partition key. The delivery-attempts policy joins events on the composite key, so adding the timestamp to the join condition lets the planner prune to a single partition instead of probing every one of them — and there's a new partition every month, so that cost grows forever. Visibility doesn't change: the composite foreign key already guarantees exactly that row exists. The correctness fix turned out to be the performance fix.

The realtime policy can't do the same thing, because its channel name carries only an event id and there's no timestamp available to prune on. Worth knowing which of your policies can be pinned and which can't.

Guarding it

The fix is one migration. The guard is the part that matters, because the whole reason this survived as long as it did is that no schema assertion could see it. Everything a normal migration test inspects was correct.

So the check runs against pg_policies directly, in CI, in two directions:

  • Negative: no policy outside the archived partition may reference the archived table by name.
  • Positive: the delivery-attempts read policy must still join events.

The positive half exists because a future rename could equally well cause someone to drop the join rather than repoint it, and a check that only looks for the wrong table would wave that through.

Then we planted a deliberately broken policy and watched both assertions fail. A guard nobody has ever seen fail is not yet a guard — it's a guard-shaped block of SQL that has never been asked a question.

What to take from this

  • When rows are present but invisible, read policy bodies, not table definitions. pg_policies is the first stop, and it should be an early one.
  • A rename is a policy migration event. Renaming a table rewires every policy that references it — including policies attached to other tables, which is the set you won't think to check. Grep your policy bodies for the old name before you rename, not after.
  • Schema-qualify inside policy bodies.
  • Bypass paths hide blast radius. A SECURITY DEFINER function sitting next to an RLS-gated read means half your product can be right while the other half is wrong, and the difference between them looks like a UI bug.

The most expensive part of this wasn't the fix — it was one line of SQL, twice. It was that two reasonable theories were both consistent with the symptom, and the only way past them was to stop reasoning about what should be true and go read what the database actually had stored.

Dispatch receives webhooks from the tools your team already uses, verifies and filters them, and delivers them to Discord, Slack, Telegram, and more — with retries, replay, and full delivery history.