Richard Yen

Are You .ready?

Mon, 06 Jul 2026 08:00:00 +0000

A practical guide to what `.ready` and `.done` mean, and why WAL sticks around

Introduction

It is 9:12 a.m. on a Monday. Someone on your team opens pg_wal/archive_status/ during a storage scare and sees a long list of files ending in .ready. They ask the question many of us have asked at least once: “Is replication broken?” Streaming replicas still look mostly fine, but .ready files keep piling up, disk usage keeps climbing, and nobody is fully sure what .ready and .done are actually telling you.

What is .ready and what (if any) action do I need to take? Let’s talk about that today.

Hint: It’s About WAL Delivery

Think of WAL delivery as three independent steps:

Generate WAL
Transport WAL
Replay or consume WAL

archive_command is one way to do transport. Streaming replication is another. Note, logical replication also has a transport channel, but what it transports is decoded logical change data rather than raw WAL segment files.

What `archive_command` Actually Does

When archive_mode=on, Postgres tries to copy each completed WAL segment to long-term storage by running archive_command.

Typical example:

archive_mode = on
archive_command = 'rsync -a %p backup@walbox:/archives/%f'

%p is the local path to the WAL segment in pg_wal
%f is just the filename

Postgres runs this command from the archiver process. If the command exits with status 0, Postgres treats it as success. Any non-zero exit code means failure, and it retries later.

Info: Archiving usually happens when a WAL segment is complete (typically 16 MB), not every transaction. So pure archive shipping can have more lag unless segment switches happen frequently.

What `.ready` and `.done` Are For

Inside pg_wal/archive_status/, Postgres tracks each WAL segment’s archiving state with tiny marker files.

For a segment named:

000000010000000A000000FE

you may see:

000000010000000A000000FE.ready
000000010000000A000000FE.done

`.ready`

.ready means: “this WAL segment should be archived (or retried).”

Postgres creates .ready when the segment becomes eligible for archiving. If archive_command fails, .ready remains and the archiver keeps retrying. Note, you’ll see information about failures in the Postgres text logs, so check there if you see many .ready files and not a lot of .done files.

`.done`

.done means: “archiving for this WAL segment succeeded.”

After a successful archive_command, Postgres marks completion with .done and no longer retries that file.

In short:

.ready = pending/retry queue item
.done = WAL archive was successful

These are local bookkeeping files. They are not WAL themselves.

If `.done` Exists, When Is It Deleted?

Great question, because .done does not mean “delete immediately.”

In practice, The WAL segment in pg_wal can only be recycled/removed after a checkpoint and only when Postgres no longer needs it. This includes retention constraints like replication slots, wal_keep_size, or standby/recovery requirements. If a segment is still needed for any of those reasons, it stays in pg_wal even if archiving already succeeded.

The .done file is just archive-status metadata. It is cleaned up later as part of normal WAL housekeeping, and it can exist for a while even after archival success.

What If `.done` Files Pile Up and Do Not Disappear?

If .done files accumulate, first separate “cosmetic” from “capacity risk”:

Check whether pg_wal disk usage is actually growing.
If disk usage is stable, this can be harmless housekeeping lag.
If disk usage is growing, treat it as WAL retention pressure.

Then run the diagnostic queries in the Beginner-Friendly Checks section below.

Finally, remember that WAL recycle/removal is checkpoint-driven. If checkpoints are infrequent and WAL is still needed, .done files can legitimately linger.

How This Relates to Streaming Replication

Streaming replication and WAL archiving can both exist at the same time, and they are distinct WAL delivery mechanisms with different telemetry and failure modes.

With streaming replication, WAL is pushed over a live replication connection (walsender -> walreceiver) and can be delivered before a segment is fully closed. It shows up in pg_stat_replication and can use physical replication slots.

On the other hand, archive shipping copies WAL files out via archive_command in a segment-oriented way (usually after file completion), does not require a direct live receiver session, and does not appear as a row in pg_stat_replication.

Important: .ready/.done are about the archiver path, not the streaming path. A standby can be fully healthy on streaming while archiving is broken, or vice versa.

What About `pg_receivewal`?

pg_receivewal is a useful middle ground: it uses the streaming replication protocol to receive WAL continuously, but writes WAL segment files to disk like an archive pipeline.

In other words, transport is streaming, while storage is file-based.

That means:

It can reduce archive lag compared to waiting for archive_command segment completion behavior.
It can use a replication slot (recommended) so WAL is not lost if the receiver is briefly down.
It shows up as a replication sender/receiver relationship, unlike local .ready/.done bookkeeping.

Also important: pg_receivewal does not replace .ready/.done on the primary by itself. Those files are specifically tied to archive_command/archiver bookkeeping on that server.

Replication Slots: Where People Get Confused

Replication slots and archive_command solve different problems.

Physical replication slot: A physical slot protects WAL needed by a streaming standby. Postgres will retain WAL until that standby has consumed it.
Logical replication slot: A logical slot protects WAL needed for logical decoding/subscribers. If the subscriber lags, WAL is retained.
Archiving: Archiving is about making durable WAL copies somewhere else. It does not tell Postgres what a standby has consumed.

So WAL retention pressure can come from multiple places at once:

archiving not succeeding (files stay pending)
lagging physical slot
lagging logical slot
infrequent checkpoints (recycle/removal cadence is delayed)

When disk fills in pg_wal, it is often one of these, or a combination. Even after successful archiving, WAL segments are only eligible for recycle/removal after checkpoint processing.

Beginner-Friendly Checks

A few quick checks I like:

-- Archiver health
SELECT archived_count, failed_count, last_archived_wal, last_failed_wal
FROM pg_stat_archiver;

-- Streaming health
SELECT application_name, state, sync_state, write_lag, flush_lag, replay_lag
FROM pg_stat_replication;

-- Slot retention pressure
SELECT slot_name, slot_type, active, restart_lsn, confirmed_flush_lsn
FROM pg_replication_slots;

-- Quantify physical slot lag in bytes from current WAL position
SELECT slot_name,
	   active,
	   pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS bytes_behind
FROM pg_replication_slots
WHERE slot_type = 'physical';

What to look for:

pg_stat_archiver: failed_count should not climb continuously. archived_count and last_archived_wal should move forward during write activity. If last_failed_wal keeps changing but last_archived_wal does not, archiving is unhealthy.
pg_stat_replication: state should usually be streaming (or sync_state aligned with your design). Persistent high write_lag/flush_lag/replay_lag, or missing expected standbys, points to transport/replay trouble.
pg_replication_slots: watch for inactive slots with old restart_lsn (physical) or stale confirmed_flush_lsn (logical). Large bytes_behind is a strong signal that a slot is pinning WAL and growing pg_wal.

And on disk:

ls -1 $PGDATA/pg_wal/archive_status | tail

If .ready files keep piling up, your archive path is unhealthy. If .done files pile up and pg_wal keeps growing, check slot lag and checkpoint cadence. If slot lag keeps growing, your consumer is unhealthy.

Final Thoughts

.ready and .done files are not mysterious once you view them as a local queue and completion marker for archive_command.

They are adjacent to replication, but not identical to streaming replication or logical replication.

Getting comfortable with these distinctions makes debugging much faster, especially when someone says, “replication is broken,” and you need to answer: which replication path?

Disaster Recovery is a Process, Not a Tool (Part 2)

Mon, 29 Jun 2026 08:00:00 +0000

Picking Up Where We Left Off

In the previous post, I tried to lay out the framing half of this material: what actually counts as a disaster, why preparation and prevention aren’t the same as recovery, and how RPO and RTO end up being conversations with leadership rather than numbers an infrastructure team gets to declare on its own.

That part is largely about understanding the problem. This part is about actually building the capability to deal with it. And in my experience, this is where most teams quietly stumble – not because they don’t have backups or replication, but because they’ve never really practiced using either of them under stress.

Runbook Engineering: Why Runbooks Fail

Once you’ve accepted that DR is a process, the natural next step is to write some runbooks. And in my experience, this is where a lot of teams quietly stumble.

Runbooks usually get written by experts, for experts, in a calm conference room. That’s almost the opposite of the environment they’ll actually be run in. Real recovery happens at 3AM, under stress, often with incomplete information and sometimes with someone who isn’t deeply familiar with the system. A runbook’s job, really, is to reduce ambiguity in that moment – not to be a comprehensive description of the system, but to be the thing you can follow even when you’re tired and scared.

That framing alone changes what a good runbook looks like.

Anti-patterns

A few patterns I see often that make runbooks worse, not better:

Giant wiki pages. I think a lot of teams try to kill two birds with one stone by sprinkling DR steps inside their general system documentation, so they don’t have to maintain a separate doc. What you end up with is something that’s hard to follow under stress, and that drifts quietly as the surrounding documentation evolves.
Stale commands and hostnames. Infrastructure changes over the years. Commands that worked in Postgres 11 are sometimes not quite right anymore. I’ve definitely been the person who wrote a runbook step that turned out to be wrong by the time anyone needed it.
No rollback criteria. If the runbook tells you to run a command, it should also tell you what success and failure look like, and what to do if the command does something unexpected.
No clear ownership. Everyone wants to ship the next feature, expand the cluster, do the upgrade. Updating the runbook is the thing that gets put off. In my experience, this works best when the reliability engineering team owns the runbooks explicitly, rather than treating them as collective property nobody is really responsible for.
Vague instructions. “Inspect the logs for errors or warnings” is a step I see all the time. But where are the logs? Every organization seems to put them in a slightly different place. When I drop into a customer’s database server, finding the Postgres log is often surprisingly nontrivial – and that’s the most important source of information about the health of the system, so it really shouldn’t be a scavenger hunt.

What a good runbook looks like

A good runbook is procedural and deterministic. It tells you “this is the situation; here is exactly what to do.” If split brain happens, what command do you run? If replication lag is over a certain threshold, do you fail over or wait? Those decisions should already be made, by the people who had the luxury of making them calmly.

Instead of “fail over to the read replica,” a good runbook says which replica, what the promotion command is, what to check after promotion to make sure it actually worked, and what to do if it didn’t. It includes the pre-flight checks before you bring traffic back, and the rollback path if the recovery itself goes sideways.

Non-technical essentials

The non-technical pieces are easy to overlook, and in my experience they make at least as much difference as the technical ones.

One thing I really appreciated when I worked in EDB Support was that every ticket had a clearly designated commander – the person who talked with the customer, gathered the information, and directed the other support engineers. That role makes a huge difference in an incident, because somebody needs to be coordinating, and it generally shouldn’t also be the person at the keyboard.

When I moved to Microsoft, I noticed an additional layer that I’d underrated before: a separate communications owner. When Azure has an incident, there’s an incident commander focused on getting the system back up, and there’s a different person whose job is to communicate – via email, social media, status pages, Discord, whatever the channel is – so customers know what’s happening and what to expect. Splitting those two roles takes a lot of pressure off the technical recovery, and I’d recommend it to anyone running services that customers notice.

Beyond those two roles, a few other things really do belong in the runbook:

Stakeholder notification cadence. Even if there’s no new information, telling stakeholders “no change, still working on it” every fifteen minutes is far better than silence.
A clear escalation chain. Primary on-call, backup on-call, the right engineering contact, the business owner. In my experience, escalations to engineering often happen a little prematurely – and a little unclearly – because nobody mapped out who specifically to call.
Customer impact thresholds. At what point do we change our response posture? If more than half our customers are affected, what changes?
Risk authorization. Some recovery actions are dangerous on their own – restoring a month-old backup, accepting data loss, dropping a replication slot. Who is authorized to make those calls, and how do you reach them?

Validation

Once you have a runbook, you have to actually validate it. Some questions I think are worth asking:

Can a new engineer follow it? This is genuinely worth testing. I think it’s a fine onboarding exercise to take a dev environment, break it in a controlled way, hand the new hire the runbook, and see if they can get it back. If they can’t, that’s not a failing of the new hire – it’s feedback on the runbook.
Does the runbook assume privileged access the people running it won’t have under stress – passwords, certificates, VPN, bastion access?
Are the specifics still right? Hostnames change. Schemas change. Resource group names change. The runbook needs to drift with them, or, better, be re-tested often enough to catch the drift.

Honestly, keeping runbooks current is close to a full-time job at any reasonably-sized organization. I think it’s reasonable for it to be somebody’s main job – regularly talking to teams, keeping the docs aligned with the infrastructure, organizing the drills.

Game Days

Which brings me to game days. In my experience, this is where DR shifts from a document to an actual capability. A DR plan is not really proven by the fact that it exists; it’s proven through repeated, successful execution.

We started doing this at Turnitin around our release cycle. For a meaningful release, we’d set up an environment that was as close to production as we could make it – same hardware where possible, same CNAMEs, maybe just a different subnet – and we’d practice the release on it. It started as a release-rehearsal habit, but the same idea applies to DR: a controlled failure simulation designed to test the systems, the procedure, the coordination, and – maybe most importantly – the assumptions people are quietly making.

I think it helps to think of game days as having levels you grow into.

You start simple: restore a backup, promote a standby, redirect traffic, validate that the application actually behaves correctly after the cutover. Just being able to do that end-to-end, on demand, puts you ahead of a lot of teams.

Then you start adding pressure. Give the team an SLA on the drill – “we’re going to break something at 10:00 AM, and you have 30 minutes to be back up.” This is where you find out what your actual achievable RTO is, as opposed to the aspirational one in the spreadsheet.

Then you start adding chaos. What if the person who normally runs the failover is unavailable? What if a certificate has expired and now you have to renew it while the outage is happening? What if you scheduled the drill – not entirely by accident – on a day when a key engineer happens to be on vacation? These sound mean, but they’re realistic. Real incidents are not polite about your team’s calendar.

A few practical rules I’ve found useful for running drills without making things worse:

Start in staging or on a standby; not on the production primary.
Schedule the drill and announce it. The drill is the surprise; the date is not. You’re testing the runbook, not your team’s reflexes at 2AM on a holiday.
One failure mode per drill, at least at first. Don’t combine “disk fails AND network partitions AND VP of Engineering is on a plane” until you’ve nailed each of those individually.
Define what success looks like before you start. “We restored within 30 minutes” is testable. “It went well” is not.
Run a retro within a couple of days, while it’s still fresh.

What to Measure

When you finish a drill (or a real incident), the obvious thing to measure is “did recovery succeed?” That’s important, but it’s also kind of binary, and you’ll learn more from the other things you can measure:

How long did each phase take – detection, decision-making, execution, validation, communication? That’s where you’ll find your real bottlenecks.
Where did the runbook turn vague? Anywhere a participant had to ask “wait, what does this mean?” is a small documentation bug worth fixing.
Which dependencies turned out to be undocumented?
Which credentials or permissions were missing or expired?
How close were the actual results to the stated RTO and RPO? If the gap is big, that’s important information for the next conversation with leadership.

Don’t Blame, or You’ll Feel Lame

I want to spend a minute on the cultural side, because I think it might matter as much as the technical side.

Incidents and drills are stressful. I don’t think we always give that enough weight – we’re more comfortable thinking about CPU and IOPS than we are about the very real fact that people freeze under pressure, or hide uncertainty, or act too quickly to make the discomfort go away.

In a blame-heavy environment, people shrivel up. They stop volunteering information. They don’t want to admit they don’t know something, or that they did something that didn’t work, so they make assumptions instead of asking. What you end up with is delayed escalations, silent failures, and risky decisions made by people who didn’t feel safe enough to talk through their thinking. None of that helps your RTO.

The opposite culture – one that encourages people to verbalize what they’re seeing, what they’re not seeing, and where they need help – is much harder to build than a runbook, but pays off in every incident. Post-incident reviews really should be about improving the system, not punishing the person. And when a drill or a real recovery is hard, I think it’s genuinely worth acknowledging that out loud. Order pizza, go out for sushi, do something to thank the team. It’s not a substitute for fixing the technical gaps, but it does make people show up to the next drill.

Wrapping Up

If I had to compress this whole series into one line, it would be roughly this: make your RPO worth it by investing in your RTO.

Lower RPO is something you can largely buy – with replication, with hardware, with cloud spend. Lower RTO isn’t really something you can buy. It’s something you build, slowly, with runbooks, drills, retrospectives, and a culture that treats incidents as opportunities to learn instead of opportunities to assign blame.

Tools and AI both have a place in this work, and the tooling around Postgres is genuinely good. But in my experience, the difference between a team that handles a disaster well and one that doesn’t isn’t usually the tools they had – it’s the process they’d practiced.

If nothing else, I’d suggest picking a day next month, killing a replica in a safe environment, and seeing what happens. The first drill is almost always the most informative one. Good luck out there!

pg_stats: How Postgres Internal Stats Work

Mon, 22 Jun 2026 08:00:00 +0000

Introduction

I recently had the privilege of speaking at POSETTE 2026 about pg_stats and how Postgres internal statistics work (YouTube Recording). This post is a written companion to that talk – aimed at giving you a working understanding of what pg_stats is, how it’s populated, and how it shapes the decisions the query planner makes on your behalf.

Imagine a customers table that looks roughly like this:

CREATE TABLE customers (
    id          bigserial PRIMARY KEY,
    city        text NOT NULL,
    state       text NOT NULL,
    signup_date date NOT NULL
);
-- Insert 1,000,000 rows

Consider a query you’ve probably written many times:

SELECT * FROM customers WHERE state = 'CA';

With separate indexes on state and city, you might expect an index scan on state. But the EXPLAIN ANALYZE output may look something like this:

                              QUERY PLAN
-----------------------------------------------------------------
 Seq Scan on customers  (cost=0.00..19682.66 rows=173829 width=26)
                        (actual time=0.025..120.574 rows=172001 loops=1)
   Filter: (state = 'CA'::text)
   Rows Removed by Filter: 827972
   Buffers: shared hit=4601 read=2582
 Planning:   Buffers: shared hit=139
 Planning Time: 0.371 ms
 Execution Time: 128.136 ms

A sequential scan, even with an index available. We’ll get into the reasons for this today.

Query Plans Are Made by the Query Planner

When you submit a query to Postgres, the query planner is responsible for deciding how to execute it. You may assume the planner reads your actual data – it doesn’t. What it really reads is a summary of your data, stored in pg_statistic.

That summary tells the planner things like:

How many distinct values appear in a column
What the most common values are, and how often they show up
What the rough distribution of values looks like across a range
Whether the data is laid out on disk in roughly the same order as the column’s natural sort order

pg_statistic itself is a bit hard to read directly – the values are stored in formats optimized for the planner, not for humans. Fortunately, Postgres provides a view called pg_stats that exposes the same information in a far more readable form.

ANALYZE: How the Summary Gets Built

The summary in pg_statistic doesn’t populate itself. It’s built (and refreshed) by the ANALYZE command:

ANALYZE customers;

ANALYZE scans the table (or a sample of it), computes a handful of statistics per column, and writes the results into pg_statistic. Autovacuum runs ANALYZE for you in the background, but after large data loads or migrations, you’ll often want to run it manually.

Let’s look at what comes out of it. Using our customers table:

SELECT attname, n_distinct, null_frac, correlation
FROM pg_stats
WHERE tablename = 'customers'
ORDER BY attname;

   attname    | n_distinct | null_frac |  correlation
--------------+------------+-----------+--------------
 city         |      10106 |         0 |  0.0021338463
 id           |         -1 |         0 |             1
 signup_date  |       1822 |         0 |             1
 state        |         50 |         0 |    0.06440461
(4 rows)

A few things worth pointing out:

n_distinct is the estimated number of distinct values. For state, it’s exactly 50 – which lines up with the number of states in the United States. city reports around 10,106, which is believable for U.S. cities. A value of -1 in the id column means the column is unique: every row has a distinct value.
null_frac is the fraction of rows where the column is NULL. All four columns here are NOT NULL, so the values are all 0.
correlation is a number between -1 and +1 that estimates how well the on-disk physical ordering of the table matches the logical ordering of the column. A value of +1 means the data is perfectly sorted on disk (e.g., an incrementing id column or a date column in an append-only table). Values close to 0 mean the data is randomly ordered on disk relative to the column’s values. Values close to -1 or +1 encourage index scans; values near 0 discourage them. The docs go into more detail, but practically it acts as a penalty multiplier in the cost calculation.

Most Common Values (MCV)

ANALYZE also captures a list of the most common values in a column, along with their frequencies. These live in two parallel arrays: most_common_vals and most_common_freqs.

SELECT
    unnest(most_common_vals::text::text[]) AS state,
    unnest(most_common_freqs)              AS frequency
FROM pg_stats
WHERE tablename = 'customers' AND attname = 'state'
LIMIT 5;

 state |  frequency
-------+-------------
 CA    |  0.17403333
 TX    |      0.1165
 NY    |  0.08586667
 FL    |      0.0666
 IL    |      0.0474
(5 rows)

So CA appears in about 17.4% of the rows, TX in 11.6%, and so on. These frequencies feed directly into how the planner estimates rows – and therefore how it picks scan types.

Costs: How the Planner Picks a Plan

The planner makes its choices based on cost. Three of the more commonly-encountered cost parameters are:

random_page_cost – the cost of fetching a random page from disk (think: index scan plus heap fetch)
seq_page_cost – the cost of fetching a page sequentially (sequential scan)
cpu_tuple_cost – the cost of processing each row pulled out of a page

These costs, along with row estimates from pg_statistic, drive two big decisions:

Scan Type

Sequential Scan
Index Scan
Bitmap Heap Scan

Join Type

Nested Loop
Hash Join
Merge Join

The planner generates several candidate plans, calculates the cost of each, and picks the cheapest. Bad statistics lead to bad row estimates, which can lead to bad plan choice. As you can see, having good statistics is vital to query performance.

A Tale of Two States

Watch what happens when we query for CA versus WY:

EXPLAIN ANALYZE SELECT * FROM customers WHERE state = 'CA';
-- Seq Scan on customers  (cost=0.00..19682.66 rows=174029 ...)
--                        (actual time=0.042..50.257 rows=172001 loops=1)

EXPLAIN ANALYZE SELECT * FROM customers WHERE state = 'WY';
-- Index Scan using customers_state_idx on customers
--    (cost=0.42..13116.39 rows=4233 ...)
--    (actual time=0.045..21.238 rows=4300 loops=1)

CA matches about 18% of the table – around 180,000 rows. For every matching row, an index scan would need to look up the row in the index, fetch the page off disk, and pull the tuple out. Doing that 180,000 times turns out to be more expensive than just reading the whole table sequentially. So the planner picks a sequential scan.

WY, on the other hand, matches only about 4,000 rows. At that selectivity, the index scan wins by a wide margin.

We can confirm this is really about cost by forcing the issue. If we temporarily disable sequential scans, the planner is forced to use the index:

SET enable_seqscan = off;

EXPLAIN ANALYZE SELECT * FROM customers WHERE state = 'CA';
--  Index Scan using customers_state_idx on customers
--    (cost=0.42..32172.73 rows=170529 width=26)
--    (actual time=0.053..75.656 rows=172001 loops=1)

The planner’s original choice of a sequential scan (cost ~19,682) was cheaper than this forced index scan (cost ~32,172). The MCV statistics told the planner that CA shows up a lot, and the planner correctly judged that a sequential scan would be cheaper. Skewed data is exactly when MCV earns its keep.

Histograms: For Everything That Isn’t Equality

MCVs are great when you’re searching for specific values. But what about ranges – signup_date BETWEEN ..., or id > ...? For that, ANALYZE builds a histogram.

By default, the histogram has 100 buckets, each holding roughly the same number of rows (it’s an equi-depth histogram). You can look at the bucket boundaries:

SELECT (unnest(histogram_bounds::text::date[]))::date AS bucket_bound
FROM pg_stats
WHERE tablename = 'customers' AND attname = 'signup_date'
LIMIT 8;

 bucket_bound
--------------
 2018-01-01
 2018-03-01
 2018-04-28
 2018-07-04
 2018-09-05
 2018-10-28
 2018-12-27
 2019-02-14
(8 rows)

Each bucket here covers roughly two months – about 1% of the table. That’s fine for many cases, but if you have a large table with a skewed time distribution, you may want more precision.

Consider this side-by-side comparison:

A low-precision histogram (few buckets) might tell you “the most data lives somewhere between 30 and 65.”
A high-precision histogram (many buckets) might tell you “the peak is between 50 and 52, with a clear dip around 65–70 and a second smaller peak near 80.”

Both are technically correct. Only one helps the planner make a good decision when your query is WHERE value BETWEEN 65 AND 70.

You can increase the bucket count on a per-column basis:

ALTER TABLE customers
    ALTER COLUMN signup_date SET STATISTICS 1000;
ANALYZE customers;

Now the same query shows buckets that are about two days wide instead of two months:

 bucket_bound
--------------
 2018-01-01
 2018-01-03
 2018-01-05
 2018-01-07
 2018-01-10
 2018-01-12
 2018-01-14
 2018-01-16
(8 rows)

Beware of trade-offs. More buckets means more precision, but also more work for ANALYZE and a larger pg_statistic row to traverse during planning. Don’t increase default_statistics_target across the entire database – target only the columns where you actually have problematic estimates.

Correlation Between Columns

So far we’ve looked at a single column at a time. Things get more interesting – and more wrong – when you filter on two columns at once:

EXPLAIN ANALYZE
SELECT * FROM customers
WHERE city = 'Cheyenne' AND state = 'WY';

-- Index Scan on customers  (cost=... rows=8 width=...)
--                          (actual rows=4012)

The planner estimates 8 rows, but the query returns 4,012. That’s a 500x miss.

This happens because, by default, the planner assumes columns are statistically independent:

\[P(\text{city} = \text{Cheyenne} \;\wedge\; \text{state} = \text{WY}) = P(\text{city}) \times P(\text{state})\]

In reality, city and state are correlated. There’s basically one Cheyenne in the U.S., and it’s in Wyoming. (There’s also a Cheyenne, Oklahoma, but with a population of around 700, it doesn’t really move the needle.) So filtering on city = 'Cheyenne' is almost equivalent to filtering on state = 'WY', but the planner doesn’t know that.

Since Postgres 10, you can tell it:

CREATE STATISTICS customers_city_state (dependencies, ndistinct)
ON city, state FROM customers;

ANALYZE customers;

The dependencies and ndistinct arguments are statistic types – they tell Postgres what kind of cross-column information to track. After re-running ANALYZE:

EXPLAIN ANALYZE
SELECT * FROM customers
WHERE city = 'Cheyenne' AND state = 'WY';

-- Index Scan on customers  (cost=... rows=4087 width=...)
--                          (actual rows=4012)

The estimate of 4,087 versus actual 4,012 is essentially perfect. Just as importantly, this estimate now feeds correctly into any joins or aggregations that sit on top of this scan. Mis-estimates at the bottom of a plan tend to compound – a wrong scan choice deep in the tree can cause cascading mistakes in the joins above it. That’s part of why getting the foundational statistics right matters so much.

Note: Creating extended statistics isn’t free. It adds a small amount of overhead to ANALYZE and to the query planning process itself. You should only create them when you’ve identified a clear case of mis-estimation due to correlated columns.

A Quick Checklist for Query Performance

When a query plan looks wrong, here’s roughly the order I work through:

Compare estimated vs. actual rows. Run EXPLAIN ANALYZE and look for the deepest node where the estimate disagrees with reality. That’s usually where the problem starts.
Check pg_stats for that column. Look at n_distinct and the MCV list. Do they match what you know about your data?
If the stats look stale, run ANALYZE. This is especially common after a big batch load, a migration, or a partition swap. Autovacuum may not have caught up.
If estimates are still off on a single column, raise the statistics target. ALTER TABLE ... ALTER COLUMN ... SET STATISTICS 1000; and re-ANALYZE.
If the bad estimate involves two columns in the same WHERE clause, consider correlation. Create extended statistics on the pair.
Only after all of that should you consider rewriting the query – or reaching out for help.

Conclusion

The Postgres query planner is impressively good at its job, but it isn’t magic. It makes decisions based on a summary of your data, and the quality of those decisions is bounded by the quality of that summary. pg_stats is your window into what the planner thinks is true about your tables – and when reality and the planner’s beliefs diverge, that’s usually where bad plans come from.

The next time EXPLAIN ANALYZE surprises you, before you start setting enable_seqscan = off in production or rewriting the query out of frustration, take a look at pg_stats first. More often than not, the answer is there.

The Query Planner is only as smart as the statistics you feed it.

Disaster Recovery is a Process, Not a Tool (Part 1)

Mon, 15 Jun 2026 08:00:00 +0000

The Landscape Has Changed

When I was at Turnitin, we were still kind of riding the tail end of the dot-com boom. People were rushing to ship things, and brief outages were not exactly good, but they were considered a normal part of running software on the internet. If the site was down for a few minutes, you’d shrug, dig in, and fix it.

That’s not really the world we live in anymore. Uptime is much more sensitive than it used to be. Five nines used to be the stretch goal – now four nines is something a lot of teams just treat as the expectation, and even a few minutes of outage in a month feels like a lot. We don’t really track averages in our metrics anymore, either; we track p99 latencies, because we actually care about that last 1% of users having a good experience.

The other thing that’s changed is how quickly outages get socialized. A noticeable hiccup in your service can end up on social media before your on-call has even finished acknowledging the page. In my experience, the worst situations are the ones where customers find out about an issue before the company does. That has both a financial cost and a reputational cost, and the reputational cost tends to linger long after the incident is resolved. Frequent outages chip away at users’ willingness to keep using your product.

Postgres is, of course, no exception. So that’s the world a Postgres DR plan has to operate in.

What Counts as a Disaster?

When people hear “disaster recovery,” I think the natural mental picture is a natural disaster – a flood, an earthquake, a wildfire, or maybe a long utility outage that takes a data center offline. And those are real concerns; we put generators and solar panels and multi-region replication in place partly to deal with exactly that.

But in my experience, most of the disasters that take a Postgres database down don’t look anything like that. They look like:

A performance regression after a failover, where the service is technically “up” but slow enough that customers can’t really use it.
Corruption from a bad migration – something the deployment pipeline didn’t catch, and now half the rows in a table look wrong.
A security incident, where somebody got in and may have tampered with data.
A subtle application bug that writes the wrong values, or reads them back the wrong way, for days before anyone notices.
Replication that quietly broke, or WAL that quietly went missing.
Accidental deletes – the classic missing WHERE clause.

If I had to put a definition on it, I’d probably say something like: a disaster is any sustained event that compromises a system’s availability, correctness, or business trust. Availability is the one that gets the most attention, but the other two are arguably more dangerous, because they tend to be discovered later and resolved with less confidence.

How DR Is Usually Done

If you ask most teams how they do disaster recovery, you’ll usually hear two words – not because they’re wrong, but because they’re the first things that come to mind. Those words are preparation and prevention.

Preparation looks like checklists, backups, monitoring, scenarios to think through, and playbooks of various levels of detail. Prevention looks like alerting, automated remediation, self-healing systems, Patroni, redundancy, load balancing, and so on. Both are good. Benjamin Franklin’s “an ounce of prevention is worth a pound of cure” is on the wall of more than one ops team I’ve worked with, and there’s a reason for that – prevention really is cheaper than recovery on average.

But preparation and prevention only get you so far, and I don’t think they’re really the same thing as recovery. Recovery is what happens after preparation and prevention have already failed to keep the lights on. It’s the act of taking a system that’s already down (or already untrustworthy) and restoring business operations.

That distinction sounds almost too obvious to say out loud, but in my experience it’s the part teams are least ready for. A lot of the customers I worked with at EDB were genuinely well-prepared, with great backups and good monitoring, and they were still unprepared the day they actually had to recover. I’ve been in that seat too, as a DBA – everything was in place on paper, and we still fumbled the first real incident. Recovery is its own skill.

Postgres Already Gives Us Most of the Tools

One nice thing about Postgres is that the toolbox for recovery is already pretty good. Off the top of my head, there’s pg_dump and pg_restore for logical backups, pg_basebackup for physical ones, pg_stat_replication to see what your standbys are doing, pg_stat_activity to see what your sessions are doing, point-in-time recovery for anything more granular than “last night’s backup,” and tools like repmgr and EFM (and pgBackRest, Barman, and others) for orchestration and richer backup workflows.

These tools are not the bottleneck. In nearly every case I worked at EDB, the question wasn’t “do we have the technology to recover?” It was, “do we know when to use it, how to use it, and who gets to make the call?” I had a customer once who had perfectly good backups – they really did – but they opened a P1 ticket asking me to walk them through the keystrokes for the restore. I think they actually knew what to do; they were just afraid, in the moment, of typing the wrong thing. That’s a process gap, not a tool gap, and no amount of additional automation would have fixed it.

I’d add a slightly uncomfortable note here: as a vendor’s support engineer, I was always happy to help, but we probably shouldn’t be the centerpiece of anyone’s DR plan. Support engineers can hand you tools and walk you through documentation, but we don’t know your data the way your team does, and there’s a liability we’re not really supposed to take on. If the first time a team reads the failover documentation is during the outage, a support contract alone isn’t going to close that gap.

RPO and RTO, and Why They’re Negotiations

You can’t really talk about recovery without talking about RPO and RTO, so let me do that briefly.

RPO (Recovery Point Objective) is roughly “how much data are we willing to lose?” Do we restore from last night’s backup and accept losing the day’s writes? Or do we replay WAL and try to get as close as we can to the moment of the outage?

RTO (Recovery Time Objective) is “how long are we allowed to be down before we’re considered back up?”

Every choice on either of these axes is a trade-off – against cost, against complexity, against operational burden, against acceptable business loss. And the reality is that during an outage, you really are losing business; transactions don’t happen, shopping carts don’t get checked out, customers get frustrated. At the same time, getting up faster usually means accepting more data loss, or paying more for the infrastructure to avoid it.

It’s helpful to think about RPO in tiers:

A 24-hour RPO is basically “restore last night’s backup.” One or two people can usually handle it, the moving parts are simple, and the data loss can be substantial. That’s fine for some workloads. It’s not really acceptable for high-traffic services where 24 hours of writes is a lot.
A 15-minute RPO generally means WAL archiving or shipping, monitoring to make sure none of that WAL goes missing, regular validation that you can actually restore in 15 minutes, and operational discipline around retention. That’s reasonable for many systems, but probably not acceptable for, say, a financial institution.
A near-zero RPO typically means synchronous replication and tightly managed failover. Now you’re dealing with latency between nodes, distributed-systems complexity, split-brain scenarios, and a much bigger operational footprint.

Lower RPO isn’t just “better.” It’s a design and operational commitment, and that commitment costs money, time, and people’s attention.

The same is true of RTO. Driving RTO below five minutes generally requires automation and – this part is important – rehearsal. If you hand someone a document for the first time during an actual outage, they are not going to execute it quickly, no matter how clear the document is.

This is why I think RPO and RTO really need to be negotiated, not just declared. On the surface it’s almost a no-brainer – of course everyone wants an RPO of zero and an RTO of seconds. But when you actually go to leadership and lay out what those numbers cost, you tend to find out pretty quickly where their priorities really sit. In a lot of cases, they’d rather spend that money on something that looks more directly tied to the business – a new feature, a marketing push, another engineer on the product team – and they’re willing to accept a softer RPO or RTO in exchange. That’s a legitimate answer; it just needs to be made explicitly, instead of being assumed one way or the other by the infrastructure team.

Three Layers of DR Planning

When I think about what a DR plan needs to cover, I find it useful to break it into three layers.

The first layer is infrastructure failure. This is the one most teams think of first: a region goes down, storage fails, a corruption bug bites, credentials leak, somebody accidentally deletes a table, replication breaks. Hardware and platform behaving badly.

The second is procedural failure. Even if the infrastructure problem is well-understood, you can still fail recovery because the procedure is wrong. Maybe the sequence values weren’t included in the backup and you didn’t realize. Maybe the runbook references a CNAME nobody can find the host for anymore. We used to have a setup at Turnitin where, on every failover, we had to repoint a CNAME to the new primary, and we eventually realized that nobody had documented which CNAME pointed to which underlying host. Maybe the validation step is vague. Procedural failure tends to be invisible until the moment you actually need the procedure.

The third is human failure. People behave differently under duress. Some panic. Some zone in so hard on one screen that they miss the bigger picture. There are conflicting instructions between managers, between teams, between people trying to be helpful. There’s the 3AM call where the on-call is barely awake and not entirely sure what’s going on. And there’s the person who can’t wait for the process and decides to just do something heroic and fast – which sometimes works, and sometimes makes things significantly worse.

To make the layers concrete: I had a 3AM incident at Turnitin once where we rolled out a change in the evening and got paged a few hours later. The disk had filled, and the filesystem ended up unmounted. That was the infrastructure failure. In the scramble to bring it back, somebody tried to remount it as ext4 instead of xfs – that was strike one, a procedural failure, because the runbook didn’t make the filesystem type explicit. Then we sat for a while waiting on the CTO, because nobody on the bridge had clear authority to call any of the next steps – strike two, no incident commander. And then somebody prematurely brought the web servers back up before the database was really healthy, causing a second round of errors – strike three, the hero move. No single one of those was catastrophic; together they turned a one-hour problem into a much longer night. That’s what the three layers look like in practice.

Recovery Isn’t Always About Failing Over

A lot of DR talks (and a lot of DR vendors) make it sound like “recovery” basically means “fail over to the standby.” That’s one tool in the box, but it’s nowhere near the whole box.

Here’s a story that’s stuck with me. I was on a small team that shipped a release, and the migration looked clean – everything came up, the smoke tests passed, we went home feeling pretty good. Later, somebody noticed that the application code had a small typo in its SQL: an extra apostrophe was getting written into every comment in a comment thread. The data wasn’t lost. The system was up. But the data was wrong, and it kept getting more wrong every minute the application stayed online.

In that particular case, a careful UPDATE across the table was probably the right call, with all the locking and performance impact that implies. But if you change the details a little – say the corruption is medical records, or it isn’t discovered for a few days, or some of those rows have already been read by other systems and propagated outward – a simple UPDATE stops being the answer. Now you’re asking whether you have enough WAL retained to do point-in-time recovery, whether you can safely update some rows in place, when exactly the corruption started, and so on.

I bring it up because that scenario is just as much a disaster, and just as worth planning for, as a disk failing or a region going dark. And it can’t be solved by failing over – the standby would have the bad data too.

While I’m telling stories about quietly-bad situations: another underrated failure mode is “the engineer who knew this part of the system went on vacation,” or quit, or moved teams, and the documentation never quite got updated. Real DR plans have to assume some of that, too.

What Lower Numbers Actually Cost

Negotiating RPO and RTO sounds abstract until you start listing the consequences. Wanting an RPO of zero pushes you toward synchronous replication and forces you to live with the latency that comes with it. Wanting an RTO of under five minutes pushes you toward automation that has to be built, tested, and maintained, and toward rehearsal cadence that has to be on someone’s calendar. Multi-region pushes operational complexity up significantly – you’ve got clusters in different regions talking to each other, you’ve got cross-region replication lag to tolerate, and now your monitoring story has to account for all of it. Even something as innocuous as “we’d like to be able to do point-in-time recovery to any second over the last 30 days” can mean keeping terabytes of WAL around and paying for storage you barely look at.

None of this is a reason not to do these things. It’s just a reason to have honest conversations about which of them you actually need.

To Be Continued

That covers what I think of as the framing half of the talk: what counts as a disaster, why preparation and prevention aren’t the same as recovery, and how RPO and RTO end up being negotiations rather than declarations.

In two weeks, I’ll get into the part that I think could reduce RTO (something that can’t be replaced by AI): runbook engineering, game days, what to measure, and the cultural piece that holds it all together.

PGDay Boston 2026

Wed, 10 Jun 2026 08:00:00 +0000

Introduction

PGDay Boston 2026 was a rewarding reminder of why I value the PostgreSQL community so much. It was delightful to reconnect with familiar faces, meet new people, and finally put some faces to names for the first time. One of the best parts of the day was the sense that this community is larger than any one employer or project. It is built on shared curiosity, shared responsibility, and a willingness to help one another learn. I’m honored to have been able to share my own thoughts in my Disaster Recovery talk as well.

The keynote, Michael Stonebraker’s “Where Did Postgres Come From?”, was a standout for me. I especially appreciated the history of Postgres and the years before Postgres, during the Ingres era. It was striking to hear how the project could have ended up as just another academic system, yet instead grew into something enduring because people outside of UC Berkeley took ownership of it and built a broader community around it. That story felt like a good reminder that open source succeeds not only through technical merit, but through stewardship and continuity.

I also enjoyed Brian Brennglass’s talk, “Managing and Observing Locks.” His demos made an intimidating topic much easier to follow, and I found the practical framing especially useful. Shree Vidhya Sampath’s session on leveraging Patroni’s synchronous replication while running PostgreSQL on Kubernetes was another highlight. I appreciated the clear discussion of election behavior, synchronous replication, and failover scenarios, including failure modes I had not experimented with myself.

Robert Haas’ “pg_plan_advice: Plan Stability and User Planner Control for PostgreSQL?” was impressive in his attention to detail, especially the way he tested edge cases that people might not think to check. Bruce Momjian’s “What’s Missing in Postgres?” was also thought-provoking because it framed missing features not as oversights, but often as deliberate choices shaped by the needs of the broader community. Ryan Booz’s “Mastering PostgreSQL Partitioning: Supercharge Performance and Simplify Maintenance” rounded out the day well with a useful refresher on partitioning behavior, tradeoffs, and current workarounds.

Overall, the event benefited me in both personal and professional ways. Professionally, it deepened my understanding of PostgreSQL internals, operational patterns, and ecosystem tooling. Personally, it renewed my appreciation for the community that has grown around Postgres and the care that goes into keeping it healthy. Thank you Tom Kincaid, Ken Rugg, Erik Pohi, Greg Burd, Kanchan Mohitey, Shihao Zhong, and so many others – along with PGUS – who worked hard to make the first PGDay Boston a smashing success! I look forward to staying involved and attending future events in Boston and beyond.

Foreign Tables and Materialized Views: A Dynamic Duo

Mon, 25 May 2026 08:00:00 +0000

Introduction

I recently wrote a post about WAL log shipping and how a standby built on log shipping is a great way to give data analysts production data without putting the primary at risk. Having access to the production data in this way is great, but it’s read-only. How can we create views of this data for better analytics work? I want to make the case today that Foreign Data Wrappers and Materialized Views can make a great solution – not only in accessing production Postgres data, but also working with other data sources.

Moving Beyond FDW Demos

Most people meet foreign data wrappers (FDWs) through a quick demo, and I’ve highlighted some of their features in previous conference talks. There is high novelty in being able to query MySQL from Postgres, but the reality is often that the latency between the local database and the foreign table can be pretty high. Sometimes, predicate push-down isn’t what you’d expect, and indexing may not be very transparent. In the end, setting up and managing FDWs may seem more work than it’s worth, and that’s a mistake. Used correctly, foreign tables are one of the most practical tools for analytics across heterogeneous data sources – especially when paired with materialized views.

The Real Problem: Heterogeneous Data

Modern data rarely lives in one place:

Legacy systems in MySQL
Operational data in PostgreSQL
Flat files sitting in object storage (I’ve seen people do this with AWS Athena)
Maybe even some CSVs someone refuses to migrate

Foreign tables give you a unified SQL interface, but under the hood, the query performance can be unpredictable as you may be forced to rely on another engine’s query planner (and in the case of that CSV data source, it might not even be indexed).

In other words, FDWs optimize developer experience, not query performance.

The Pattern: FDW + Materialized Views

Instead of querying foreign tables directly in analytics workloads, we can opt to use FDWs as ingestion points, not as the serving layer itself. To achieve this, we can do the following:

Step 1: Define the foreign table

CREATE FOREIGN TABLE ext_orders (
  id bigint,
  customer_id bigint,
  total numeric,
  created_at timestamp
)
SERVER mysql_server
OPTIONS (table 'orders');

Step 2: Build a materialized view

CREATE MATERIALIZED VIEW orders_mv AS
SELECT
  id,
  customer_id,
  total,
  created_at::date AS order_date
FROM ext_orders;

Step 3: Index it like a real table

CREATE INDEX ON orders_mv (order_date);
CREATE INDEX ON orders_mv (customer_id);

Now we’ve turned a slow, remote dataset into a locally optimized analytical structure.

The materialized view lives inside PostgreSQL, supports full indexing, eliminates network latency during queries, and gives predictable performance. We essentially have a read-optimized cache on top of the foreign tables. We can do this with the read-only Postgres replicas as well, to slice up the columns and rows to fit nicely in a view that analysts would want to use.

Refreshing Without Blocking

When it comes to caching, data gets stale, and we’re sort of back at the same problem every ETL pipeline faces. However, Postgres can refresh a materialized view without blocking users, simply with the CONCURRENTLY syntax. This results in production-quality data with a little bit of staleness, but the nice thing is that it’s all built-in to the Postgres cluster (no separate ETL pipeline to manage, just all the data accessible from one central place). Note, however that in order to use the CONCURRENTLY syntax, a UNIQUE key is required.

Good Applications for the Pairing

The pairing of FDWs and indexed Materialized Views could be very beneficial in a handful of use cases:

1. Poorly Indexed Remote Systems

If your upstream system:

Lacks proper indexes
Is shared with OLTP workloads
Is not under your control

This approach isolates analytics from those constraints.

2. High-Latency Data Sources

Examples:

Cross-region databases
Cloud object storage via FDWs
Athena-backed datasets

Instead of paying the latency cost on every query, you pay it once per refresh.

3. Flat Files and Large Data

Yes, people do this:

Querying CSVs via FDWs
Treating object storage as a “database”
Large JSONB sets that are hard to index well

Final Thoughts

Foreign tables aren’t just novelty – they’re a powerful bridge across messy, real-world data systems.

It is important to distinguish that FDWs make a data access layer, while Materialized Views are the analytics engine. If you 1) layer materialized views on top of FDWs, 2) add proper indexing, and 3) refresh intelligently (preferably concurrently), you can get the best of both worlds: flexibility of federated queries and performance of local analytics.

XID Wraparound's Equally-Evil Twin

Mon, 18 May 2026 06:00:00 +0000

Introduction

If you’ve been running PostgreSQL for any length of time, you’ve probably heard about transaction ID (XID) wraparound. It’s one of the most well-known maintenance concerns in Postgres, and there’s no shortage of blog posts, conference talks, and war stories about it. But there’s a quieter, less-discussed cousin that can cause the exact same kind of outage: MultiXact ID wraparound.

I’ve seen this surprise more than a few experienced DBAs. They’ve got their autovacuum tuned, they’re monitoring age(datfrozenxid), and they’re feeling good – and then out of nowhere, Postgres starts refusing certain writes because it’s approaching MultiXact ID wraparound.

The fix is the same as regular XID wraparound – a simple vacuum. But the reason is different, and understanding it can help you keep your monitoring complete.

What’s a MultiXact ID?

In Postgres, every row has a system column called xmax. In the simplest case, xmax holds the transaction ID of the transaction that deleted or updated the row. But what happens when multiple transactions hold locks on the same row at the same time?

Consider SELECT ... FOR SHARE. Multiple transactions can hold a shared lock on the same row concurrently. Postgres needs to record all of those transactions somewhere, but xmax is only wide enough to store a single transaction ID. The solution is the MultiXact mechanism.

A MultiXact ID is essentially a pointer into a separate structure (stored as a file in the pg_multixact/ dir) that maps to a list of transaction IDs and their lock modes. When multiple transactions need to lock a row, Postgres:

Allocates a new MultiXact ID
Records the set of transaction IDs (and their lock types) in the MultiXact member data
Stores the MultiXact ID in the row’s xmax field, with a flag (specifically, the HEAP_XMAX_IS_MULTI infomask bit in the tuple header) indicating it’s a multi-xact reference rather than a plain XID

This lets the xmax field stay a fixed 32-bit value while still representing an arbitrary number of concurrent row-level lockers.

When Are MultiXact IDs Created?

MultiXact IDs come into play in several scenarios:

SELECT ... FOR SHARE – The classic case. Multiple transactions can hold shared row locks simultaneously.
SELECT ... FOR KEY SHARE – Used implicitly by foreign key checks. If you have a parent table with foreign key references, every insert or update on the child table takes a FOR KEY SHARE lock on the referenced parent row. On a busy system with many concurrent inserts referencing the same parent rows, this generates MultiXact IDs rapidly.
Combination locks – If one transaction holds a FOR KEY SHARE lock and another holds a FOR NO KEY UPDATE lock on the same row, the two locks don’t conflict, and the resulting multi-lock is stored as a MultiXact.

The foreign key scenario is particularly noteworthy because it’s invisible to most application developers. You won’t see any queries explicitly calling out FOR SHARE in application code, but Postgres is silently creating MultiXact IDs behind the scenes to manage the implicit locks.

MultiXact IDs Need Freezing Too!

Just like transaction IDs, MultiXact IDs are 32-bit counters. And just like XIDs, they wrap around. Postgres can only “see” about 2 billion MultiXact IDs into the past. If a row still references a MultiXact ID that’s about to fall off the visible horizon, Postgres has a problem: it can no longer determine whether the locks represented by that MultiXact are still relevant.

To prevent this, Postgres needs to freeze MultiXact IDs, just as it freezes regular XIDs. Freezing a MultiXact means replacing the MultiXact reference in the row’s xmax with either the zero value, a single transaction ID, or a newer multixact ID, depending on whether the lock information is still meaningful.

The relevant settings mirror those for XID freezing:

XID Setting	MultiXact Equivalent
`vacuum_freeze_min_age`	`vacuum_multixact_freeze_min_age`
`vacuum_freeze_table_age`	`vacuum_multixact_freeze_table_age`
`autovacuum_freeze_max_age`	`autovacuum_multixact_freeze_max_age`

When the MultiXact age of a table exceeds autovacuum_multixact_freeze_max_age, autovacuum will trigger an aggressive (whole-table) vacuum specifically to freeze old MultiXact IDs – even if the table has no dead tuples and wouldn’t otherwise qualify for autovacuum.

Don’t Let MultiXact Fly Under the Radar

The query is straightforward:

SELECT datname,
       age(datfrozenxid) AS xid_age,
       mxid_age(datminmxid) AS mxid_age
  FROM pg_database
 ORDER BY mxid_age DESC;

For per-table granularity:

SELECT c.oid::regclass AS table_name,
       age(c.relfrozenxid) AS xid_age,
       mxid_age(c.relminmxid) AS mxid_age
  FROM pg_class c
 WHERE c.relkind IN ('r', 't', 'm')
 ORDER BY mxid_age DESC
 LIMIT 20;

Keep an eye on any table where mxid_age is approaching autovacuum_multixact_freeze_max_age (default: 400 million). If it gets close, autovacuum should kick in, but on large tables or systems with constrained autovacuum workers, it may not complete in time.

Practical Recommendations

Add MultiXact monitoring alongside XID monitoring. If your alerting triggers at, say, 500 million XID age, add a similar alert for MultiXact age.
Watch your foreign key parent tables. If you have a users or accounts table that’s referenced by every other table in the schema, it’s likely accumulating MultiXact IDs faster than you’d expect.
Consider autovacuum_multixact_freeze_max_age tuning. The default of 400 million is higher than the XID autovacuum_freeze_max_age default of 200 million. But in workloads with heavy foreign key activity, you may want to lower it – or configure per-table autovacuum settings on hot parent tables.
Don’t ignore “unnecessary” vacuums. If you see autovacuum running on a table that has zero dead tuples, don’t assume it’s wasting resources. It may be performing MultiXact freezing work that’s critical for preventing wraparound.

Conclusion

MultiXact ID wraparound is the kind of problem that bites you precisely because you didn’t know to look for it. The mechanism exists for a good reason – efficiently tracking shared row locks is fundamental to Postgres’s concurrency model. But the maintenance burden it creates is real, and it demands the same vigilance as XID wraparound.

If you take one thing away from this post: go check mxid_age(datminmxid) on your databases today. If you’ve never looked at it before, now’s a good time to start.

Making JSONB More Queryable with Generated Columns

Mon, 11 May 2026 06:00:00 +0000

Introduction

Over the past year, I’ve worked in a handful of contexts managing large volumes of data stored as JSONB in PostgreSQL. The scenario is common: users appreciate the flexibility of a document-oriented storage model, avoiding the need to predefine schemas or constantly migrate table structures as their data requirements evolve. JSONB documents can be deeply nested with numerous optional fields, and they scale to hundreds of kilobytes per record without issue. However, when the time comes to query these documents – filtering by user ID, event type, timestamps, or nested action properties – the queries can become slow and/or cumbersome to work with.

The problem I want to address is: “How do we make searching JSONB data more efficient without breaking apart our documents or forcing it into columns in a relational database?” There are several approaches available in Postgres, each with different tradeoffs. I hope to shed some light on those approaches in this article.

The Setup

I created a basic, no-frills table for the sake of this test:

CREATE TABLE events (
    id BIGSERIAL PRIMARY KEY,
    data JSONB NOT NULL
);

Here's the document shape I used for testing and writing this post -- it's representative of the event logs and audit trails I've encountered: a mix of primitive fields, nested objects, and metadata that accumulates over time.

-- Representative JSONB document
{
  "user_id": 5234,
  "event_type": "event_42",
  "timestamp": 1712341200,
  "session_id": "sess_abc123...",
  "ip_address": "192.168.1.42",
  "action": {
    "type": "click",
    "target_id": 87654,
    "coordinates": {"x": 512, "y": 768},
    "duration_ms": 1234
  },
  "device": {
    "type": "mobile",
    "os": "iOS",
    "screen_width": 1920,
    "screen_height": 1080
  },
  "performance": {
    "page_load_time": 1234,
    "dns_lookup": 123,
    "tcp_connection": 234,
    "server_response": 876
  },
  "custom_fields": { ... }
}

The queries that matter are straightforward equality and range filters on known fields: find all events for a given user, filter by event type, narrow to a time window. With this setup, we’ll try to discern which kind of index actually serves the specific access pattern, and what the real cost of each option is.

All tests run on PostgreSQL 18.2 in Docker on an Apple M-series host. Tables contain 50,000 rows with realistic JSONB event documents. Query benchmarks run 20 times on a warm cache and report avg/min/max. Insert benchmarks run 5 trials of 5,000 rows each. Schema and scripts are included throughout so you can reproduce these results.

Three Approaches to Indexing JSONB

There are three realistic options for this access pattern. Let’s look at each in turn – what it costs to build/maintain, what queries it actually helps, and where it falls down.

Option 1: GIN Indexes

The natural candidate for indexing a JSONB column would be a GIN (Generalized Inverted Index) index. After all, GIN indexes are specifically designed for JSON documents and full-text search. It indexes every key and value pair in every document, making the entire structure searchable:

CREATE INDEX idx_gin ON events USING GIN (data);
-- or the path-only variant:
CREATE INDEX idx_gin_path ON events USING GIN (data jsonb_path_ops);

As a refresher, I’ll mention that GIN is designed for containment and key existence operators (@>, ?, ?|, ?&), not for equality on extracted fields:

-- This query uses a GIN index correctly:
SELECT id FROM events WHERE data @> '{"user_id": 5234}';

-- This query does NOT use a GIN index, even if one exists:
SELECT id FROM events WHERE cast(data->>'user_id' AS INT) = 5234;

For the containment form, the GIN index is used and the query is fast – but still slower than a B-tree on the same field, because GIN lookups involve more bookkeeping:

-- GIN jsonb_ops + containment operator
Bitmap Index Scan on idx_gin
  Index Cond: (data @> '{"user_id": 5234}')

lanning Time: 1.173 ms  |  Execution Time: 1.295 ms

-- GIN jsonb_path_ops + containment operator
Bitmap Index Scan on idx_gin_path
  Index Cond: (data @> '{"user_id": 5234}')
Planning Time: 3.342 ms  |  Execution Time: 0.450 ms

The jsonb_path_ops variant is smaller and faster for containment queries, but it trades away support for key-existence operators (?, ?|, ?&). Neither GIN variant can help with range predicates like ts > 1700000000 – those always fall through to a filter step.

Option 2: Expression Indexes

Postgres lets you create an index on an expression, including JSONB extraction:

CREATE INDEX idx_user_id ON events (cast(data->>'user_id' AS INT));

This is a B-tree index on the result of evaluating the expression. When the query predicate matches the indexed expression exactly, and after ANALYZE has gathered statistics on it, the planner will use it:

SELECT id FROM events
WHERE cast(data->>'user_id' AS INT) = 5234;

Bitmap Heap Scan on t_expr
  Recheck Cond: ((data ->> 'user_id')::integer = 5234)
  Heap Blocks: exact=3
  ->  Bitmap Index Scan on idx_user_id
        Index Cond: ((data ->> 'user_id')::integer = 5234)
Planning Time: 1.168 ms  |  Execution Time: 0.341 ms

The execution time on this equality operator seems to be pretty similar to the performance of the GIN index.

Option 3: Generated Columns

Generated columns (available since PostgreSQL 12) let you extract JSONB values into regular typed columns at write time. The values are stored physically alongside the row and kept in sync automatically:

CREATE TABLE events (
    id         BIGSERIAL PRIMARY KEY,
    data       JSONB NOT NULL,
    user_id    INT    GENERATED ALWAYS AS ((data->>'user_id')::INT)    STORED,
    event_type TEXT   GENERATED ALWAYS AS (data->>'event_type')        STORED,
    ts         BIGINT GENERATED ALWAYS AS ((data->>'timestamp')::BIGINT) STORED,
    action     TEXT   GENERATED ALWAYS AS (data->'action'->>'type')    STORED
);

CREATE INDEX idx_user_id ON events (user_id);
CREATE INDEX idx_event_type ON events (event_type);
CREATE INDEX idx_ts ON events (ts);
CREATE INDEX idx_action ON events (action);

Queries against generated columns are plain typed-column lookups. The planner sees them as regular B-tree columns and produces tight estimates:

SELECT id FROM events WHERE user_id = 5234;

Bitmap Heap Scan on t_gen
  Recheck Cond: (user_id = 5234)
  Heap Blocks: exact=3
  ->  Bitmap Index Scan on idx_user_id
        Index Cond: (user_id = 5234)
Planning Time: 1.159 ms  |  Execution Time: 0.407 ms

You also get native support for range queries and composite indexes at no extra complexity – just combine columns as you normally would:

-- Indexed range query on generated timestamp column
CREATE INDEX ON events (event_type, ts);

SELECT id FROM events
WHERE event_type = 'event_42' AND ts > 1700000000;
-- Execution Time: 0.698 ms (vs 6.6 ms with GIN + post-filter)

Side-by-Side: Query Performance

With all three approaches set up, here are the warm-cache query results averaged over 20 runs for an equality filter on user_id:

Approach	Avg (ms)	Min (ms)	Max (ms)
GIN jsonb_ops + `@>`	0.198	0.101	1.769
GIN jsonb_path_ops + `@>`	0.197	0.032	3.115
Expression index	0.106	0.018	1.705
Generated column B-tree	0.112	0.016	1.839

Expression indexes and generated columns perform very similarly for equality queries—both around 0.1ms on warm cache. The real work is done in the B-tree lookup and both produce the same index structure. GIN with the correct @> operator is nearly as fast in PG 18.2 – still slightly slower than B-tree for this access pattern, but the gap has narrowed. GIN lookups still require a recheck step that B-tree lookups avoid, and the variance remains notable: GIN max of 3.1ms vs B-tree max of 1.8ms on warm cache.

The more surprising result is what happens if the GIN index is present but the query is written with extraction-based equality:

-- GIN index exists, but this query gets a seq scan:
SELECT id FROM events WHERE cast(data->>'user_id' AS INT) = 5234;
-- Execution Time: 47.935 ms (same as no index at all)

GIN doesn’t support that operator class. This is by far the most common confusion teams run into with JSONB indexing.

The Full Cost Picture: Storage and Writes

Storage

Here’s what the same 50,000 rows cost on disk under each approach:

Approach	Table size	Index size	Total
Expression indexes (4)	18 MB	3.5 MB	21 MB
Generated columns + B-tree (4)	20 MB	3.5 MB	23 MB
GIN jsonb_path_ops	18 MB	13 MB	31 MB
GIN jsonb_ops	18 MB	18 MB	36 MB

Expression indexes and generated column B-tree indexes produce identical index sizes for the same fields – this makes sense, since the index structures are the same; the only extra cost of generated columns is the 2 MB of additional stored column data in the table (~40 bytes per row for four typed columns). GIN indexes are substantially larger: 13–18 MB for a single index vs 3.5 MB for four targeted B-tree indexes. The jsonb_path_ops variant is smaller because it only stores value hashes for the @> operator path, but it still dwarfs the targeted approach.

One caveat: these numbers reflect documents with short keys and compact values. Documents with verbose key names, deeply nested structures, or large string values will inflate GIN indexes proportionally more – because GIN indexes every key path. B-tree and expression indexes are unaffected by document verbosity, since they only store the extracted value.

Write Throughput

Here’s what 5,000 INSERTs per trial, 5 trials each, on a table already containing 50,000 rows looked like:

Approach	Avg (ms)	Min (ms)	Max (ms)
Generated columns + B-tree (4)	157	91	317
Expression indexes (4)	163	93	366
GIN jsonb_path_ops	171	73	408
GIN jsonb_ops	334	225	525

Generated columns and expression indexes are now very close in write cost, with generated columns slightly edging out on average. GIN jsonb_path_ops has become more competitive with both. However, the default GIN jsonb_ops variant is dramatically more expensive: 2× slower than expression indexes and generated columns. It must decompose the entire document into key-value pairs and insert entries for each one. The high variance is also worth noting: GIN jsonb_ops max of 525ms vs 366ms for expression indexes.

Choosing the Right Approach

The benchmarks above tell a consistent story for workloads dominated by equality and range filters on a known set of fields:

Expression indexes are the lowest-cost migration path. They add no schema structure, require no application changes to insert logic, and impose minimal write overhead. If your team already has a table in production and just needs to speed up a handful of known slow queries, a well-placed expression index is your first move. The catch: every query must exactly match the expression as written in the index definition, which can be fragile to maintain as codebases evolve.
Generated columns take slightly more storage and impose more write overhead than expression indexes, but they offer something the others can’t: the extracted values become first-class columns. You can build composite indexes across them, reference them in views, expose them via ORMs, and sort or aggregate on them without embedding extraction logic everywhere. For new tables or for tables you’re willing to migrate, they’re the most maintainable long-term solution.
GIN indexes serve a different purpose. They’re the right tool when your query patterns are flexible or unknown – searching for the existence of a key, filtering on any field in an ad-hoc fashion, or supporting containment queries on arbitrarily-shaped documents. For those access patterns, they’re genuinely powerful and there’s no clean B-tree equivalent. But for consistent equality and range filters on known fields, they cost more in storage, impose higher write latency, and only work with one operator class (@>, not =).

Here’s a rough decision guide:

Situation	Recommended approach
Unknown or ad-hoc field queries	GIN (`@>`, key existence)
Known fields, few queries, no schema change	Expression index
Known fields, high query volume, evolving codebase	Generated columns
Known fields + range queries (e.g., timestamps)	Generated columns + composite B-tree
Mixed: some known fields + some ad-hoc	Generated columns + GIN (both)

Caveats and Considerations

Regardless of which approach you choose, a few things apply broadly:

The real win is making data typed and relational again. Generated columns aren’t magic. The reason they (and expression indexes) outperform GIN for equality filters is that they produce typed scalar values with precise statistics, letting the planner make accurate row-count estimates and choose cheap comparison operations. JSONB is flexible but opaque; once you extract a field into a typed column or expression, Postgres can reason about it properly.

Expression indexes require exact predicate matching. An index on cast(data->>'user_id' AS INT) will not be used by a query written as (data->>'user_id')::int. The cast form must be identical. Generated columns avoid this fragility – any query that references the column name will benefit.

Generated column expressions must be immutable. The expression cannot reference functions that depend on time, session state, or anything external. NOW(), CURRENT_USER, and similar functions are off-limits.

Generated columns cannot be directly updated. Their value is always derived from the source column. If you UPDATE the JSONB data, the generated columns recompute automatically.

GIN maintenance overhead compounds on write-heavy tables. GIN indexes build an internal pending list and flush it periodically (controlled by gin_pending_list_limit). Under sustained write load, this flushing can cause the latency spikes visible in the benchmark max values above. B-tree indexes don’t have this mechanism.

These benchmarks cover one dataset shape and one machine. At much larger row counts (hundreds of millions), cache-miss behavior and index bloat will dominate—relative rankings should hold, but absolute numbers will differ. When in doubt, benchmark on your own data before committing to a migration.

Conclusion

For workloads dominated by equality and range filters on a predictable set of JSONB fields, the data is clear: B-tree indexes on typed values – whether via expression indexes or generated columns – outperform GIN both on read latency and write throughput. GIN’s strength is flexibility, not speed for known-field access patterns; when you know exactly which fields you’ll filter on, a targeted B-tree beats the GIN every time.

If you’re starting from scratch or are willing to migrate a table, generated columns are the most maintainable path. They make your frequently-queried fields easily accessible, eliminate JSONB extraction logic from your application’s query layer, and support composite indexes and range queries naturally. If you need to add indexing to an existing table without a schema change, expression indexes get you 90% of the way there with a fraction of the write overhead.

GIN still belongs in your toolkit – but for the right job: ad-hoc containment searches, key-existence checks, and cases where the query patterns genuinely vary by document. For everything else, make your JSONB fields relational.

Potential Consequences of Using Postgres as a Job Queue

Mon, 04 May 2026 06:00:00 +0000

This post was originally published on the Microsoft Tech Community Blog.

Introduction

At small scale, using Postgres as a job queue is totally fine, and I’d even say it’s the right call. Fewer moving parts, one less system to manage, ACID guarantees on your jobs. What’s not to love?

The problem is that “small scale” has a ceiling, and the ceiling is lower than most people expect. When you’ve got thousands of concurrent workers hammering a jobs table with SELECT ... FOR UPDATE SKIP LOCKED, things start to behave in ways that aren’t obvious from the application layer. CPU usage creeps up. Also vacuum sometimes can’t keep up. Finally, in the wait event stats, you start seeing ominous entries like LWLock:MultiXactSLRU stacking up across many backends.

This pattern has tripped up teams more than a few times, and it usually plays out the same way: everything works fine in dev and staging, then goes off a cliff in production once the concurrency gets real. So let’s dig into why this happens, and what the alternatives look like.

The Typical Pattern

When using Postgres as a job queue, the standard approach looks something like this:

CREATE TABLE job_queue (
    id         bigserial PRIMARY KEY,
    status     text NOT NULL DEFAULT 'pending',
    payload    jsonb NOT NULL,
    created_at timestamptz NOT NULL DEFAULT now(),
    locked_by  text,
    locked_at  timestamptz
);

CREATE INDEX idx_job_queue_status ON job_queue (status) WHERE status = 'pending';

Workers grab jobs with:

UPDATE job_queue
   SET status = 'processing',
       locked_by = 'worker-42',
       locked_at = now()
 WHERE id = (
     SELECT id FROM job_queue
      WHERE status = 'pending'
      ORDER BY created_at
      LIMIT 1
        FOR UPDATE SKIP LOCKED
 )
 RETURNING *;

And then mark them done:

UPDATE job_queue SET status = 'completed' WHERE id = $1;

Some users may DELETE the row entirely. Either way, the lifecycle is: insert, lock-and-update, update-or-delete. Repeated thousands of times per second.

At low concurrency, this works very smoothly. SKIP LOCKED means workers don’t block each other waiting for the same row. Postgres handles the locking, visibility, and ordering. It’s elegant.

So where does it break?

The MultiXact SLRU Problem

When multiple transactions hold locks on the same row, Postgres stores the set of lockers as a MultiXact ID – a pointer into a side structure under pg_multixact/.

With SELECT ... FOR UPDATE SKIP LOCKED, users might think MultiXacts aren’t involved – after all, SKIP LOCKED is supposed to avoid contention. But in practice, with many concurrent workers all racing to lock rows, there are brief windows where multiple transactions reference the same row before one of them “wins” and the others skip. If you combine this with any FOR SHARE or FOR KEY SHARE locks (which are commonly created implicitly by foreign key checks), MultiXact IDs start accumulating quickly.

The MultiXact data lives in SLRU buffers (Simple Least Recently Used) – a small, fixed-size shared memory cache. When backends need to read or write MultiXact data, they acquire LWLocks to access these buffers. Under high concurrency, this becomes a bottleneck:

wait_event_type | wait_event
-----------------+-------------------
LWLock          | MultiXactMemberSLRU
LWLock          | MultiXactOffsetSLRU

You’ll see dozens or hundreds of backends piled up on these waits. The SLRU cache is small (by design – it’s a fixed number of pages in shared memory), and when the working set of MultiXact lookups exceeds what fits in the cache, you get constant eviction and re-reads from disk. Every lock acquisition and release on a job row potentially triggers a MultiXact SLRU lookup, and at thousands of concurrent sessions, those lookups serialize on LWLocks.

The result: CPU gets pegged, throughput collapses, and latency spikes – not because the queries are expensive, but because the locking infrastructure itself is overwhelmed.

Bloat: The Silent Killer

The other side of this coin is table and index bloat. Every job row goes through multiple updates (and possibly a delete), and each of those operations creates a new tuple version in the heap. The old versions stick around until VACUUM cleans them up.

On a busy job queue table:

Dead tuples accumulate faster than autovacuum can clean them. By the time autovacuum finishes one pass, tens of thousands of new dead tuples have appeared. The table grows and grows.
Index bloat compounds the problem. Every index on the table also accumulates dead entries. The partial index on status = 'pending' gets thrashed especially hard, since rows constantly enter and leave that condition.
Sequential scans get slower. As the table bloats, even index scans start doing more I/O because the heap pages are sparsely populated. Vacuum reclaims space at the end of the table, but can’t reclaim space in the middle (unless the pages are completely empty).

Job queue tables can grow to tens of gigabytes when the actual “live” data was only a few megabytes. It makes everything slower: scans, vacuum, even pg_dump.

You can mitigate this by running vacuum more aggressively (lower autovacuum_vacuum_scale_factor, higher autovacuum_vacuum_cost_limit), or by partitioning the table and dropping old partitions. But at some point, you’re fighting the fundamental mismatch between MVCC’s design goals and the write pattern of a job queue.

CPU and Lock Overhead

Beyond the SLRU contention and bloat, there’s just the raw overhead of using Postgres’s full transactional machinery for what is essentially a FIFO dispatch operation:

Every lock/unlock is a full WAL-logged transaction. Grabbing a job writes WAL. Marking it complete writes WAL. Deleting it writes WAL. On a system processing thousands of jobs per second, the WAL volume from the job queue alone can saturate your wal_writer and checkpoint processes.
SKIP LOCKED still touches rows. The name suggests rows are skipped, but Postgres still has to find them, check their lock status, and move on. With high concurrency, many workers end up scanning past the same locked rows before finding one they can claim. This is wasted CPU.
Snapshot management overhead also becomes an issue. Each transaction needs a consistent snapshot, and with thousands of concurrent transactions, the ProcArray (the structure that tracks active transactions) becomes a contention point itself. You might see LWLock:ProcArrayLock waits alongside the MultiXact ones.
Vacuum contention. While vacuum is cleaning up dead tuples, it needs locks too. On a table under constant write pressure, vacuum can interfere with the workers and vice versa. I’ve seen systems where disabling autovacuum on the job queue table improved throughput in the short term.

Better Alternatives

So what should you use instead? It depends on your requirements, but there are several options that handle high-throughput job dispatch more gracefully than a Postgres table.

Advisory Locks (Staying in Postgres)

If you want to stay within Postgres and avoid adding infrastructure, advisory locks are worth considering for certain queue patterns. Instead of locking rows, you lock on an abstract numeric key:

-- Worker tries to acquire a lock on the job ID
SELECT pg_try_advisory_lock(id) FROM job_queue
 WHERE status = 'pending'
 ORDER BY created_at
 LIMIT 1;

Advisory locks are lightweight – they don’t touch the heap, don’t create MultiXact entries, and don’t generate dead tuples. They live entirely in shared memory. The trade-off is that you lose the atomicity of FOR UPDATE SKIP LOCKED: you need to handle the case where a lock is acquired but the job processing fails, and you need to release the lock explicitly (or rely on session-end cleanup).

This approach works well when the queue depth is manageable and you want to avoid the MVCC overhead. But it’s still Postgres, so you’re still subject to connection limits, ProcArray overhead, and general resource contention at very high session counts.

pgq (Skytools)

pgq is purpose-built for exactly this problem. It’s a queue implementation that sits inside Postgres but uses a batching model that avoids most of the row-level locking and MVCC pitfalls. Events are written to a queue table, but consumers read them in batches and the queue maintenance is done via a ticker process that manages rotation.

The key advantages:

No row-level contention. Consumers don’t lock individual rows.
Built-in batch processing. Events are consumed in chunks, reducing transaction overhead.
Efficient cleanup. Old events are rotated out rather than vacuumed row-by-row.

The downside is that pgq is not as actively maintained as it once was, and it adds operational complexity (the ticker daemon, consumer registration, etc.). But for teams already deep in the Postgres ecosystem, it’s a battle-tested option.

PgQue

Coincidentally, during the writing of this post, Nikolay Samokhvalov has built PgQue, which is a derivative of pgq. Like pgq, it sits inside Postgres, but ships as a single SQL file – no C extension and no external daemon – making it deployable on managed services like RDS, Aurora, Cloud SQL, AlloyDB, Supabase, and Neon. Producers INSERT events into rotating event tables (recycled via TRUNCATE instead of row-by-row deletion), and consumers read batches by diffing two pg_snapshot values captured by a periodic ticker – so the hot path contains zero UPDATEs, DELETEs, or SELECT ... FOR UPDATE SKIP LOCKED, and therefore produces no dead tuples on the event tables. For a deeper dive into the algorithm, see Christophe Pettus’s writeup.

Redis

For many teams, Redis is the natural choice for job queues. Using Redis lists (BRPOPLPUSH or the Streams API), you get:

Sub-millisecond dispatch latency. No disk I/O, no MVCC, no vacuum.
Atomic pop operations. Workers grab jobs without any locking protocol.
Simple scaling. Redis handles thousands of concurrent consumers trivially.

The trade-off is durability. Redis can persist to disk, but it’s not ACID. If Redis crashes between a pop and the job completing, you might lose or duplicate work (though Redis Streams with consumer groups mitigate this significantly). For most job queue use cases, at-least-once delivery is acceptable, and Redis does that well.

Kafka

For truly high-throughput, distributed workloads, Apache Kafka is the heavyweight option. Kafka partitions give you parallel consumption with ordering guarantees per partition, durable storage, and replay capability. It’s the right tool when:

You need to process thousands of events per second
Multiple consumers need to read the same events
You want event replay or audit trails
Your architecture is already event-driven

The operational overhead is nontrivial – ZooKeeper (or KRaft), brokers, topic management, consumer group coordination. But for teams already running Kafka for other reasons, adding a job queue topic is practically free.

Choosing the Right Tool

Here’s a rough decision guide:

Scenario	Recommendation
Under 100 concurrent workers, simple jobs	Postgres with `SKIP LOCKED` is fine
Moderate concurrency, want to stay in Postgres	Advisory locks or pgq
High throughput, low-latency dispatch	Redis (Lists or Streams)
Massive scale, distributed, event replay	Kafka

Many teams that start with Postgres (reasonably) hit scaling problems and then try to fix Postgres rather than recognizing that the workload has outgrown the tool. They throw more autovacuum workers at it, increase max_connections, add connection poolers – all of which help at the margins, but don’t address the fundamental issue: Postgres’s MVCC and locking machinery wasn’t designed for this access pattern at high concurrency.

Conclusion

Postgres is great, but it can’t be the best tool for every job. Using it as a job queue is a perfectly valid choice when your scale is modest. But when you’re running thousands of concurrent workers, the combination of MultiXact SLRU contention, heap bloat, vacuum pressure, and raw locking overhead will eventually push you toward a purpose-built solution.

The good news is that you don’t have to rip out everything. Advisory locks can buy you headroom without adding infrastructure. Redis can handle dispatch while Postgres keeps owning the data. And if you’re already using Kafka, a job topic is a natural fit. Take your pick – there are many queueing options out there!

Understanding Bitmap Heap Scans in PostgreSQL

Mon, 27 Apr 2026 08:00:00 +0000

Introduction

When people first start reading PostgreSQL execution plans, they quickly learn a few common scan types: Seq Scan, Index Scan, Index Only Scan. But eventually another one appears that is less obvious: Bitmap Heap Scan, which is almost always accompanied by Bitmap Index Scan.

At first glance, it sounds like two scans on the same table – a very inefficient choice?! But bitmap scans are actually one of the planner’s most practical tools for balancing random I/O vs sequential access. Understanding how they work can make execution plans much easier to interpret, so we’ll dive into that a little bit today.

The Basic Idea

A bitmap scan is a two-step process:

Step 1: Build a bitmap of matching rows using one or more indexes.

Step 2: Visit the heap pages containing those rows referenced in the bitmap.

In an execution plan this usually appears as:

Bitmap Heap Scan on orders
-> Bitmap Index Scan on orders_customer_id_idx

The important part is that the index lookup and heap access are separated – this separation allows Postgres to explain heap access costs and actuals more clearly.

Why Not Just Use an Index Scan?

With a normal index scan, the query executor does something like this:

Find a matching entry in the index
Jump to the heap page
Fetch the row
Repeat

If the query returns only a few rows, this works well. But if the query returns thousands of rows scattered across the table, the database ends up doing many random heap fetches. Random I/O can become expensive, so a bitmap scan solves this problem.

How the Bitmap Is Built

During the Bitmap Index Scan phase, the executor does not immediately fetch rows. Instead it records which heap pages contain matching rows. Conceptually, the structure looks like this:

Page 101 -> rows 2, 7
Page 205 -> rows 1, 3, 8
Page 410 -> row 5

These page references are stored as a bitmap structure in memory. Once the bitmap is complete, the executor can visit heap pages in physical order rather than jumping around randomly. Visiting heap pages in physical order means less random I/O and therefore less latency.

Multiple Indexes Can Be Combined

One particularly powerful feature is that bitmap scans allow the query planner to combine multiple indexes. For example:

WHERE status = 'active'
AND created_at >= '2025-01-01'

The plan might look like:

Bitmap Heap Scan
-> BitmapAnd
-> Bitmap Index Scan on status_idx
-> Bitmap Index Scan on created_at_idx

Each index produces a bitmap, and the planner combines them using logical operations, such as BitmapAnd and BitmapOr. This allows the planner to efficiently use multiple indexes even when a single composite index does not exist.

When Does the Planner Chooses Bitmap Scans?

The planner usually prefers bitmap scans in situations where the query returns more rows than a typical index scan, but not enough rows to justify a full sequential scan. In other words, bitmap scans often appear in the middle selectivity range.

Very roughly:

Selectivity	Likely Plan
Very small	Index Scan
Medium	Bitmap Heap Scan
Very large	Seq Scan

This is not a strict rule, but it helps explain the planner’s reasoning.

Pros and Cons

As with everything in databases, there’s no free lunch. Here are some advantages and disadvantages for bitmap scans

Advantages of Bitmap Heap Scans
- Reduced Random I/O: By grouping heap page accesses, bitmap scans avoid excessive random disk reads.
- Ability to Combine Indexes: Bitmap operations allow the query planner to use multiple independent indexes efficiently.
- Better Performance for Medium Selectivity: Queries returning thousands of rows often benefit from bitmap access patterns.
- Predictable Heap Access: Because heap pages are visited in order, caching behavior tends to improve.
Disadvantages of Bitmap Heap Scans
- Memory Usage: The bitmap structure is stored in memory. If the result set becomes too large, the query executor may switch to a lossy bitmap, where only page-level information is stored. This can cause additional filtering work later.
- Two-Phase Execution: Because the bitmap must be built before heap access begins, the query cannot stream rows immediately. This can increase latency for queries expecting early rows.
- Extra CPU Work: Maintaining and combining bitmap structures adds overhead compared to simple index scans.

Lossy Bitmaps

When memory limits are reached, the query executor may degrade the bitmap representation. Instead of tracking individual tuple offsets, it only records:

Page 205 -> possible matches

During the heap scan, the executor must then recheck all rows on that page. In execution plans you may see mention of Recheck Cond. This indicates that the bitmap became lossy. While still correct, this can reduce efficiency.

Final Thoughts

Bitmap heap scans are one of the planner’s most practical optimization tools, as they allow the database to reduce random I/O, combine multiple indexes, and handle medium-sized result sets efficiently.

While they may look complicated at first, the core idea is simple: Find matching rows first, then fetch heap pages efficiently. What a great concept!

Richard Yen

Are You .ready?

A practical guide to what .ready and .done mean, and why WAL sticks around

Introduction

Hint: It’s About WAL Delivery

What archive_command Actually Does

What .ready and .done Are For

.ready

.done

If .done Exists, When Is It Deleted?

What If .done Files Pile Up and Do Not Disappear?

How This Relates to Streaming Replication

What About pg_receivewal?

Replication Slots: Where People Get Confused

Beginner-Friendly Checks

Final Thoughts

Disaster Recovery is a Process, Not a Tool (Part 2)

Picking Up Where We Left Off

Runbook Engineering: Why Runbooks Fail

Anti-patterns

What a good runbook looks like

Non-technical essentials

Validation

Game Days

What to Measure

Don’t Blame, or You’ll Feel Lame

Wrapping Up

pg_stats: How Postgres Internal Stats Work

Introduction

Query Plans Are Made by the Query Planner

ANALYZE: How the Summary Gets Built

Most Common Values (MCV)

Costs: How the Planner Picks a Plan

A Tale of Two States

Histograms: For Everything That Isn’t Equality

Correlation Between Columns

A Quick Checklist for Query Performance

Conclusion

Disaster Recovery is a Process, Not a Tool (Part 1)

The Landscape Has Changed

What Counts as a Disaster?

How DR Is Usually Done

Postgres Already Gives Us Most of the Tools

RPO and RTO, and Why They’re Negotiations

Three Layers of DR Planning

Recovery Isn’t Always About Failing Over

What Lower Numbers Actually Cost

To Be Continued

PGDay Boston 2026

Introduction

Foreign Tables and Materialized Views: A Dynamic Duo

Introduction

Moving Beyond FDW Demos

The Real Problem: Heterogeneous Data

The Pattern: FDW + Materialized Views

Step 1: Define the foreign table

Step 2: Build a materialized view

Step 3: Index it like a real table

Refreshing Without Blocking

Good Applications for the Pairing

1. Poorly Indexed Remote Systems

2. High-Latency Data Sources

3. Flat Files and Large Data

Final Thoughts

XID Wraparound's Equally-Evil Twin

Introduction

What’s a MultiXact ID?

When Are MultiXact IDs Created?

MultiXact IDs Need Freezing Too!

Don’t Let MultiXact Fly Under the Radar

Practical Recommendations

Conclusion

Making JSONB More Queryable with Generated Columns

Introduction

The Setup

Three Approaches to Indexing JSONB

Option 1: GIN Indexes

Option 2: Expression Indexes

Option 3: Generated Columns

Side-by-Side: Query Performance

A practical guide to what `.ready` and `.done` mean, and why WAL sticks around

What `archive_command` Actually Does

What `.ready` and `.done` Are For

`.ready`

`.done`

If `.done` Exists, When Is It Deleted?

What If `.done` Files Pile Up and Do Not Disappear?

What About `pg_receivewal`?