Free chapter: Six outages, and the rule each one left

Some of these I caused. All of them left a rule. With no team to catch my mistakes, most of those rules now sit where my AI agent reads them every session (section 9.7).

These are six of the outages, chosen because each left a different rule. A few smaller ones (a slow tunnel, a bad daytime swap) are told in Chapter 8.

The setup. Mathter, a Next.js app, runs on one small server in my house, behind the Cloudflare tunnel (a program on my server that dials out to Cloudflare, so no port is open at home). Until October it ran in a virtual machine on my Windows desktop. One person runs it, with one AI agent that I gave access to the servers. A nightly job deploys whatever I merged that day.

My agent typed some of the commands in this chapter. They’re still mine, because I gave it the access. Two of the outages (9.3 and 9.4) I didn’t cause at all: Windows Update and the home network did. They’re here because they shaped the setup in Chapter 8.

9.1 The “quick check” that took the site down

August 2026. Site fully down.

In the early days, the live site ran straight out of the project folder. A production server (next start) served pages from a build folder called .next.

My agent wanted to check a small change, so it started a development server (next dev) in the same folder. The development server rebuilds that .next folder continuously, in a different format from a production build. The live server suddenly found files missing or half-replaced. Pages failed with “module not found”, and the site went down.

The worst part: it had done the same thing once already that session, without harm. “Just a quick check” was the reasoning both times. The second time it broke production.

This happened on Next.js 14. Next.js 16 keeps the development server’s build files in a separate folder (.next/dev) by default, so this exact collision is less likely there. The rule below still holds.

What changed:

A written rule: never run a development server in the folder production serves from. It’s in the agent’s instructions and in its memory, with the incident linked.
Worktrees for all experiments. A worktree is a second checkout of the same repo in its own folder, with its own build folder. Every change now gets built and tested in one.
Later, production moved to its own machine entirely, so a slip on the development box can’t touch it at all.

9.2 A build that died halfway

October 16, 2026. Site partly broken for about seven hours, overnight.

The nightly deploy built the new version in place: in the same .next folder the live site was reading. A production build starts by deleting that folder, then compiles a new one.

That night the build tried to fetch a web font, the font server didn’t answer, and the build died about 20 seconds in. The live site was left serving a half-written folder: no build ID, chunks from two different builds. Visitors got errors.

Why didn’t the heartbeat check, which runs every two minutes, catch it? It checks that the site answers, and how fast. It doesn’t open every page, so a site that is only partly broken can pass it.

It had happened before, on August 25, when a test sweep shared the machine with a build (section 6.6). The fix then moved sweeps off production, but the build itself still ran in place.

What changed: builds now happen in a separate folder and are swapped in only when finished and checked, one build at a time. Section 8.3 describes how. The font files now sit in the repo, so the build no longer needs the font server.

Still open: a partly broken site can still pass the heartbeat. The deploy’s own checks now catch a half-built folder before it goes live.

Lesson: any step that deletes before it creates needs to happen somewhere the live site isn’t looking.

9.3 The reboot that took the tunnel with it

August 2026. Down for about an hour, late evening.

Relevant if you host at home or on a virtual machine.

At the time, the site ran in a virtual machine on my home PC, behind the tunnel. To connect, the tunnel program first has to look up Cloudflare’s address.

Windows rebooted the host PC for updates, three times in half an hour. Each time, the virtual network came back with a new random address range (the set of local addresses the machines on that network get), and its built-in DNS (the service that turns names into addresses) often came back dead. The tunnel program couldn’t look up Cloudflare’s address, so it never connected. Cloudflare showed visitors a 530 error, together with Cloudflare error 1033.

The site came back the moment I set the server’s DNS by hand.

What changed:

DNS on every server is pinned to public resolvers (name-lookup services anyone can use). It’s no longer left to whatever the virtual network offers.
No configuration anywhere uses a hard-coded address for the home VMs, because those addresses change on every reboot.
A note I now check first: a 530 with error 1033 on a tunnelled site means the tunnel isn’t connected. Suspect the server’s DNS or network before the app. Knowing that one code saves the first half hour of an outage like this.

9.4 A network that dropped out every six minutes

September–October 2026. Weeks of short drops, a few real outages.

Relevant if you host at home or on a virtual machine.

This was the hardest. The site mostly worked, but the tunnel logs filled with “no recent network activity”. Bursts of 10 to 30 seconds with no network, roughly every six minutes, day and night. It went from one or two a day to over a hundred a day in under a week. Most of the time some of the tunnel’s four connections survived. When all four dropped at once, the site was offline.

What made it hard was the shape of it: delay, not loss. Pings (tiny “are you there?” messages between machines) mostly got through, eventually. That signature looks like bufferbloat, like a busy ISP, like the virtual network, like the network card’s power saving. I suspected each in turn.

Then the agent and I measured, one layer at a time:

The servers never froze: a timer loop on each ran without a gap.
Both virtual machines lost the internet and the home network in the same seconds, so it wasn’t the internet provider.
A connection between the two virtual machines on a private switch (a virtual network link between just those two) kept flowing through a burst, so it wasn’t the virtualization software.
The host PC itself lost pings to the router in the same seconds. The problem sat between the PC and the router.

The bursts started the night of a Windows update reboot. They stopped, for a while, after the router went down and came back. When production later moved to a dedicated machine plugged straight into the router, the same bursts followed it. That pointed at the router side.

What changed:

Two tunnel copies instead of one, so one dropping doesn’t take the site with it.
A watchdog that restarts a tunnel copy if it stays disconnected, one at a time.
A modem and router reboot, which helped. The home network was later reorganized, with the old Wi-Fi router turned into a simple access point. It now only provides Wi-Fi; the ISP’s router does the routing.
My note says, in bold: don’t re-diagnose the network card, the power saving or the ISP; it’s already been ruled out, with the evidence listed.

Lesson: when a fault looks like everything, stop guessing and cut the system in half. Test one boundary at a time, and write down what each test ruled out, so you never pay for it twice.

9.5 A week of lost email nobody noticed

October 2026. Inbound email down for 8 days.

When production moved to its new machine, I changed one DNS record to point at it. During that move, one of the domain’s mail records was lost. Cloudflare’s audit log shows it removed by an automated entry two minutes before my change. I never found what triggered it. The fix for next time is the same either way: check the mail records after any DNS change.

Outgoing email (receipts, trial emails) used a separate service and kept working. But email to the domain was undeliverable for eight days. I found out only because a test email to an address on my own domain never arrived.

The fix was one button in Cloudflare: “Enable Email Routing”.

What changed:

After any DNS change, a one-line check that the mail records are all there. It’s in the deploy notes and the agent’s memory.
The weekly health report now checks the mail records too.

9.6 The safe restart that wasn’t

October 2026. Down for about two minutes.

A security patch for Node.js came out. Updating it on the server meant moving the process manager (PM2, which keeps the app running) onto the new version. The documented command for that is pm2 update.

On my server, PM2 itself runs under systemd, the system’s service manager. During the update, the old PM2 process exits. Systemd saw it exit, decided the service had stopped, and ran its stop command, which kills all PM2 processes, including the new one. The whole app went down, and visitors saw a 502 for about two minutes until I restored the saved process list.

What changed:

The rule: on production, restart the service through systemd, never with pm2 update.
After any restart at that level, check the process list and the live site straight away.
Save the process list before every reboot.
Know what supervises your process manager before you restart it. A command that’s safe on a laptop can break production, where an extra layer is watching.

9.7 Turning incidents into guardrails for your agent

My agent doesn’t learn from an outage by itself. Every incident above ended the same way: an incident note in the repo (docs/incidents/), written the same day; a rule in CLAUDE.md (the instruction file the agent reads at the start of every session) if it’s something never to do again; a memory note for facts about the environment; and a test where one is possible, like the deploy’s check that a build is complete before it swaps anything in.

Two habits come from this chapter in particular:

Write down what you ruled out. Half the cost of the network drops (9.4) was re-testing things already cleared.
Correct notes that turn out to be wrong. A stale note is worse than none, because the agent will trust it.

9.8 Checklist

Never run a development server where production serves from; use worktrees
Build out of place; swap only a finished, checked build; one build at a time
Pin DNS on servers; never hard-code addresses that can change
Know what a 530 with error 1033 (or your host’s equivalent) means
Run at least two tunnel copies, restarted one at a time
When a fault looks like everything, test one boundary at a time and record what you ruled out
Check mail records after every DNS change, and again in a weekly report
Know what supervises your process manager before restarting it
For every incident: a note, a rule, a memory, and a test if you can

The deploy, hosting and agent-rules details behind these fixes are in Chapters 3 and 8.

Next chapter: free pages as the front door, and how to tell whether a change to one did anything.