Keeping it running

Tier 0 — one machine, one process, a person noticing when it stops — is where a Novaterra node lives by default. A tsx watch crash is survivable at Tier 0 because somebody is looking at the terminal. That is not a property of the software, it is a person.

This page is the set of commands that turns each of those into a property. None of it changes the architecture. All of it changes whether the node comes back, and whether you find out when it doesn’t.

The four things Tier 1 actually is

	Command	What it closes
Supervision	`pnpm node:supervise`	a crash, a wedged process, a boot loop nobody notices
WAL checkpointing	automatic, in the API	a write-ahead log that grows without bound
A backup	`pnpm node:backup create`	disk failure, a bad migration, `rm` at 2am
A restore you have performed	`pnpm drill:restore`	the backup being a hope rather than a backup

The last row is the one that is usually skipped, and it is the only one that proves the third.

And there is a fifth thing, which is not a command but an installation: none of the above is a property of your machine until something starts it at boot. On Windows that is deploy/windows/install-node.ps1, and §1.1 below is what it does.

1. Supervision

pnpm node:supervise

This starts the API and keeps it started. It is a script with no dependencies, and it exists because the operating system’s own supervisors answer only half the question:

	Starts at boot	Restarts on crash	Notices a wedged process	Stops it gracefully on Windows
systemd (Linux)	yes	yes	no	yes
NSSM service (Windows)	yes	yes	no	yes, if configured for console stop
Task Scheduler (Windows)	yes	crudely	no	no
pm2	yes	yes	no	no, and it is another daemon to keep alive
`pnpm node:supervise`	no	yes	yes	yes

So use both: the OS starts the supervisor at boot, and the supervisor watches the node. A systemd unit or an NSSM service whose command is pnpm node:supervise gets every column, and adds no package to the tree.

The three states, and why only two of them restart

Every 60 seconds the supervisor reads /api/health and sorts what it finds into three buckets:

Dead — no answer twice running. Restart.
Wedged — it answers, but db.ok is false three checks in a row. Event loop blocked, database locked. A plain supervisor never sees this, because the process is alive. Restart.
Degraded — the LLM budget is exhausted, embeddings are behind, there is no sandbox, free disk is low, or the database has stopped accepting writes. Log it and leave it alone. Restarting fixes none of these, and a supervisor that restarts on any unhappy signal turns a budget alert into an outage.

It also backs off — 1s, 2s, 5s, 10s, 20s, 40s, 60s — and gives up out loud after six restarts in five minutes, with a line saying “this is a crash loop, not a crash”. Restarting a process that cannot start, once a second, forever, fills the disk with logs and buries the cause.

On Windows, `taskkill /F` is not a graceful stop

This one costs people time, so it is worth stating plainly. SIGTERM does not exist on Windows. process.kill(pid, 'SIGTERM'), child.kill() and taskkill /F all become TerminateProcess: the shutdown handler in apps/api/src/core/shutdown.ts never runs, in-flight requests are cut off, the write-ahead log is not folded back in, and the next boot correctly reports a crash that never happened.

The stops that do reach the handler on Windows are Ctrl+C / Ctrl+Break in the console, and an IPC message from a parent process. pnpm node:supervise uses the second. If you stop your node any other way on Windows, expect an unclean verdict next time it starts, and know that it is telling the truth.

Once the supervisor is started at boot rather than by you, it is in a different session — often a different user, because a boot task runs as SYSTEM — and there is no console to press Ctrl+C in. So there is a command for it, and it waits until the node is actually down before returning:

pnpm node:supervise stop      # clean: the database closes with a final checkpoint
pnpm node:supervise status    # is one running, and since when

Measured on the machine this was written on: 1.2 seconds, and node_runs records the stop as parent asked rather than as a crash.

1.1 Starting it at boot

# Look first. This prints every command it would run and changes nothing.
powershell -NoProfile -ExecutionPolicy Bypass -File deploy\windows\install-node.ps1 -DryRun

# Then, in an ELEVATED PowerShell:
powershell -NoProfile -ExecutionPolicy Bypass -File deploy\windows\install-node.ps1

That registers a scheduled task that starts the supervisor at boot as SYSTEM — before anyone logs in — plus an hourly database backup and a nightly full one, and stores the backup passphrase DPAPI-protected for that machine alone. -Remove undoes all of it, stopping the node cleanly first. deploy/windows/README.md lists every change it makes, in full, before you run it.

Two options are set that are wrong by default and produce no error when they bite:

Task Scheduler’s ExecutionTimeLimit defaults to three days, after which it terminates the task. For a supervisor meant to run forever that is a hard kill every three days, recorded as an unclean stop, for no reason at all.
On a laptop, a task will not start on battery and is stopped when you unplug — both by default.

There is one thing a scheduled task cannot do, and it is worth choosing deliberately: Windows terminates a scheduled task at shutdown, so a Windows Update reboot leaves an unclean verdict. A real service does get told to stop, so -Method Nssm (which needs nssm on PATH) makes reboots clean too. That is the only difference between the two, and it is the only reason to take the dependency.

2. A crash now leaves a mark

Before this, a crash left nothing at all. The process vanished, something restarted it or nobody did, and “did it crash last night?” had no answer beyond a gap in a log file nothing rotates.

Every boot now writes a row and every clean shutdown closes it, so the next boot can say what happened. It is the same principle the Studio already applies to its runs — a lie is worse than a failure — applied to the process itself.

GET /api/health reports the verdict:

"lastStop": { "verdict": "unclean", "at": "…", "reason": null, "uncleanStreak": 2 }

Verdict	Means
`first-boot`	nothing came before on this host
`clean`	the previous run wrote its own stop mark: drained, checkpointed, closed
`unclean`	the previous run never reached the shutdown path. Killed, power cut, or a crash
`concurrent`	the previous run is still alive. Two processes, one database — a different problem with the opposite fix

An uncleanStreak above 1 is a crash loop rather than a crash, and the boot log says so in those words.

clean is asserted as hard as unclean in the tests, deliberately. A crash detector that reports a crash on every restart is one the owner learns to ignore, and then the real crash goes unread.

Prove it, on your own machine

pnpm drill:kill

That takes a throwaway copy of your world, starts a real API against it, plants a Studio run in the state a killed process leaves, kills the process the way a power cut does, restarts, and checks what the first one left behind. It never touches your live database and never calls a model.

3. The write-ahead log

journal_mode = WAL is the right setting and is not the problem. The problem is that SQLite’s automatic checkpoint only fires around 1000 pages and only completes when no reader is holding an older snapshot, so a long-lived process with a steady trickle of writes can go a very long time without a full one — and the -wal file just grows.

This is not hypothetical. When this was written, the live novaterra.db was 2.9 MB with a 4.1 MB -wal beside it: the log was larger than the database it was logging. Every reader paid for it and every backup carried it.

The API now checkpoints on a schedule from inside the process, and once more at boot — the quietest moment the database ever has, and the one time an oversized log left by a previous process truncates without contending with anything. Nothing to install and nothing to schedule.

/api/health reports storage.walBytes and storage.consecutiveBusy. One busy checkpoint is routine; a run of them means checkpoints are never completing, which is exactly how a -wal reaches gigabytes.

A cron job calling an external checkpoint script would be the wrong shape here: the API is the single writer, so a second process opening the file to checkpoint it is a second writer contending with the first, and can only ever get busy while the API is doing anything.

4. A full disk, and the gauge that lied

This machine has hit literal zero three times, and one of those failed a login — logging in writes a session row, and SQLite could not write it.

The reason nobody caught it deserves stating, because it is not obvious. On a full disk, reads keep working. SELECT succeeds, PRAGMA integrity_check says ok, and /api/health’s db.ok — which is a SELECT count(*) FROM beings — stays true while every write fails with SQLITE_FULL. So the health endpoint reported a perfectly healthy database on the day the login failed, and any watchdog polling it saw green.

Two things now say otherwise:

storage.freeBytes / storage.level, checked on a schedule, with a warning in the log below a 1 GB floor. Absolute rather than a percentage: what SQLite needs is an amount, not a fraction.
storage.writable — the outcome of the node’s own periodic write. This is the field that goes false when the disk fills, because none of the others do.

Both are degraded, not wedged: the supervisor logs them and does not restart, because restarting a process cannot create free disk and a restart loop on a full disk is worse than the full disk.

5. Backups, which are also a set of private signing keys

pnpm node:backup create                 # database + workspace + secrets, encrypted, into ./backups
pnpm node:backup create --keep 14       # and prune to the newest 14
pnpm node:backup list
pnpm node:backup verify ./backups/novaterra-….nova

Since identity landed, a backup of your node contains two kinds of private key, and a third secret that unwraps both:

being_keys.secret — every being’s Ed25519 signing key. Whoever holds it is that being. There is no revocation and no reset link.
sealing_keys.secret — every being’s X25519 sealing key. There is no forward secrecy here, so a backup stolen today opens messages that were sealed before it was taken — and retired sealing keys keep their secrets for a thirty-day grace so that mail in flight still arrives, which means a stolen archive opens a month of it.
.env, whose ENCRYPTION_KEY unwraps both. Lose it and a restored node has every address and can sign nothing.

So:

Archives are encrypted by default, AES-256-GCM under a scrypt-derived key. The passphrase is never taken from the command line: arguments land in shell history and in the process table. Type it, pipe it with --passphrase-stdin, point at a file with --passphrase-file, or set NOVA_BACKUP_PASSPHRASE for an unattended nightly run.
A plaintext archive is refused into a cloud-sync folder. OneDrive, Dropbox, Google Drive, iCloud and Nextcloud paths are recognised and rejected, because otherwise “back it up to the cloud” and “hand a stranger’s server every key on the node” are the same command.
If you mistype the passphrase and do not notice, the archive is unreadable and the keys in it are gone. That is why create asks for it twice.

The database inside is a snapshot, taken through SQLite’s online backup API while the node is running — never a file copy. Copying novaterra.db on a live WAL database gives you the database without the transactions still sitting in the -wal: it restores cleanly and is silently missing the last few minutes of your world. That is the worst kind of backup, one that looks like it worked.

The archive is a gzipped ustar behind a one-line header, so an unencrypted one opens with tar -tzf on any machine — including Windows’ built-in tar. This tool is deliberately not the only way to get your world back out of it.

A backup also refuses to run when there is not room for it. A backup that fills the disk it is protecting is not a backup, it is the outage. Concretely, on a disk that cannot hold the archive:

it refuses before it starts, naming both numbers, and creates nothing — not even the output directory;
a write that fails part-way leaves no archive and no partial, because otherwise a full disk would cost you the backup and the space, and tomorrow’s attempt would start from a worse position than today’s;
a partial abandoned by a power cut is reclaimed on the next run, with a line saying how much came back.

Size it against your own disk before trusting a retention number. Measured on one real node — 22 beings, 10 Studio projects, a 3 MB database and a 92 MB workspace — a full archive is 24 MB and a database-only one is 0.8 MB, essentially all of the difference being workspace/projects at roughly 9 MB per Studio project. That is what makes hourly database-only plus nightly full the right default: it takes the worst case from 24 hours to 1 hour for everything except Studio output files, and costs 38 MB. And note that --keep 14 needs room for fifteen archives, because the new one is written before the oldest is pruned — deliberately, since pruning a good archive before its replacement exists is how a retention policy eats your last backup.

6. The restore drill — the part that is usually skipped

An untested backup is a hope. Rehearse it:

pnpm drill:restore

That is the whole rehearsal in one command: it takes a throwaway copy of your world, performs real key rotations on it so the archive under test contains real rotation chains, backs it up, restores it, and then signs new passports with the restored keys.

That last step is the one that matters, and it is why the drill exists rather than just the restore below. Every row in a restored being_keys looks identical whether the private halves came back or not: a node that restored public-only boots fine, shows every correct did:key address, and cannot prove it is anybody — and nothing inside the node notices until a peer asks. Signing something that did not exist when the backup was written is the only check that can tell the difference. The drill also signs with the wrong key and with an edited payload, and requires the verifier to reject both, because “verify returned true” proves nothing on its own.

It does the same for the sealing keys, which cannot be checked the same way: a signature is publicly verifiable, a Diffie-Hellman is not, so the only proof is to seal a fresh message to the restored public half and open it with the restored private one.

To rehearse one archive you already have, by hand:

pnpm node:backup restore ./backups/novaterra-….nova --into ./drill

--into unpacks to a scratch directory and touches nothing live, so it is safe to run whenever you like. It verifies the archive before anything is written, integrity-checks the database it contains, and then reads back what actually landed at the destination — printing every being, their did:key address, and their rotation chain:

Restored database now holds:
  integrity_check   ok
  beings            20 (1 owner)
    @jon  jon  (owner)
      address       did:key:z6Mkw5VuTYJbcsokZUgeKqPVzmEtboM9KwUW6CSy8eTHY2ht
      keys          1 (1 with the private half)
    @aria  Aria
      address       did:key:z6MkonD6i7mENyivhNSAQnjq8UDc9ZQSwVkERNFXCum1AJQz
      keys          3 (3 with the private half)
      chain         did:key:z6MkjLfJ… -> did:key:z6MkjkQe… -> did:key:z6MkonD6…*

The chain line is the one to read. A lossy restore keeps the active key — so the node looks completely fine — and drops the retired predecessors, and then nothing that being signed before its last rotation can be verified by a peer ever again. That loss is invisible from inside the node, which is why the drill prints the whole chain rather than a row count.

When you are ready to restore for real, --live replaces the node’s data. It keeps a safety copy of the current database beside it as your undo, removes the stale -wal and -shm (leaving one database’s log beside another’s pages is how a restore becomes corruption), and writes .env as .env.restored rather than over your working one.

Stop the API before --live.

7. Moving your whole world

Moving your node to your own hardware covers identity: who your beings are, and the keys that make them them. This covers the rest — the database, workspace/ (Studio outputs, uploads, installed plugin bundles) and the secrets .env holds.

Together they are the whole node, and the move is the drill you have already rehearsed:

On the old machine: pnpm node:backup create.
Carry the one .nova file across, along with the passphrase, by different routes.
On the new machine: clone the repo, pnpm install, then pnpm node:backup restore <file> --live.
Copy .env.restored over .env and start the node.

The tunnel does not need to change: a Cloudflare Tunnel is identified by its credentials file, not by the machine it runs on.

What Tier 1 still does not give you

Stated plainly, because a page like this is easy to read as more than it is.

Zero-downtime restart is not achievable, and no amount of scripting changes that. See Scaling and redundancy for the reason. What you get is a graceful restart of a couple of seconds where nothing is lost.
The machine is still one machine. Supervision recovers a crashed process in seconds; it does nothing at all for a dead power supply. The cheap next step is a second box you own, holding a restored copy and the same tunnel credentials, ready to be started by hand.
RPO is however old your last backup is. Nightly means up to 24 hours. Run it hourly if that is too much; the archives are small once --no-workspace is in play.