Scaling and redundancy

Novaterra runs on one machine today, on purpose. This page is about what happens when one machine is not enough — and, just as importantly, about how to tell whether you have actually reached that point or only feel like you have.

Everything here is checked into the repo under deploy/scaling/.

The four tiers

Tier	What it is	Survives	Where
0	One process, SQLite, a Cloudflare Tunnel	the process crashing	`deploy/vercel-cloudflare/`
1	One host, supervised: auto-start, health watchdog, WAL checkpointing, backups with a rehearsed restore	reboots, logouts, updates, disk failure	`pnpm node:supervise`, `pnpm node:backup`, `deploy/scaling/tier1-single-host/`
2	Two or more API hosts, Postgres, Redis, shared object storage, one elected scheduler	a whole machine dying	`deploy/scaling/tier2-multi-host/`
Local	Someone’s own private Novaterra, one command	—	`deploy/scaling/local-install/`

Tier 1 is the recommended next step for almost everyone. Tier 2 is a large piece of work that mostly buys availability, not throughput — and it is worth understanding why before starting it.

What actually breaks when you run two processes

The full audit lives at deploy/scaling/statelessness-audit.md, with file and line references for every item. The short version:

Already fine. Sessions are stored in the sessions table and read back on every request, so once the database is shared, no sticky sessions are needed and no session store has to be introduced. This is the thing most applications get wrong, and Novaterra does not.

Breaks quietly. Several loops run on a timer inside the API process and would run once per process:

the Twin scheduler (every 10 minutes: syncs plus a reflection call per being),
the square-life loop (every 4 minutes: AI citizens posting),
the Telegram long-poller — and Telegram itself returns 409 Conflict when two pollers share a token, so messages get split between them, some answered twice and some dropped.

Each of those is a language-model call. Two nodes means double the spend against the same budget, and duplicate replies that look like a bug in the Twin rather than a deployment problem.

Breaks visibly. The WebSocket registry is a map in one process’s memory. A user connected to node A never receives an event produced on node B — so a Muse reply simply never arrives, and it looks like the model failed.

Breaks architecturally. workspace/ (Studio outputs, uploads) is a local directory, the Docker sandbox runs on whichever node took the request, and the Twin’s file watcher watches a path that only exists on one machine.

And underneath all of it: SQLite. Two processes on one machine can share a WAL SQLite database. Two hosts cannot, and putting the file on a network share does not rescue it — WAL mode needs shared memory and POSIX locks that SMB and NFS do not implement correctly. The failure mode is not an error message, it is corruption. Multi-host means Postgres. There is no incremental path.

Tier 1 — the step worth taking

Tier 1 changes nothing about the architecture. It changes whether the thing comes back on its own. Keeping it running is the page with the commands; this is the summary.

Supervision — pnpm node:supervise, a dependency-free script that restarts with backoff, gives up out loud on a crash loop, and distinguishes dead (restart), wedged — responding but with a database that will not answer (restart), and degraded — budget spent, embeddings failing, no sandbox, disk full (log, never restart). A watchdog that restarts on any non-green signal turns a budget alert into a restart loop. Let the OS start it at boot; systemd and NSSM config for that ships in deploy/scaling/tier1-single-host/.
A crash leaves a mark. Every boot writes a durable row and every clean shutdown closes it, so /api/health can say whether the previous run ended clean, unclean or concurrent, and whether the node is in a crash loop rather than having had a crash. Before this, “did it crash last night” had no answer but a gap in a log.
WAL checkpointing, on a schedule inside the API — the only process that can do it without contending with itself — and once at boot, which is the one quiet moment an oversized log left by a previous process truncates cleanly.
A disk gauge that is not a lie. On a full disk SQLite fails every write while every read still succeeds, so db.ok — a SELECT — stays true while logins fail. That happened here. storage.writable is the field that goes false.
Backups with a restore you have actually run. pnpm node:backup create writes one encrypted archive of the database, the workspace and the secrets; restore … --into ./drill rehearses the restore into a scratch directory without touching production, and prints every being, their did:key address and their rotation chain so a lossy restore cannot pass as a good one.
Graceful restart. The API stops its heartbeat, closes every socket with code 1001 (“server restarting”), drains in-flight requests, folds the write-ahead log back in and closes the database cleanly. True zero downtime is not achievable with one process and one SQLite file — a ~2–4 second reconnect is, and the section below says why the stronger claim is not available.

There is also a cold standby pattern that costs no code: a second machine with the repo, the env file and the tunnel credentials, service installed but disabled. The Cloudflare Tunnel is identified by its credentials, not by the machine, so failover needs no DNS change. Manual, a few minutes, and it turns “down until someone is free” into “down for ten minutes”.

Tier 2 — and the conflict you must resolve first

Multi-host redundancy means: Postgres instead of SQLite (a 3–6 day port, including replacing FTS5 full-text search and rewriting how memory recall ranks results), Redis for WebSocket fan-out, an object store for workspace/, a leader lease so exactly one node runs the scheduled loops, and a separate sandbox host.

If those machines are rented, this conflicts directly with a privacy promise Novaterra has already made. The Security overview states that the API, the database and the Docker sandbox run on hardware the company owns and operates directly — not in a data centre, not behind a load balancer. A managed Postgres puts every memory, message and encrypted credential on a third party’s disk. That is a new subprocessor, a DPIA that needs redoing, and a customer notification — not a deployment detail.

There is a version that keeps the promise: run Tier 2 on hardware you own, two machines in the same building. You lose geographic redundancy and you gain redundancy against a machine crashing, a disk dying or an update rebooting a node — most of the value, at the cost of one more computer, with no new subprocessors. The shipped compose file runs either way; the only difference is which machines the containers sit on.

The repo says this plainly, in deploy/scaling/README.md, so the decision gets made deliberately rather than discovered afterwards.

Tier 2 is not the answer to “more people”

Worth separating, because the two questions get conflated constantly. Everything above is about making one node survive its own hardware — a crash, a reboot, a dying disk. That is a real problem and Tier 1 solves most of it cheaply.

Scaling one node to hold many people’s worlds is a different thing entirely, and it is the one road Novaterra is deliberately not taking. One install has one owner, enforced in the database, because that is what makes “your data is on hardware you control” a sentence rather than a slogan. Multi-tenancy is what you build when you cannot federate. The intended answer to many people is many nodes that mesh — each authoritative for its own beings, exchanging signed data with peers. That part is built and runs today: every being has a keypair, nodes exchange signed sealed messages, and the federation courier (above, and apps/api/src/modules/federation/courier.ts) is the store-and-forward loop that already drains a queue when a sleeping peer wakes. What is still missing is population, not mechanism — every node in this repository was started by this team, so “many nodes” has been proven working, not proven at scale with strangers on it. That honest status is on the roadmap.

The directory scales separately from any node

services/directory (see Finding other nodes) is worth naming here because it is easy to fold into “the mesh” and it should not be: it is a separate deployable — its own Fastify process and its own SQLite file, run by Vocabotics rather than by any one node — and it is a lookup, not an authority: records are signed by their subject, so a directory instance can withhold, delay or serve a stale answer but cannot forge one. That bound is exactly what lets it scale independently. Because it holds no keys and asserts nothing, growing it to many more nodes looking each other up is a question about that process’s own hosting (more read replicas of one small SQLite-backed lookup table), never a question about any individual node’s Tier 0/1/2 above, and a slow or unreachable directory degrades lookups, not federation itself — a node with peers already added keeps talking to them with DIRECTORY_ENDPOINT unset entirely. As with the rest of this page: the directory’s own capacity numbers are reasoned from what the code does, not measured against real traffic, and should be treated that way until someone runs the benchmark.

What actually limits Novaterra today

Worth knowing before adding hardware, because a second node fixes almost none of it. In the order these bite:

The machine itself — sleep, reboot, power, ISP. Tier 1 exists to move this down the list.
The LLM budget. A handful of genuinely active users exhausts a daily budget in an afternoon, and the API then correctly refuses calls. No amount of hardware changes this.
Rate limits on a single model-provider key. A second node does not give you a second key.
The Docker sandbox — one container per execution, each with its own memory ceiling.
Memory recall. Embeddings are compared in-process across everything a being remembers. That is fine while the numbers are small — and it is the sharpest cliff in the system, arriving with one heavy user rather than with many users. pgvector fixes it, which is one of the better arguments for the Postgres port having nothing to do with redundancy.

Adding a node addresses none of 2, 3 or 5.

A note on honesty: the capacity numbers in the repo’s Tier 1 notes are reasoned from the source code, not measured. A benchmark plan is included; run it before promising anyone a concurrency figure.

Running your own

If what you want is not scale but your own private Novaterra, that is a first-class path and it is one command — see the repo’s deploy/scaling/local-install/, which generates real secrets rather than the development defaults, offers to point inference at a local Ollama so that nothing leaves your machine at all, sets a conservative spend guard, seeds a world and starts it.

A local install that you want running all the time is simply Tier 1 on your own machine.

Why zero-downtime restart is not on that list

It is worth being blunt rather than leaving it as a caveat, because “rolling restart” is what people expect from a service and it is not available here.

better-sqlite3 is in-process and synchronous, and the database is one file with one writer. A zero-downtime restart means two processes serving at once, and two processes on this database is not a performance problem, it is a correctness one:

Boot is not idempotent under concurrency. Both processes run migrations and the empty-world seed; the seed check is a read followed by a write with nothing holding the gap.
Every scheduled loop would run twice — the Twin’s reflections, the square-life posts, the federation courier — each of which is a model call billed to the same budget. Telegram’s API returns 409 Conflict outright when two pollers share a token.
The WebSocket registry is a map in one process’s memory, so a client connected to the old process would never receive an event produced by the new one.
The boot-time reapers assume they are alone. A second process starting up would reap the first one’s live work as stranded.

There is no configuration that fixes this; the fix is Postgres and external pub/sub, which is Tier 2, which is the conversation below. What Tier 1 gives you instead is a polite restart: every socket closed with code 1001 so clients reconnect rather than log out, in-flight requests drained, the log folded back in, and a couple of seconds of 502 at the tunnel. Schedule it for 04:00 and it costs nothing anybody notices.