Local-first inference

PLAN.md §7b’s ambition was to make the owner’s own GPU a personal cloud reachable from any device, with the switch being configuration, not a rewrite. That’s built now: every tier resolves independently to openrouter or local, packages/llm/src/providers.ts is the single place that decides it, and Settings → Inference (apps/web/src/settings/InferencePanel.tsx) shows you the result and hands you the .env lines to change it.

What “local-first” actually means here

packages/llm’s client only ever speaks one protocol — OpenAI-compatible chat completions, embeddings, and /models — against a baseUrl that’s resolved per tier, per request. Ollama, llama.cpp’s server, vLLM, and LM Studio all answer that same protocol, so “run it locally” is nothing more exotic than pointing one tier’s baseUrl at http://localhost:11434/v1 instead of https://openrouter.ai/api/v1.

When a tier is local, for that tier’s calls:

The prompt text never leaves your machine — no memories, files, or messages quoted into it cross the network, because the request goes to a process on localhost (or your LAN), not a third party’s data centre.
The model’s reply never leaves your machine either — it’s generated on your GPU and read straight out of process memory.
No API key is sent anywhere, because there’s nothing to authenticate to (unless you’ve configured one for vLLM/LM Studio — see below — and even then it only ever goes to your own server).
Everything else about Novaterra is unchanged: your database, your other API keys and connection credentials, all of it already never left the machine running apps/api. Moving a tier local only changes where that tier’s prompts are answered. See Security & privacy for the full account of what leaves the house on every path, not just this one.

A tier you haven’t moved still goes to OpenRouter exactly as before — local-first is opt-in, per tier, and the default (nothing set) is byte-identical to the pre-§7b behaviour.

Setting up a local server

Ollama is the simplest starting point:

ollama serve                # exposes http://localhost:11434/v1
ollama pull qwen3:8b         # or any model below

vLLM is the alternative worth knowing about, especially once you’re running one model continuously and want higher throughput or an OpenAI-compatible API key requirement of its own:

vllm serve Qwen/Qwen3-8B --host 0.0.0.0 --port 8000 --api-key <a-token-you-choose>

vLLM’s --api-key flag is what LOCAL_LLM_API_KEY below is for. llama.cpp’s server binary and LM Studio’s local server both work the same way as Ollama — point LOCAL_LLM_BASE_URL at whichever one you’re running (http://localhost:8080/v1 for llama.cpp, http://localhost:1234/v1 for LM Studio) and everything else on this page applies unchanged.

The env vars, per tier, with worked examples

Tiers are cheap, standard, strong, image, embed — the same five everywhere else in Novaterra (see Models and the budget guards).

# Which provider answers this tier: openrouter (default) | local | webllm
LLM_PROVIDER_CHEAP=local
LLM_PROVIDER_STANDARD=local
LLM_PROVIDER_STRONG=            # left unset -> stays on OpenRouter

# The OpenAI-compatible root of your own machine. Unset = http://localhost:11434/v1 (Ollama's default).
LOCAL_LLM_BASE_URL=http://localhost:11434/v1

# The model id to ask that server for, per tier — exactly as `ollama list` prints it.
LOCAL_LLM_MODEL_CHEAP=qwen3:8b
LOCAL_LLM_MODEL_STANDARD=qwen3:32b

# Optional bearer token. Ollama ignores it; vLLM (--api-key) and LM Studio can require one.
LOCAL_LLM_API_KEY=

# What happens if the home box is unreachable when a local tier is called. See below — the default matters.
LOCAL_LLM_FALLBACK=

With the block above: cheap and standard run entirely on your GPU; strong (and image, embed — anything left unset) keeps going to OpenRouter. Restart the API after editing .env — provider choice is deliberately configuration, checked once at process start via packages/llm/src/providers.ts, not a runtime toggle a stray click could flip mid-thought.

LLM_PROVIDER_<TIER>=local without a matching LOCAL_LLM_MODEL_<TIER> does not refuse the call outright — refusing would take a working world down over a missing env var. Instead that tier quietly stays on OpenRouter, and the Inference panel shows a note explaining exactly why (LLM_PROVIDER_STANDARD=local but LOCAL_LLM_MODEL_STANDARD is not set — this tier is still going to OpenRouter.).

Suggested models for a 24GB RTX 3090

From SUGGESTED_LOCAL_MODELS in packages/llm/src/discovery.ts — the same list the Inference panel shows next to each tier:

Tier	Suggested	Why
`cheap`	`qwen3:8b`, `llama3.1:8b`	Comfortably fits in 24GB at Q8, fast enough for workers/summaries/classification
`standard`	`qwen3:32b`, `gemma3:27b`	Fits at Q4 on a 3090; the step up in quality Muse chat and agent reasoning benefit from
`strong`	(none)	Nothing 70B-class fits in 24GB at a usable quantisation yet — this tier is meant to stay on OpenRouter for now
`image`	(none)	No local image-generation path yet
`embed`	`nomic-embed-text`	Small, fast, and good enough for memory/file semantic search

These are suggestions, not a whitelist — any model id your server actually lists works in LOCAL_LLM_MODEL_<TIER>.

The “Detect local models” button

Settings → Inference has a Detect local models button that calls GET /api/llm/inference/detect (owner-only). It sends GET {LOCAL_LLM_BASE_URL}/models to your configured endpoint with a 2.5-second deadline, and understands both response shapes an OpenAI-compatible server might return: the standard { data: [{ id }] } list, and Ollama’s native { models: [{ name, size, details }] } shape from /api/tags — so pointing LOCAL_LLM_BASE_URL at whichever of Ollama’s two APIs you actually meant still produces a useful model list instead of a shrug. A dead endpoint isn’t an error from this route’s point of view — it’s a normal, informative answer ({ ok: false, error: "Nothing is listening at …" }) so the panel can say something a human can act on rather than showing a stack trace.

The button always probes the server’s own configured LOCAL_LLM_BASE_URL — it does not accept an arbitrary endpoint from the request. A route that fetches a caller-supplied URL from the server is a textbook SSRF vector (it could just as easily be pointed at a cloud metadata endpoint or another internal service), and the fix here is not to carefully allow-list private ranges but to give the route nothing to redirect: it only ever asks the one endpoint the owner already put in .env.

What leaves over the Cloudflare Tunnel, from a phone

If you’re running the Vercel + Cloudflare topology, your phone already reaches the API at api.novaterra.world through the tunnel exactly like any other device — nothing about local inference changes that path. As long as the API process and your GPU box are the same machine, the API just calls http://localhost:11434/v1 on itself; the tunnel only carries phone ↔ API traffic, never touches the LLM call, and your prompts still never leave the house.

If the API runs on a different, always-on machine from the GPU box (the common case once the API lives on a small always-on server and the 3090 lives in a gaming PC that isn’t always powered on), LOCAL_LLM_BASE_URL needs to reach the GPU box over the network instead of localhost. Add a second public hostname to the same Cloudflare Tunnel config (deploy/vercel-cloudflare/) pointing at the GPU box’s Ollama/vLLM port, and point LOCAL_LLM_BASE_URL at it:

LOCAL_LLM_BASE_URL=https://llm.novaterra.world/v1

This is still “local-first” in the sense that matters: the prompt only ever crosses infrastructure you control (your own tunnel, your own box), never OpenRouter — it just isn’t limited to one physical machine.

Three behaviours that will surprise you

A dead local box does NOT silently fall back to OpenRouter. By default, a tier set to local has no cross-provider fallback: if the GPU box is asleep, unreachable, or the model errors, the call simply fails with a clear error — nothing is sent to OpenRouter instead.

LOCAL_LLM_FALLBACK=openrouter   # opt in explicitly

Unset (or anything other than openrouter) means none — the safe default. This default exists on purpose: the entire point of moving a tier local is that its prompts stay home. If a sleeping GPU silently rerouted to OpenRouter, “local-first” would quietly become “OpenRouter with extra steps” on exactly the nights you’re not watching — the one time you’d actually notice a leaked prompt is the one time you’re not there to catch it. Setting LOCAL_LLM_FALLBACK=openrouter is a real, supported option for “available beats private” workloads — just know it means an outage sends that tier’s prompts out of the house.

Local calls cost $0 and do not draw down LLM_BUDGET_*_USD, but they ARE still recorded. Every call — local or remote — is written to the llm_calls table (costUsd: 0 for local), so GET /api/llm/usage and the wallet widget keep one complete history regardless of where a tier was answered. The three budget guards in apps/api/src/core/budget.ts sum real recorded cost, so a local call simply never touches those ceilings — see Models and the budget guards.

webllm is a declared seam, not an implementation. LLM_PROVIDER_<TIER>=webllm is accepted — the env contract is stable so a future in-browser WebGPU runtime can slot in without another env migration — and it’s reported faithfully by GET /api/llm/inference (requested: 'webllm'). But today, server-side calls for that tier still go to OpenRouter; the Inference panel says exactly that (LLM_PROVIDER_<TIER>=webllm runs in the browser; server-side calls for this tier still go to OpenRouter.). Nothing about WebGPU-in-the-browser is built yet — say so plainly rather than implying it works.

The two API routes

GET /api/llm/inference          # any signed-in being; the Inference panel's main data
GET /api/llm/inference/detect   # owner only; probes LOCAL_LLM_BASE_URL, see above

Neither route accepts a write — there’s no PATCH /api/llm/inference. Provider choice is a bigger decision than a web toggle should make silently: the panel instead renders the exact .env lines a change implies, copyable, so you can review them and restart the API deliberately.