cp653-659: offline install through all roles (docker images, mcp user, ipfs deferral) + Step-N-of-N wizard + password step + upgrade mirror rotation + wizard net-loss guard

2026-08-05 18:02:22 -07:00

25 KiB

Raw Permalink Blame History

Upgrading Morphit

A practical guide to keeping your Morphit instance current with upstream releases, written for sysadmins.

TL;DR. When you see a new release announced (Matrix channel, morphit-release-monitor alert, or the Forgejo release page), run:

sudo -u morphit npx morphit-ops upgrade

That command checks for a new release, shows you the notes, asks for confirmation, backs up your current install, applies the new tarball, runs npm ci, rebuilds and redeploys the web frontend (then verifies it's actually being served), rebuilds the compiled morphit-ops/MCP bundles, redeploys and restarts the MCP server (then checks it answered on its configured bind), and restarts services. If anything fails, it rolls back automatically.

The rest of this doc covers the details: how the trust chain works, the manual procedure (for operators who prefer to apply each step themselves), the automated mode, rollbacks, and building from source.

How releases work

A Morphit release is an annotated git tag of the form v<MAJOR>.<MINOR>.<PATCH> (e.g. v1.0.0-beta.1). The release workflow at .forgejo/workflows/release.yml runs on every tag push, in this order:

Verifies the tag is signed by an authorized maintainer's GPG key (the public keys live in .forgejo/release-signers/ in the repo — anyone with repo read access can audit who's authorized).
Runs the full validation gate: typecheck across all workspaces, ansible-lint in production profile, and the complete triple-pulse smoke suite (thousands of self-checking scenarios).
Builds the release tarball and SHA-256 checksum file.
Bakes a provenance manifest (release-info.json) into the tarball recording the tag, commit SHA, and CI build time.
Publishes both artifacts to the release page.

What you trust to use a release: Forgejo's HTTPS server. Once the artifacts are signed in the release UI, the SHA-256 chains the download to the announced version, and release-info.json chains that to the signed commit. If you want one more layer (defense against a compromised Forgejo), see the "Belt-and-braces verification" section below.

Maintainer note — the pre-release flag and morphit-ops upgrade. morphit-ops upgrade now finds the newest release even when it is flagged pre-release (it falls back from /releases/latest to the newest release of any kind when no stable exists). But that fallback only runs in the version the operator already has installed. An operator on a build that predates this fix (anything at or before v1.0.0-beta.2) is still running the old logic, which only sees the newest non-pre-release release. So until every operator has upgraded onto a build that contains the fallback, leave the release they need to upgrade to un-flagged as pre-release (uncheck "pre-release" in the Forgejo release UI), or their morphit-ops upgrade will report "already on the latest" and never see it. Once operators are on a fixed build, you can flag betas pre-release freely.

What if my instance is several releases behind?

Common case: you stood up an instance, didn't touch it for a while, and now you're a number of releases out of date. You do not need to apply each intermediate release in order. Morphit releases are cumulative tarballs (a full install, not a diff), so morphit-ops upgrade jumps you straight from whatever you're running to the latest published release in one step — the same check → backup → apply → npm ci → restart → auto-rollback flow. Run exactly the same command no matter how far behind you are:

sudo -u morphit npx morphit-ops upgrade

Two things to do first when you've been away a while:

Read the release notes for every version between yours and the latest, not just the newest one. The upgrade prints the latest release's notes, but if you've skipped several, any manual step or behavior change called out in an intermediate release still applies to you. The notes for each tag are linked from the Forgejo release page; --check-only prints the URL.
Confirm you're not crossing a major version. This tool assumes same-major upgrades (e.g. v1.x → v1.y). A major-version bump (v1.x → v2.0) may have manual migration steps that the release notes for that major will spell out — do those by hand before/after as instructed. Within a major, schema changes apply automatically: the indexer runs any pending migrations on startup, which the upgrade flow triggers when it restarts services, so there's no separate migration command.

Beta note (pre-1.0): during the beta the database layout is a single baseline that we sometimes edit in place between versions rather than as a separate numbered migration. Because the upgrade never touches your database (see the carry-forward note below), an existing database won't pick up such an in-place change on its own — so after a beta upgrade that crossed a schema change, the indexer can be missing a new table or column. This is harmless to fix: the indexer database is rebuilt from the chain, so you reset and re-sync it. morphit-ops upgrade prints a reminder when it crosses a schema change, and morphit-ops doctor detects a drifted database and points you at the fix. Full steps: OPERATIONS.md §46. (This caveat goes away at 1.0, when every schema change ships as its own migration that does apply automatically on restart.)

If you're so far behind that you're unsure what changed, the safest path is: take a database backup (see OPERATIONS.md), read the notes for each skipped release, then run the upgrade. Your config and signing key (morphit.config.env, morphit.env, apps/relay/keystore.*, apps/relay/altnet/) live inside the install dir, so the upgrade explicitly carries them forward into the new release (step 8b above) with their permissions intact — your settings and active key survive every upgrade, and your PostgreSQL database is never touched.

Recommended: `morphit-ops upgrade`

The fast path for trusted operators. Reads the current installed version from /opt/morphit/release-info.json, polls the Forgejo release API, and walks you through the upgrade.

Check without applying

sudo -u morphit npx morphit-ops upgrade --check-only

Exits 0 if up-to-date, 1 if a newer release exists. Suitable for cron and the morphit-release-monitor sidecar (see below).

Add --json to get machine-readable output:

sudo -u morphit npx morphit-ops upgrade --check-only --json
{
  "current": "v1.0.0-beta.1",
  "latest": "v1.0.0-beta.1",
  "up_to_date": true,
  "release_url": "https://git.agorise.net/agorise/morphit/releases/tag/v1.0.0-beta.1",
  "published_at": "2026-05-22T15:30:00Z"
}

Apply an upgrade

sudo -u morphit npx morphit-ops upgrade

Steps the command takes, in order:

Reads /opt/morphit/release-info.json for the current version.
Finds the release to offer you. It first asks Forgejo for the latest stable release (/releases/latest, which by Forgejo's rules returns the newest release not marked pre-release). If there is no stable release yet — which is the case throughout the beta period, when every release is a pre-release — it falls back to the newest release of any kind (/releases?limit=1). So during beta you always get the newest beta even if it's flagged pre-release; once a stable ships, the stable is preferred and you are not pushed onto a newer beta automatically.
If you're already on it, exits 0 (no-op).
Otherwise, shows you the release notes and prompts y/N.
Downloads the tarball + .sha256 to /tmp/morphit-upgrade-<ts>/.
Verifies the SHA-256 against the downloaded checksum file. Refuses to proceed if it doesn't match.
Renames /opt/morphit → /opt/morphit.bak-<timestamp> (backup).
Extracts the new tarball to /opt/morphit. 8b. Carries your config and signing key forward from the backup into the freshly-extracted tree — morphit.config.env, morphit.env, apps/relay/keystore.json (or .wif), apps/relay/altnet/ (your Tor/Lokinet/I2P keys), and morphit-hardening-checklist.md, all with their 0600 permissions preserved. The release tarball deliberately does not contain these (they're your secrets), so the upgrade copies them across for you. You do not re-run morphit-ops init after an upgrade, and you don't re-enter your passphrase config — your instance comes back up exactly as it was, on the new code.
Runs npm ci --no-audit --no-fund in the new install dir. 9b. Rebuilds and republishes the web frontend. The indexer, relay, and matrix-bot run straight from TypeScript source via tsx, so npm ci is all they need — but the website is a static SvelteKit build, and the release tarball does not ship a prebuilt frontend. So the upgrade always runs the web build (npm run build in apps/web) — on every upgrade, regardless of how your site is served — and then publishes it to wherever it's served from:
- Bare-metal nginx: if your web root exists (MORPHIT_WEB_ROOT, default /var/www/morphit-frontend), the new build is copied in, preserving that folder's ownership. nginx serves the new files immediately — no reload needed.
- BunkerWeb (or any Docker frontend): the upgrade finds the running container that bind-mounts the build directory (<install>/apps/web/build) — identified by the mount itself, so it works no matter what the container is called (morphit-frontend, bunkerweb-frontend-1, a hand-rolled stack, …) — and docker restarts it so it re-binds the freshly-built files. No compose file is required, and a container left running would otherwise keep serving the previous build.
- Both can apply (then it does both). If neither target is found — a genuinely non-standard setup — the rebuilt files are left on disk and the upgrade tells you exactly where to copy them from; the backend still upgrades.
Why this matters (fixed in beta.10): before beta.10 the rebuild only ran when /var/www/morphit-frontend existed, so a BunkerWeb host silently skipped the frontend rebuild, upgraded the backend, and reported success — leaving visitors on the old build (and never showing the "Load it now" update prompt). The rebuild is now unconditional and the publish target is auto-detected.

Update (beta.11): the Docker publish step now identifies the frontend container by the build-dir mount it carries and docker restarts it, instead of recreating a container matched by name through the example compose file. Earlier releases missed custom stacks (a different container name, or no compose file on the host); mount-based detection handles them, so an operator running a bespoke reverse-proxy stack gets the new build without manual steps. 9c. Verifies the new frontend is actually being served (beta.14). Publishing the build and serving it are two different things — a frontend container that bakes the build into its image (rather than bind-mounting it), or any other publish gap, would leave the old site live and the "Load it now" update prompt would never fire. So after publishing, the upgrade reads the service-worker version it just built and compares it to what the live frontend actually serves (the copied file for bare-metal; the container's own port for a Docker frontend, with a short retry while it restarts). It then tells you plainly:
- fresh → "Verified the live frontend is serving this build … Returning visitors get the 'Load it now' update prompt within ~60s."
- stale → a loud warning naming the served-vs-built versions and the specific fix (e.g. your container bakes the build in — rebuild its image, or switch it to bind-mount <install>/apps/web/build).
- can't tell → a one-line note with the manual curl to check.
This check is best-effort and never fails the upgrade; it exists so a silent publish failure becomes a visible, actionable one.
Refreshes the installed systemd unit files from the templates in the freshly-extracted ops/systemd/. Those units are static files you copy to /etc/systemd/system/ once at install, so a fix to a unit (for example an added RestrictAddressFamilies=AF_UNIX, without which a service can crash-loop) used to ship in the repo but never reach an already-installed box — the upgrade only restarted services, it didn't update the unit files. Now it does, safely:
- Only units you already have installed and whose contents differ from the template are touched — it never installs a unit you didn't choose to run, and never rewrites an identical one.
- The previous file is saved to <unit>.bak before it's overwritten, so a hand-edited unit is never silently lost.
- Drop-ins (<unit>.d/*.conf) are never touched — your overrides survive (so a unit you fixed by hand with a drop-in keeps working after the base unit is refreshed).
- It then runs systemctl daemon-reload so the restart below picks up the new units. This whole step is best-effort and never fails the upgrade. (Set MORPHIT_SYSTEMD_DIR if your units live somewhere other than /etc/systemd/system.)
Restarts these systemd services if they're active:
- morphit-indexer.service
- morphit-relay.service
- morphit-matrix-bot.service
Prunes old backups, keeping the 3 most recent (tunable via MORPHIT_BACKUP_KEEP).

If any step from 7 onwards fails, the command:

Removes the partial extract at /opt/morphit
Renames the backup back to /opt/morphit
Restores the previous web frontend (if it had already redeployed)
Restarts services on the previous version
Exits with code 3 ("upgrade failed, rolled back")

In the rare case rollback itself fails (filesystem error, etc.), it exits with code 4 and prints the manual recovery steps.

Automated mode (opt-in)

By default the command prompts y/N before applying. To skip the prompt — for cron jobs or unattended automation — set:

export MORPHIT_AUTO_UPGRADE=1
sudo -u morphit -E npx morphit-ops upgrade

MORPHIT_AUTO_UPGRADE=1 is opt-in, not the default, by design: Morphit's posture is that an operator should consciously decide to upgrade. Auto-upgrade is a power-user feature for sites that want to track upstream tightly.

Equivalent inline:

sudo -u morphit npx morphit-ops upgrade --yes

Configuration

Env var	Default	What it does
`MORPHIT_AUTO_UPGRADE`	unset	Set to `1` to skip the y/N prompt
`MORPHIT_RELEASE_HOST`	`git.agorise.net`	primary Forgejo host (trusted hash anchor)
`MORPHIT_RELEASE_REPO`	`agorise/morphit`	repo path
`MORPHIT_RELEASE_MIRRORS`	unset	comma-separated fallback sources, each `host` or `host/owner/repo`, tried after the primary
`MORPHIT_INSTALL_DIR`	`/opt/morphit`	install location
`MORPHIT_WEB_ROOT`	`/var/www/morphit-frontend`	bare-metal nginx web root; if it exists, `upgrade` copies the freshly-built `apps/web/build` here. Set it for a custom bare-metal path. The web app is always rebuilt regardless; on a host running a Docker frontend the container that bind-mounts the build dir is `docker restart`ed instead (detected by the mount, any container name), and if neither target is found the rebuilt files are left on disk with a warning
`MORPHIT_BACKUP_KEEP`	`3`	how many `.bak-*` backups to retain

Mirrors and how integrity is protected

upgrade can fall back to mirrors if the primary is slow or down, set via MORPHIT_RELEASE_MIRRORS (e.g. MORPHIT_RELEASE_MIRRORS=mirror.example,codeberg.org/agorise/morphit). Crucially, a mirror is never trusted just because it served the bytes — verifying a mirror's tarball against that same mirror's checksum proves nothing. Two integrity paths apply, in trust order:

GPG signature (source-independent). If the release carries a *.tar.gz.asc, upgrade verifies it against the release-signer public keys that ship in your install at .forgejo/release-signers/*.asc. The trust anchor is local and code-reviewed, so a signed tarball is trustworthy no matter which mirror served it — this is what makes a fully standalone mirror safe, even if the primary is censored or gone.
Anchored SHA-256. When there's no signature, the tiny .sha256 is always fetched from the trusted primary over HTTPS; the big tarball bytes may come from a mirror; the bytes are checked against the primary's hash. A hostile mirror can't forge this. If the primary is completely unreachable and the release is unsigned, upgrade refuses rather than trust a mirror blindly.

To publish signed releases (recommended — it's what enables fully independent mirrors), set two repo secrets in Forgejo → repo → Settings → Actions → Secrets:

MORPHIT_RELEASE_SIGNING_KEY — an ASCII-armored private signing key whose public half is committed under .forgejo/release-signers/ (so every operator can verify it).
MORPHIT_RELEASE_SIGNING_PASSPHRASE — its passphrase (or empty).

If those secrets are absent, CI still publishes a working release, but unsigned — mirror installs then rely on the anchored-SHA-256 path only.

Get notified about new releases — `morphit-release-monitor`

If you'd rather not poll manually, the morphit-release-monitor sidecar runs every 6 hours, calls morphit-ops upgrade --check-only, and emits an INFO event when a newer release is available. The event surfaces via journalctl -u morphit-release-monitor, the matrix-bot alert relay (if enabled), or whatever else you point your structured-event ingestion at.

The sidecar never applies upgrades itself — it only watches. You still run morphit-ops upgrade manually (or with MORPHIT_AUTO_UPGRADE=1 cron) when you decide to apply.

Enable it via the Ansible role release_monitor (default OFF; opt-in via enable_release_monitor: true in group_vars/all.yml).

Manual upgrade procedure

If you'd rather apply each step by hand — for review, for an air-gapped install, or because something about the automated flow doesn't fit your environment — here's the explicit recipe.

# 1. Download the release artifacts (replace VERSION).
cd /tmp
VERSION=v1.0.0-beta.1
curl -fLO "https://git.agorise.net/agorise/morphit/releases/download/${VERSION}/morphit-${VERSION}.tar.gz"
curl -fLO "https://git.agorise.net/agorise/morphit/releases/download/${VERSION}/morphit-${VERSION}.tar.gz.sha256"

# 2. Verify the checksum.  Output must say "OK"; refuse to proceed otherwise.
sha256sum -c "morphit-${VERSION}.tar.gz.sha256"

# 3. Stop the running services.
sudo systemctl stop morphit-indexer morphit-relay
# Matrix bot is optional; stop if installed:
sudo systemctl is-active --quiet morphit-matrix-bot && sudo systemctl stop morphit-matrix-bot

# 4. Backup the current install.
sudo mv /opt/morphit /opt/morphit.bak-$(date -u +%Y%m%dT%H%M%S)

# 5. Extract the new tarball.
sudo mkdir -p /opt/morphit
sudo tar -xzf "morphit-${VERSION}.tar.gz" -C /opt/morphit
sudo chown -R morphit:morphit /opt/morphit  # adjust to match your install

# 6. Install workspace dependencies.
cd /opt/morphit
sudo -u morphit npm ci --no-audit --no-fund

# 6b. Rebuild the static web frontend and deploy it to your web root.
#     (indexer/relay/matrix-bot run from TS source via tsx and need no
#     build step; the website is static files nginx serves from a folder.)
(cd apps/web && sudo -u morphit npm run build)
sudo cp -r apps/web/build/* /var/www/morphit-frontend/
sudo chown -R www-data:www-data /var/www/morphit-frontend  # match your web root's owner

# 7. Verify the new version's release-info.json matches what you downloaded.
cat /opt/morphit/release-info.json
# Confirm "tag" field === "${VERSION}"

# 8. Restart services.
sudo systemctl start morphit-indexer
sudo systemctl start morphit-relay
sudo systemctl is-active --quiet morphit-matrix-bot.service || true && sudo systemctl start morphit-matrix-bot

# 9. Tail logs for a minute to confirm clean startup.
journalctl -fu morphit-indexer -u morphit-relay

If the new version misbehaves, see Rollback below.

Belt-and-braces verification (optional)

The default SHA-256 chain trusts Forgejo's HTTPS server to serve the right .sha256 file. To add a layer — defense against a compromised Forgejo — verify the underlying git tag's GPG signature directly:

# 1. Clone the source (anywhere, doesn't have to be /opt/morphit).
git clone https://git.agorise.net/agorise/morphit.git /tmp/morphit-verify
cd /tmp/morphit-verify

# 2. Import all authorized signer keys.
gpg --import .forgejo/release-signers/*.asc

# 3. Verify the tag is signed by one of those keys.
git tag -v v1.0.0-beta.1
# Output should end with "Good signature from..." and the
# fingerprint should match an authorized signer.

# 4. Optional: confirm the released tarball's git tree matches
#    the tag's tree.  This is brittle (depends on tar's filename
#    ordering matching git's), so the easier check is to compare
#    individual file contents:
git checkout v1.0.0-beta.1
diff -r . /opt/morphit
# Should show only node_modules/ and similar build-artifact paths
# as different (those aren't in the source tree).

If git tag -v says "Good signature" and the diff is clean, the tarball you applied is provably the source the tag points to.

Rollback

morphit-ops upgrade automatically rolls back on any failure between extract and service-restart. If you find a problem AFTER the upgrade completes — slow startup, missing feature, bug reported by a user — you can manually swap to the previous install:

# 1. Stop services.
sudo systemctl stop morphit-indexer morphit-relay
sudo systemctl is-active --quiet morphit-matrix-bot && sudo systemctl stop morphit-matrix-bot

# 2. Find the most recent backup.
ls -ltd /opt/morphit.bak-* | head -1

# 3. Swap.
sudo mv /opt/morphit /opt/morphit.bad-$(date -u +%Y%m%dT%H%M%S)
sudo mv /opt/morphit.bak-<timestamp> /opt/morphit

# 4. Restart services.
sudo systemctl start morphit-indexer morphit-relay

If the issue is a database-side bug (e.g. a new schema migration broke something), additional steps may be needed — these would be called out in the release notes.

Building from source

For maximum trust, or to develop locally, build from the git source instead of the released tarball:

git clone https://git.agorise.net/agorise/morphit.git
cd morphit

# Verify the tag (optional but recommended).
gpg --import .forgejo/release-signers/*.asc
git tag -v v1.0.0-beta.1

# Check out the release tag.
git checkout v1.0.0-beta.1

# Install + run the validation gates (same gates CI runs).
npm ci
bash scripts/typecheck-sweep.sh
bash scripts/run-smokes.sh

# Now follow docs/RUN-A-MORPHIT-NODE.md for the first-time
# install, or — if upgrading — stop services, sync this tree
# to /opt/morphit, npm ci, restart services.

This is the path for the savviest operators (and for developers). You're not bound to released tarballs; you can run any commit on main, you can audit changes between tags by reading git log, you can build and verify locally.

When upgrades happen

Morphit follows a "stable releases" model: tagged releases come when the maintainer decides a milestone is shippable. There's no fixed cadence. Major releases (vX.0.0) may include breaking changes called out in the release notes; minor (vX.Y.0) and patch (vX.Y.Z) releases are intended to be drop-in safe.

Pre-release tags (-alpha.N, -beta.N, -rc.N) are valid release versions with the same release tooling — they're explicitly less stable and intended for operators participating in beta-testing.

Troubleshooting

"No release-info.json at /opt/morphit/release-info.json" — your install predates the release-tooling provenance manifest (v1.0.0-beta.1+). For first-time upgrades from a manual install, copy this file manually:

echo '{"tag":"unknown","commit":"unknown","build_time":"unknown","builder":"manual"}' \
  | sudo tee /opt/morphit/release-info.json

Then re-run morphit-ops upgrade. It will detect the latest version and apply normally.

"SHA-256 mismatch on downloaded tarball" — the download was corrupted or tampered with. Don't proceed. Retry the download; if the mismatch persists, alert the maintainers via the Matrix channel (#agorise:matrix.org) — this could indicate a Forgejo compromise.

"Service restart failed for morphit-indexer; rolling back" — the new version's startup failed. Check journalctl -u morphit-indexer -n 200 for the error. The rollback runs automatically; once you've diagnosed the issue, either work around it or wait for a fixed release.

"Manual intervention needed: /opt/morphit is in a partial state" — both upgrade AND rollback failed. Don't restart services. Inspect /opt/morphit and the *.bak-* backups; manually move the most-recent backup into place and restart. This case is rare and indicates a filesystem-level issue (full disk, permission drift, ...).

25 KiB Raw Permalink Blame History