morphit/docs/adr/0008-phase3b-indexer-architecture.md

ADR-0008: Phase 3b indexer architecture

Status: Accepted Date: 2026-04-18 Supersedes: none Depends on: ADR-0001 (custom_json replacement ops), ADR-0005 (Phase 3 subphase split), ADR-0007 (keygen curve + dblurt package)

Context

Phase 3a shipped the ability to create accounts and broadcast ops. Phase 3b needs to make those ops readable — specifically, the orderbook, profiles, feedback, and release-discovery UIs need to query current state derived from on-chain ops. The Blurt chain itself is not a good source for that query surface: get_account returns account data, not arbitrary op history; reading full block history for every frontend page load would be prohibitively slow and hammer public RPC nodes.

The indexer is the bridge: it follows the chain's op stream, picks out Morphit-relevant ops, and exposes queryable state over HTTP.

Decision

Ingest: pull, not push

The indexer polls condenser_api.get_block and condenser_api.get_dynamic_global_properties on a ~3-second cadence (Blurt's block interval), using the existing relay-style multi-endpoint rotator pattern.

Why pull over push:

• Blurt's HTTP-JSON-RPC is the only transport dblurt supports post-0.7; WebSocket subscriptions aren't available
• Pull-based indexing is crash-safe: the indexer records the last successfully-indexed block and resumes from there on restart
• Polling latency is bounded and predictable (~3 seconds per block), good enough for an orderbook that isn't latency- sensitive the way an order-matching engine would be
• We can rate-limit ourselves trivially (don't make more than one get_block call per block interval)

Storage: Postgres

PostgreSQL 15+ is the indexer's backing store.

Schema strategy — hybrid event log + materialized views:

• Event log table (ops): every morphit op we've ever seen, append-only, primary-keyed on (block_num, trx_in_block, op_in_trx). Never updated, never deleted. Audit surface.
• Materialized state tables (orders, profiles, feedback, releases, chat_messages): current derived state. Written by op-handler functions as new ops arrive.
• Materialized tables can be fully rebuilt by replaying the event log in order. This is the DR / schema-migration strategy.

Why not SQLite:

• Postgres's concurrency story is better (the indexer writes while the HTTP layer reads)
• Horizontal scale paths exist (read replicas, partitioning)
• tsvector full-text indexing is ~free; we'll want it for profile display names and search

Why not a graph or document DB:

• Query patterns are simple: "list orders filtered by asset + side
  • location + freshness", "get profile by account name", "tally feedback for account X". Relational works.
• No joins we can't express in standard SQL

State model: replace-in-place for logical updates

Consistent with ADR-0001, morphit ops are protocol-immutable on chain. The indexer compresses this to the user-facing view:

• morphit_order_v1 creates a row in orders with status = 'live'
• morphit_order_replace_v1 looks up the target by permlink, updates the row, preserves the original created_at and bumps updated_at. The event log keeps both the original and the replacement; the orders table shows only the latest state.
• morphit_order_cancel_v1 flips status = 'cancelled', keeps the row for audit (no hard delete)
• morphit_feedback_response_v1 attaches to the original feedback row as a nested response (separate table joined on feedback_id), not replacing the original feedback

Reorg handling: trail the irreversible block

The indexer maintains two block pointers:

• head_block: the highest block we've received from the chain
• indexed_block: the highest block we've applied to materialized state

We only apply ops from blocks where block_num <= last_irreversible_block_num (provided in every dynamic global properties response). This means the indexer trails head by ~15- 21 blocks (45-63 seconds), which is the price of never needing to roll back. DPoS irreversibility is effectively permanent; no reorg recovery logic needed.

Reconciliation + catch-up

On startup, the indexer:

1. Reads its last-applied block from the database
2. Polls current last_irreversible_block_num from the chain
3. Fetches every block in between, applying morphit ops it finds
4. Only after the catch-up is complete does it begin serving HTTP queries

This means a fresh deploy has a lengthy initial sync before the HTTP API goes live. Acceptable for a project still in launch.

HTTP API surface — REST + JSON

Endpoints the frontend needs:

• GET /v1/health — uptime, chain head, indexed head, lag in blocks
• GET /v1/orderbook — filter by asset, side, location region, freshness; paginate; returns live orders only
• GET /v1/orders/:account — orders for a specific account
• GET /v1/profiles/:account — latest profile for an account
• GET /v1/accounts/:account/feedback — feedback records + summary stats (count, weighted avg)
• GET /v1/release — latest verified morphit_release_v1 op
• GET /v1/chat/:account_a/:account_b — chat messages between two accounts, newest first, paginated

No POST endpoints — all writes go to the chain via Phase 3a's relay or via broadcast from the user's browser. The indexer is strictly read-only from the frontend's perspective.

Authentication

None. Every HTTP endpoint is public. Morphit's whole design premise is that on-chain data is public; the indexer mirrors that publicness. See ADR-0006 for why CSRF/auth concerns don't apply.

Op validation at ingest time

The indexer re-verifies every op's signature against the Blurt chain's public-key records before applying to materialized state. This is strict — an op can fail:

• Structural validation (wrong shape, missing required field)
• Signature validation (signer's posting key doesn't match the account's recorded posting key)
• Authorization (e.g. a morphit_order_replace_v1 targeting an order not owned by the signer)

Ops failing any of these land in the event log tagged as rejected (so we know they existed) but never update materialized state. The frontend never sees them.

Pagination + freshness

• All list endpoints return { items, next_cursor } cursor-based pagination, not offset-based. Cursors are opaque base64-encoded (block_num, trx_in_block, op_in_trx) tuples
• ETag + Cache-Control: max-age=3 on responses. A 3-second cache window matches the indexer's poll cadence — clients get fresh data exactly as fast as the chain produces it

What's NOT in 3b

• Price feed integration (ADR-0004 is the UI-side fallback; 3b doesn't compute or serve prices)
• Full-text search of order bodies (basic ILIKE queries are fine for launch; tsvector is wired but exposed as a 4+ phase)
• WebSocket / SSE push to the frontend (polling is fine; upgrade path is clean if we want it later) Shipped in Phase E: SSE endpoints /v1/orderbook/stream, /v1/chat/stream, /v1/instances/stream are now live. Frontend consumers buffer events and flush per-rAF (see apps/web/src/lib/orderbook/ stream.ts and apps/web/src/lib/chat/stream.ts). Each stream has a 500-event buffer cap (Audit 2026-05 findings 2-11 and NEW-10-2).
• Historical reporting / analytics endpoints
• Search-by-region geolocation (strings in, strings out; no geo math)

Alternatives considered

Chain-of-state — no event log, only derived state

Smaller storage footprint, simpler schema. Rejected: rebuilding after a schema change would require re-reading the Blurt chain from launch block, hitting public RPC nodes hard and taking hours.

TimescaleDB instead of plain Postgres

TimescaleDB's hypertables are good for time-series-heavy workloads. Rejected: Morphit's queries aren't time-series-shaped. The orderbook queries "live orders matching filters" — a set operation, not a range scan. Plain Postgres is enough.

Redis for hot materialized state

Performance argument: hot-path reads (orderbook, profile) happen often; Redis could serve them in <1ms. Rejected for 3b: adds a second stateful component to operate. Postgres with proper indexes + max-age=3 Cache-Control handles the traffic volume Morphit will see in its first year. Revisit in Phase 5 if metrics justify it.

Writing the indexer in Go instead of Node.js/TypeScript

Go is the traditional choice for a streaming indexer — high throughput, low memory, easy static binary. Rejected: same reason as Phase 3a — no actively-maintained Go library for Blurt means we'd re-implement the signature-verification logic. @beblurt/dblurt gives us the full verify path in TypeScript, and Node 22 is fast enough for our op-rate ceiling (we won't exceed a few hundred morphit ops per hour in the foreseeable future).

Consequences

Positive

• Single language (TypeScript) across relay + indexer + frontend, with shared libraries (@beblurt/dblurt provides the same signing + verification primitives all three need)
• Hybrid log + materialized strategy is standard and well-understood
• Trailing irreversibility makes reorg handling a non-issue
• REST + JSON is the broadest-compatibility API shape

Negative

• Database migrations need care. Adding a column to orders is easy; changing the primary key triggers a rebuild from the event log. Migration system design is a separate sub-decision (next section).
• No websocket push means a 3-second worst-case staleness for UI updates. Acceptable given the orderbook's human-scale decision speed.

Migration strategy (follow-up)

Database schema changes fall into three classes:

1. Additive, non-breaking (new column with default, new index): applied in-place with standard ALTER TABLE. No downtime.
2. Materialized-view rebuild: new handler logic, different derivation from event log. Rebuild the affected table from the event log, then switch over atomically in a transaction.
3. Event log schema change: rare. Requires a re-read from chain. Documented as an operator task in the indexer README.

Each type gets a separate migration script, versioned sequentially. Using a bare-minimum migration tool (single numbered .sql files

• a schema_migrations table tracking applied versions) rather than a framework — simplicity wins when there are few migrations and no ORM involved.

References

• docs/PHASE-3a-DESIGN.md — the design doc this ADR extends
• docs/PHASE-3a-STATUS.md — what shipped in 3a
• ADR-0001: custom_json immutability
• ADR-0005: Phase 3 subphase split
• apps/web/src/lib/net/config.ts — OP_IDS the indexer must recognize