t3-code-android-nightly/docs/operations/observability.md

# Observability

> For maintainers. Using T3 Code? See [docs/user](../user/).

T3 Code has one server-side observability model:

- pretty logs go to stdout for humans
- completed spans go to a local NDJSON trace file
- traces, metrics, and logs can also be exported over OTLP to a real backend like Grafana LGTM

The local trace file is the persisted source of truth for normal local launches. Those launches do not
write a separate server log file, but SSH-managed launches also persist the remote process's
stdout/stderr at `~/.t3/ssh-launch/<state>/server.log`.

## Where To Find Things

### Logs

Logs are human-facing:

- destination: stdout
- format: `Logger.consolePretty()`
- normal local persistence: none
- SSH-managed launch persistence: `~/.t3/ssh-launch/<state>/server.log`
- remote export: OTLP only, when configured

If you want a log message to show up in the trace file, emit it inside an active span with `Effect.log...`. `Logger.tracerLogger` will attach it as a span event.

Configuring a logs endpoint takes over that job. The server then exports log records, which cover
every message instead of only the ones inside an active span and carry the trace and span ids so
they still line up with the trace. `Logger.tracerLogger` is dropped in that mode, so the same
message is not exported twice and the trace file stops carrying log messages. stdout output and
SSH-managed launch persistence stay unchanged either way.

### Traces

Completed spans are written as NDJSON records to `serverTracePath`. The default depends on how the
server starts: production and explicitly configured homes use
`<home>/userdata/logs/server.trace.ndjson` (so `~/.t3/userdata/...` by default, or
`/custom/path/userdata/...` with `--home-dir /custom/path`), a linked worktree dev run uses
`<worktree>/.t3/userdata/logs/server.trace.ndjson`, and an implicit dev run outside a linked
worktree uses `~/.t3/dev/logs/server.trace.ndjson`.

Important fields common to both record types:

- `type`: `effect-span` or `otlp-span`
- `name`: span name
- `traceId`, `spanId`, `parentSpanId`: correlation
- `durationMs`: elapsed time
- `attributes`: structured context
- `events`: embedded logs and custom events

`effect-span` records also contain `exit` with `Success`, `Failure`, or `Interrupted`. `otlp-span`
records instead carry OTLP resource, scope, and optional status fields.

The `TraceRecord`, `EffectTraceRecord`, and `OtlpTraceRecord` schemas live in
`packages/shared/src/observability.ts`.

DPoP proof failures include the safe `environment.dpop.failure_code` span
attribute. A `time_window` failure means that a signed proof was too old or too
far in the future for the environment server's allowed window. It can point to
a date or time problem on either device, but it can also result from a delayed
request.

#### Summarize the trace file

`t3 trace summary` reads the trace file and its rotated backups directly, so it works while the
server is stalled or stopped. It prints counts, rates, and latency percentiles per span name. Use
it to measure background work or to compare two builds.

```bash
t3 trace summary --since 30m --limit 40
```

It reads `T3CODE_TRACE_FILE` if set, else `<home>/userdata/logs/server.trace.ndjson` for
`--base-dir` or `T3CODE_HOME`, plus the `T3CODE_TRACE_MAX_FILES` rotated backups. For a dev run or
a copied file, set `T3CODE_TRACE_FILE`. `--since 30m` keeps spans that ended in the last 30
minutes. The rate is per minute between the first and last span end.

### Metrics

Metrics are not written to a local file.

- local persistence: none
- remote export: OTLP only, when configured
- current definitions: `apps/server/src/observability/Metrics.ts`

If OTLP is not configured, metrics still exist in-process, but you will not have a local artifact to inspect.

### Event Loop Stalls

`apps/server/src/observability/EventLoopMonitor.ts` samples the server's event loop every 30 s. When
the loop stalled for more than 2 s since the previous sample, it records a root
`server.eventLoop.stall` span with a warning. The span has trace level `Warn`, so it stays when
`T3CODE_TRACE_MIN_LEVEL` is `Warn`. The warning shows in Settings > Diagnostics unless OTLP logs are
on. The span time is when the sample ran, not when the stall happened.

Some delay is not recorded:

- `delayMaxMs` is the longest stall, and can undercount it by up to 1 s. The 2 s threshold applies to
  this value, so a stall over 3 s is normally recorded, and a shorter one can be missed. A stall
  that ends just as a sample runs can be missed too.
- Time the computer spends asleep reads as delay on macOS and Windows. So a sample only counts when
  the loop was busy, not waiting for events, for at least `delayMaxMs`. Busy time covers the whole
  window, so a short sleep in an otherwise busy window can still record a false stall. The span then
  shows CPU time far below `delayMaxMs`.
- The first sample after launch is skipped. Startup work such as migrations and projection bootstrap
  can block the loop for seconds on a large database.

CPU times and page faults cover the whole process over the whole window since the previous sample.
The window is nominally 30 s, but a long stall delays the sample and makes the window longer. Other
work in the window can hide a wait, so only CPU time far below `delayMaxMs` proves the thread was
waiting. Read CPU together with page faults:

- High `cpuSystemMs` with many page faults means memory pressure. Major faults are reads from disk or swap.
  On macOS, reads from compressed memory are minor faults plus system CPU.
- High `cpuUserMs` with few page faults means JavaScript work or garbage collection.
- Low CPU with few major page faults points at synchronous disk I/O, such as SQLite reads or trace
  file writes.
- Many `involuntaryContextSwitches` mean other processes were competing for the CPU.

### Related Artifacts

Provider event NDJSON files still exist for provider runtime streams. Those are separate from the main server trace file.

## Run The Server In Instrumented Mode

There are two useful modes:

- local-only: stdout + local `server.trace.ndjson`
- full local observability: stdout + local trace file + OTLP export to Grafana/Tempo/Prometheus

The local trace file is always on. OTLP export is opt-in.

### Option 1: Local Traces Only

You do not need any extra env vars. Just run the app normally and inspect `server.trace.ndjson`.

Examples:

```bash
npx t3
```

```bash
node --run dev
```

```bash
node --run dev:desktop
```

### Option 2: Run With A Local LGTM Stack

#### 1. Start Grafana LGTM

```bash
docker run --name lgtm \
  -p 3000:3000 \
  -p 4317:4317 \
  -p 4318:4318 \
  --rm -ti \
  grafana/otel-lgtm
```

Then open `http://localhost:3000`.

Default Grafana login:

- username: `admin`
- password: `admin`

#### 2. Export OTLP env vars

```bash
export T3CODE_OTLP_TRACES_URL=http://localhost:4318/v1/traces
export T3CODE_OTLP_METRICS_URL=http://localhost:4318/v1/metrics
export T3CODE_OTLP_LOGS_URL=http://localhost:4318/v1/logs
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=development
```

Optional:

```bash
export T3CODE_TRACE_MIN_LEVEL=Info
export T3CODE_TRACE_TIMING_ENABLED=true
```

#### 3. Launch the app from that same shell

CLI:

```bash
npx t3
```

Monorepo web/server dev:

```bash
node --run dev
```

Monorepo desktop dev:

```bash
node --run dev:desktop
```

Packaged desktop app:

Launch the actual app executable from the same shell so the desktop app and embedded backend inherit `T3CODE_OTLP_*`.

macOS app bundle example:

```bash
T3CODE_OTLP_TRACES_URL=http://localhost:4318/v1/traces \
T3CODE_OTLP_METRICS_URL=http://localhost:4318/v1/metrics \
T3CODE_OTLP_LOGS_URL=http://localhost:4318/v1/logs \
"/Applications/T3 Code.app/Contents/MacOS/T3 Code"
```

Direct binary example:

```bash
T3CODE_OTLP_TRACES_URL=http://localhost:4318/v1/traces \
T3CODE_OTLP_METRICS_URL=http://localhost:4318/v1/metrics \
T3CODE_OTLP_LOGS_URL=http://localhost:4318/v1/logs \
./path/to/your/desktop-app-binary
```

Do not rely on launching from Finder, Spotlight, the dock, or the Start menu after setting shell env vars. Those launches usually will not pick them up.

#### 4. Fully restart after changing env

The backend reads observability config at process start. If you change OTLP env vars, stop the app completely and start it again.

## How To Use Traces And Metrics To Debug The Server

### Start With The Local Trace File

The trace file is the fastest way to inspect raw span data.

Resolve the path for the launch mode once. Production and explicitly configured homes store runtime
state under the base directory's `userdata` folder:

```bash
TRACE_FILE="${T3CODE_HOME:-$HOME/.t3}/userdata/logs/server.trace.ndjson"
```

A dev server started from a linked worktree defaults to that worktree's local home:

```bash
TRACE_FILE="$WORKTREE/.t3/userdata/logs/server.trace.ndjson"
```

Only an implicit dev run outside a linked worktree uses the shared dev directory:

```bash
TRACE_FILE="$HOME/.t3/dev/logs/server.trace.ndjson"
```

Tail the selected file:

```bash
tail -f "$TRACE_FILE"
```

Show failed spans:

```bash
jq -c 'select(.type == "effect-span" and .exit._tag != "Success") | {
  name,
  durationMs,
  exit,
  attributes
}' "$TRACE_FILE"
```

Show slow spans:

```bash
jq -c 'select(.durationMs > 1000) | {
  name,
  durationMs,
  traceId,
  spanId
}' "$TRACE_FILE"
```

Inspect embedded log events:

```bash
jq -c 'select(any(.events[]?; .attributes["effect.logLevel"] != null)) | {
  name,
  durationMs,
  events: [
    .events[]
    | select(.attributes["effect.logLevel"] != null)
    | {
        message: .name,
        level: .attributes["effect.logLevel"]
      }
  ]
}' "$TRACE_FILE"
```

Follow one trace:

```bash
jq -r 'select(.traceId == "TRACE_ID_HERE") | [
  .name,
  .spanId,
  (.parentSpanId // "-"),
  .durationMs
] | @tsv' "$TRACE_FILE"
```

Filter orchestration commands:

```bash
jq -c 'select(.attributes["orchestration_v2.command_type"] != null) | {
  name,
  durationMs,
  commandType: .attributes["orchestration_v2.command_type"],
  threadId: .attributes["orchestration_v2.thread_id"]
}' "$TRACE_FILE"
```

Filter git activity:

```bash
jq -c 'select(.attributes["git.operation"] != null) | {
  name,
  durationMs,
  operation: .attributes["git.operation"],
  cwd: .attributes["git.cwd"],
  hookEvents: [
    .events[]
    | select(.name == "git.hook.started" or .name == "git.hook.finished")
  ]
}' "$TRACE_FILE"
```

### Use Tempo When You Need A Real Trace Viewer

Tempo is better than raw NDJSON when you want to:

- search across many traces
- inspect parent/child relationships visually
- compare many slow traces
- drill into one failing request without hand-joining by `traceId`

Recommended flow in Grafana:

1. Open `Explore`.
2. Pick the `Tempo` data source.
3. Set the time range to something recent like `Last 15 minutes`.
4. Start broad. Do not begin with a very narrow query.
5. Look for spans from the `t3code-server` or `t3code-desktop` service, then narrow by span name or
   attributes.

Good first searches:

- service name `t3code-server` or `t3code-desktop`, plus a resource attribute such as
  `deployment.environment.name`
- span names like `sendTurn` or a Git operation such as `GitVcsDriver.statusDetails.status`
- Git spans whose `git.operation` attribute identifies the operation
- orchestration spans with attributes like `orchestration_v2.command_type`

Once you know traces are arriving, narrower TraceQL queries for names such as `sendTurn` or Git
operation names become useful.

### Use Metrics To See Systemic Problems

Traces are best for one request. Metrics are best for trends.

Good metric families to watch:

- `t3_rpc_request_duration`
- `t3_provider_turn_duration` (how long the provider adapter takes to start a turn, not the turn's run time)
- `t3_git_command_duration`

Counters tell you volume and failure rate:

- `t3_rpc_requests_total`
- `t3_provider_turns_total`
- `t3_git_commands_total`

Webhooks have their own families:

- `t3_webhook_deliveries_total` by `outcome` and `source` (`relay` or `direct`). Beyond what the
  sender sees, `queue_full` means a task already had its limit of deliveries waiting,
  `prompt_too_long` means the filled-in prompt passed the provider limit, and `duplicate` means
  the relay delivered a request this environment had already run.
- `t3_webhook_runs_total` by `outcome` (`started`, `skipped`, `failed`) for the runs those
  deliveries start, which happen after the sender has its answer.
- `t3_webhook_held_delay` for how long requests the relay held waited before arriving.

- `t3_secret_requests_total` by `status` (`saved`, `declined`, `cancelled`, `timed_out`) for secrets
  agents asked users for, and `t3_secret_refs_consumed_total` by `result` (`used`, `rejected`) for
  tools redeeming them. Neither ever carries a value.

`ScheduledTaskService.triggerWebhook` spans carry the same outcome per request, and each run
started from a delivery is its own `ScheduledTaskService.runWebhookDelivery` trace. For a request
the relay forwarded, the span also goes to the T3 Connect trace export as a child of the relay's
span; requests that reach the environment directly never join a sender's trace.

Use metrics when the question is:

- "is this always slow?"
- "did this get worse after a change?"
- "which command type is failing most often?"

Use traces when the question is:

- "what happened in this specific request?"
- "which child span caused this one slow interaction?"
- "what logs were emitted inside the failing flow?"

## Common Workflows

### "Why did this request fail?"

1. Start with the local NDJSON file.
2. Find `effect-span` records where `exit._tag != "Success"`.
3. Group by `traceId`.
4. Inspect sibling spans and span events.
5. If needed, move to Tempo for the full trace tree.

### "Why is the UI feeling slow?"

1. Search for slow top-level spans in the trace file or Tempo.
2. Check child spans for sqlite, git, provider, or terminal work.
3. Look at the matching duration metrics to see whether the slowness is systemic.

### "Are git hooks causing latency?"

1. Filter `git.operation` spans.
2. Inspect `git.hook.started` and `git.hook.finished` events.
3. Compare hook timing to the enclosing git span duration.

### "Why do I have spans locally but nothing in Grafana?"

Usually one of these is true:

- `T3CODE_OTLP_TRACES_URL` was not set
- the app was launched from a different environment than the one where you exported the vars
- the app was not fully restarted after changing env
- Grafana is looking at the wrong time range or service name

If the local NDJSON file is updating, local tracing is working. The problem is almost always OTLP export configuration or process startup.

## How To Think About Adding Tracing To Future Code

### Prefer Boundaries Over Tiny Helpers

Good span boundaries:

- RPC methods
- orchestration command handling
- provider adapter calls
- external process calls
- persistence writes
- queue handoffs

Avoid tracing every tiny helper. Most helpers should inherit the active span rather than create a new one.

### Reuse `Effect.fn(...)` Where It Already Exists

The codebase already uses `Effect.fn("name")` heavily. That should usually be your first tracing boundary.

For ad hoc work:

```ts
import { Effect } from "effect";

const runThing = Effect.gen(function* () {
  yield* Effect.annotateCurrentSpan({
    "thing.id": "abc123",
    "thing.kind": "example",
  });

  yield* Effect.logInfo("starting thing");
  return yield* doWork();
}).pipe(Effect.withSpan("thing.run"));
```

### Put High-Cardinality Detail On Spans

Use span annotations for IDs, paths, and other detailed context:

```ts
yield *
  Effect.annotateCurrentSpan({
    "provider.thread_id": input.threadId,
    "provider.request_id": input.requestId,
    "git.cwd": input.cwd,
  });
```

### Keep Metric Labels Low Cardinality

Good metric labels:

- operation kind
- method name
- provider kind
- aggregate kind
- outcome

Bad metric labels:

- raw thread IDs
- command IDs
- file paths
- cwd
- full prompts
- full model strings when a normalized family label would do

Detailed context belongs on spans, not metrics.

### Use Logs As Span Events

Logs inside a span become part of the trace story:

```ts
yield * Effect.logInfo("starting provider turn");
yield * Effect.logDebug("waiting for approval response");
```

Those messages show up as span events because `Logger.tracerLogger` is installed.

### Use The Pipeable Metrics API

`withMetrics(...)` is the default way to attach a counter and timer to an effect:

```ts
import { someCounter, someDuration, withMetrics } from "../observability/Metrics.ts";

const program = doWork().pipe(
  withMetrics({
    counter: someCounter,
    timer: someDuration,
    attributes: {
      operation: "work",
    },
  }),
);
```

## Detailed API Reference

### Runtime Wiring

The server observability layer is assembled in `apps/server/src/observability/Observability.ts`.

It provides:

- pretty stdout logger
- `Logger.tracerLogger`
- local NDJSON tracer
- optional OTLP trace exporter
- optional OTLP metrics exporter
- optional OTLP log exporter
- Effect trace-level and timing refs

The desktop main process is a second producer, assembled in
`apps/desktop/src/app/DesktopObservability.ts`. It reads the same `T3CODE_OTLP_*` names and the same
Settings entries as the backend it supervises, and covers work the backend cannot see: app startup,
window and menu handling, backend supervision, and updates. It reports as service
`t3code-desktop`, so a collector shows it alongside the backend rather than mixed into it. It
exports traces and logs only; the main process records no metrics, so the metrics endpoint applies
to the backend alone.

### Env Vars

Local trace file:

- `T3CODE_TRACE_FILE`: override trace file path
- `T3CODE_TRACE_MAX_BYTES`: per-file rotation size, default `10485760`
- `T3CODE_TRACE_MAX_FILES`: rotated file count, default `10`
- `T3CODE_TRACE_BATCH_WINDOW_MS`: flush window, default `1000`
- `T3CODE_TRACE_MIN_LEVEL`: minimum trace level, default `Info`
- `T3CODE_TRACE_TIMING_ENABLED`: enable timing metadata, default `true`

OTLP export:

- `T3CODE_OTLP_TRACES_URL`: OTLP trace endpoint
- `T3CODE_OTLP_METRICS_URL`: OTLP metric endpoint
- `T3CODE_OTLP_LOGS_URL`: OTLP log endpoint
- `T3CODE_OTLP_EXPORT_INTERVAL_MS`: export interval, default `10000`
- `T3CODE_OTLP_HEADERS`: extra headers for all three exporters, same format as
  `OTEL_EXPORTER_OTLP_HEADERS`: comma-separated `key=value` pairs with percent-encoded values.
- `T3CODE_OTLP_PROTOCOL`: `http/json` (default) or `http/protobuf`

The server and the desktop app also read the standard
`OTEL_EXPORTER_OTLP_{TRACES,METRICS,LOGS}_ENDPOINT` and generic `OTEL_EXPORTER_OTLP_ENDPOINT` (with
`/v1/traces`, `/v1/metrics`, or `/v1/logs` appended), for a collector expecting those instead. A
non-blank `T3CODE_OTLP_*_URL` wins over either, and a per-signal endpoint wins over the generic one
for its signal. A blank value counts as unset. A signal with an OTEL endpoint takes its headers from
`OTEL_EXPORTER_OTLP_HEADERS` and its protocol from `OTEL_EXPORTER_OTLP_PROTOCOL` (default
`http/protobuf`, read case-insensitively), and a per-signal
`OTEL_EXPORTER_OTLP_{TRACES,METRICS,LOGS}_HEADERS` or `_PROTOCOL` wins over the generic one for its
signal. `T3CODE_OTLP_HEADERS` and `T3CODE_OTLP_PROTOCOL` never apply to it. An endpoint that is not
an `http` or `https` URL, a protocol other than `http/protobuf` or `http/json` such as `grpc`, or
headers that are not `key=value` pairs with percent-encoded values turn that signal's export off
with a startup warning, rather than sending it to the Settings endpoint.

Service names are fixed: `t3code-server` for the backend and `t3code-desktop` for the desktop main
process, both in `service.namespace` `t3code`. `OTEL_SERVICE_NAME` and a `service.name` or
`service.namespace` in `OTEL_RESOURCE_ATTRIBUTES` are ignored. Tell installations apart with other
resource attributes, such as `OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=development`.

If the OTLP URLs are unset, local tracing still works, metrics stay in-process only, and logs stay
on stdout only.

### The Kill Switch

`T3CODE_OTEL_SDK_DISABLED` and `OTEL_SDK_DISABLED` turn off every OTLP export in both the server and
the desktop main process, overriding any endpoint from the environment or Settings. Local trace
files and stdout logs are unaffected.

`T3CODE_OTEL_SDK_DISABLED` wins when set, so `T3CODE_OTEL_SDK_DISABLED=false` re-enables export on a
machine that sets `OTEL_SDK_DISABLED` for everything else. It accepts the usual boolean spellings
(`true`/`false`, `yes`/`no`, `on`/`off`, `1`/`0`, `y`/`n`). `OTEL_SDK_DISABLED` follows the
OpenTelemetry specification and only `true` disables export, so `OTEL_SDK_DISABLED=1` does not.
Values are case-insensitive and trimmed. An unrecognized value is ignored with a startup warning.

`OTEL_TRACES_EXPORTER`, `OTEL_METRICS_EXPORTER`, or `OTEL_LOGS_EXPORTER` set to `none` turns off
just that signal, overriding an OTEL endpoint and the Settings endpoint. A `T3CODE_OTLP_*_URL` still
wins for its signal. `otlp` is the default, and any other exporter name, such as `console` or
`prometheus`, is ignored with a startup warning.

### What Is Instrumented Today

Current high-value span and metric boundaries include:

- WebSocket RPC request spans (`ws.rpc.<method>`) and metrics in
  `apps/server/src/observability/RpcInstrumentation.ts`
- startup phases
- orchestration command processing
- provider session and turn operations
- git command execution and git hook events
- terminal session lifecycle
- sqlite query execution
- event loop stalls (`server.eventLoop.stall`)

### Current Constraints

- logs outside spans are not persisted in the trace file; SSH-managed launch stdout/stderr is still
  captured in its launcher log
- metrics are not snapshotted locally

## Heap Snapshots

To see what a long-running server holds in memory, send it `SIGUSR2`. The server writes a V8 heap
snapshot to its logs dir and logs the path. This works for desktop, `npx t3`, and service installs
on macOS and Linux. Windows has no `SIGUSR2`.

Send the signal to the server pid in `server-runtime.json`, which sits in the server's state dir
next to the `logs` dir. For a dev server or a `--home-dir` launch, use that server's state dir from
[Traces](#traces). Do not send it to the desktop app or the service launcher: a process without the
handler exits on `SIGUSR2`. After a crash the file can keep a stale pid that now belongs to a
different process, so check the pid first.

```bash
pid="$(jq .pid "${T3CODE_HOME:-$HOME/.t3}/userdata/server-runtime.json")"
ps -p "$pid" -o command=
```

If `ps` shows the T3 Code server, send the signal:

```bash
kill -USR2 "$pid"
```

The file is `<logsDir>/server-<pid>-<timestamp>.heapsnapshot`, next to `server.trace.ndjson`. To
open it, use the Memory tab in Chrome DevTools and select Load.

Before you take one:

- The server stops while it writes the file. For a large heap this can take a minute or more.
  Connected clients can reconnect during the pause, and an event loop monitor, if the server has
  one, records the pause as a stall. Send the signal once. A second signal sent during a write
  takes another snapshot after the first one finishes.
- The write needs about as much free memory as the heap uses. On a machine that is already
  swapping, it can make the problem worse or crash the server.
- The file contains everything in server memory, including tokens, secrets, and thread content. Do
  not share it publicly. Delete it when you are done, because storage cleanup does not remove it.