Observability & Reliability

Logging Architecture

Framework: tracing with tracing-subscriber and EnvFilter.
Level: Configured via RUST_LOG (default: api=info,tower_http=info).
Middleware: tower_http::trace::TraceLayer records request latency, paths, client IPs, and status codes.

Framework: Python standard logging with structured JSON/extra dictionary context.
Correlation ID: Distributed tracing propagates x-correlation-id across HTTP headers, gRPC metadata, and Redis Stream event envelopes.

OpenTier provides layered health checks covering network liveness and downstream dependency readiness:

Endpoint	Protocol	Type	Checks
`GET /health`	HTTP	Liveness	Basic Rust process uptime and gateway responsiveness.
`GET /health/api`	HTTP	Liveness	Detailed gateway metrics, active connections, and runtime uptime.
`GET /health/intelligence`	HTTP	Proxy Liveness	Gateway proxies unary call to Python gRPC `Health.Check`.
`Health.Ready`	gRPC	Deep Readiness	Verifies active connections to PostgreSQL 18, Qdrant, and Redis 8.

Background workers in OpenTier v1.1.0 run an autonomous observability cycle:

When a worker boots, it publishes an ops.worker.started.v1 envelope to opentier:ops:heartbeat (HEARTBEAT_STREAM).
The heartbeat consumer group picks up and acknowledges the message, logging heartbeat processed with the msg_id and correlation_id.
This confirms full roundtrip read/write capability against the Redis Streams backbone.

Every 60 seconds, background workers execute stream_metrics():

Inspects xinfo_stream(name) to log total stream length.
Inspects xpending(name, "workers") to calculate pending/unacknowledged consumer lag.
Inspects xlen(Streams.dlq(name)) to report Dead-Letter Queue (DLQ) depth.


{
  "event": "stream_metrics",
  "stream": "opentier:intel:ingestion_jobs",
  "length": 4,
  "pending": 0,
  "dlq_depth": 0
}

Startup Sweep: On boot, the worker scans PostgreSQL for any ingestion jobs stranded in processing state by previous worker crashes and re-enqueues them to Redis.
Periodic Sweep: Runs every 5 minutes to automatically recover any jobs stranded by transient worker kills.

Each stream maintains a dedicated DLQ:
- opentier:intel:ingestion_jobs:dlq
- opentier:intel:chat_events:dlq
When a consumer handler raises exceptions after max_retries = 3, the envelope is diverted to the DLQ with the error payload and the original message is acknowledged (XACK).
Operational CLI script: python scripts/dlq.py allows operators to inspect, replay, or purge dead-lettered events.

Session Cleanup Task (auth/background.rs): Runs a detached Tokio task every hour:


sqlx::query!("DELETE FROM sessions WHERE expires_at < NOW()").execute(&db).await;

Credit Hold Expiration: Periodic reaper releases unconsumed credit holds exceeding timeout limits.

All database migrations are centralized in server/api/migrations/:

SQLx Offline Mode: .sqlx/ cache ensures Rust builds are verified at compile time against live schema requirements without requiring an active database during CI/CD.
Docker Auto-Migration: The gateway runs sqlx::migrate!() upon startup before opening port 4000, guaranteeing consistent table definitions across all running containers.