Project: Yellowstone-Vixen (Real-time Indexer)

• Exactly-once processing per (signature, ix_index) tuple (Postgres UNIQUE + ON CONFLICT)
• Reorg safety within 64-slot Solana confirmation window
• Sub-150ms p99 latency for WS subscribers
• Sustained 3,000+ TPS single-node throughput
• Idempotent parser re-execution on restarts, with dead-letter queues for malformed instructions
• Sub-1s staleness in terminal views (write-through Redis, 5s TTLs, slot-based invalidation)

indexer-bin (Entry Point & Event Loop)

• Responsibility: Firehose connection management, block-by-block orchestration, coordinating parsers
• Concurrency model: 2-task Tokio executor Task 1: FirehoseClient::stream_blocks() — indefinite reconnect loop with exponential backoff; sends raw BlockRef into bounded MPSC channel (capacity: 1024) Task 2: Writer loop — consumes blocks, calls parser functions in-sequence, batches inserts to Postgres, emits events to Redis
• Failure modes: Firehose disconnect → recover within 30s (max backoff); parser error → log + skip block + increment dead-letter counter; DB write failure → log + retry on next iteration (at-most-once semantic, corrected by idempotence)
• Trade-off: Synchronous parser invocation (simpler error handling) vs. parallel parsing (not needed; CPU-bound parsing < 10ms/block, I/O bottleneck dominates)

indexer-core (Logic & Data Access)

• Responsibility: Instruction parsing, model serialization, database schema management, Redis client initialization
• Modules: lib.rs — public API surface models.rs — sqlx-derived structs (Mint, TokenTransfer, BondingCurveTrade, Candle, Balance) spl_parser.rs — SPL Token instruction discriminators + byte-level parsing bonding_parser.rs — Pump.fun Anchor IDL interpretation raydium_parser.rs — Raydium AMM v3/v4 swap layout meteora_parser.rs — Meteora DLMM v1/v2 swap layout db.rs — sqlx prepared statements, batch insert functions, migration runner redis.rs — publish_trade(), publish_transfer() functions config.rs — serde config deserialization with env override firehose.rs — Firehose gRPC connector (stub for now; tonic integration planned)
• Invariants: All parser functions are pure (no side-effects); all DB operations are idempotent via ON CONFLICT; no global state (threadsafe).
• Testing: Unit tests per parser with hardcoded block fixtures

indexer-api (Query & Subscription Facade)

• Responsibility: HTTP REST queries, WebSocket real-time subscriptions, metrics export
• Routes: GET /health — 200 OK (Kubernetes liveness) GET /metrics — JSON counters (token_transfers_count, bonding_trades_count, last_processed_slot, total_mints) GET /transfers/:mint — recent transfers for a mint (query: limit=100, before_slot) GET /holders/:mint — token holders (query: limit=100, offset=0) GET /trades/:mint — bonding trades for a mint (query: limit=100, before_slot) GET /portfolio/:wallet — aggregated balances + recent transfers (query: limit) WS /subscribe/{mint} — real-time transfer + trade events (via Postgres LISTEN/NOTIFY → Redis Pub/Sub fanout OR direct WebSocket broadcast)
• Concurrency: Full async/await via Axum + Tokio runtime; each WS connection spawns independent task; broadcast channel for multi-subscriber fanout
• Failure semantics: Missing mint → 404; DB unavailable → 503; WS disconnect → client reconnect (exponential backoff in terminal)

Latency breakdown (typical):
• Firehose gRPC delivery: ~0ms (in-band)
• Parser execution (CPU): ~5–10ms (rayon + simd opportunities unused currently)
• Postgres batch insert: ~20–50ms (depends on row count; COPY would be 3–5ms with 1000s rows)
• Redis stream write: ~1–3ms
• Axum HTTP handler: ~2–5ms
• WebSocket push: ~1–5ms
• Total p99: ~100–150ms (measured: confirm via distributed tracing in Phase 3)

• Slot + signature uniqueness:** UNIQUE (signature, ix_index) ensures duplicate processing fails gracefully (ON CONFLICT DO NOTHING)
• Blockhash comparison:** Stored in `indexer_events.payload` (JSONB) or separate slot_hash table (future Phase 3)
• Cascade invalidation:** All derived views (balances, candles) recomputed from transfers; Redis TTLs (5s) ensure stale cache expires
• Checkpoint atomicity:** last_processed_slot updated within same transaction as data, preventing checkpoint-lag attacks

Network sends 5,000 TPS over 400ms slot duration (12,500 total TXs per slot). Parser CPU utilization approaches 80%; Postgres write throughput saturates.

• MPSC channel backpressure:** Firehose sender blocks if channel capacity (1024 blocks ≈ 400s of data) is exceeded. Recovery: exponential backoff reconnect.
• Batch coalescing:** Parser output batched (in memory) before DB insert. If batch >1000 transfers, flush early (adaptive).
• DB connection pool:** Fixed 10 connections; additional block processors queue (Tokio fair scheduler). At 100% pool utilization, add block processor tasks as needed (future autoscaling).
• Redis pub/sub fanout:** Non-blocking; WS subscribers that lag get dropped clients-side (TCP buffer exhaustion); terminal reconnects.
• Metrics & alerting:** Monitor `parser_queue_depth`, `db_write_latency_p99`, `ws_subscribers_dropped_count`.