MetricServer

Location: iris/metric_server/ ⧉ Language: Go Role: Telemetry broker between InferenceWorkers and Prometheus

MetricServer receives raw metrics from InferenceWorkers, maintains replica health in Redis, exposes a Prometheus-compatible /prom endpoint, and reports Execlet replica health to the Gateway. The Iris operator reports Kubernetes replica health.

Planned direction

This may reinvent a path to Grafana that already exists for every Cloud Run service in Terraform. The bespoke path may be replaced by the shared one. See /iris.

Architecture

InferenceWorker (each replica)
      │
      │ POST /metrics
      │ (every 30s)
      │
      ▼
MetricServer
      │
      ├─► Redis
      │   └─ Update replica health state
      │       + last report timestamp
      │
      ├─► Gateway
      │   └─ POST /report_replica_status
      │       (worker reports and stale transitions)
      │
      └─► In-Memory Prometheus Registry
          └─ Update gauges, histograms, counters
              │
              │ GET /prom
              │ (scrape every 15s)
              ▼
          Prometheus
              │
              └─► DeploymentScaler
                  (queries for scaling decisions)

Key responsibilities

1. Metrics ingestion (`POST /metrics`)

Receives metrics payloads from InferenceWorkers containing inference records, health snapshots, and resource metrics. Updates Redis, records Prometheus metrics, and forwards worker health to the Gateway. The Gateway applies it only to Execlet replicas.

2. Health state management (Redis)

Maintains replica health state in Redis (replica_health:<replica_id>) with health status, last report timestamp, and deployment metadata. Replicas are healthy if healthy == true in latest report and reported within stale threshold.

3. Staleness checking (background loop)

Runs every STALENESS_CHECK_INTERVAL (default 10 minutes). Marks replicas unhealthy if no report within STALE_THRESHOLD (60s). Evicts replicas from Redis after EVICTION_THRESHOLD (1h) to prevent memory leaks.

4. Prometheus exposition (`GET /prom`)

Exposes metrics in Prometheus text-based format:

Metric	Type	Labels	Purpose
`lyceum_inference_latency_millis`	Histogram	`deployment_id`, `user_id`, `hf_model_id`	P50, P95, P99 latency calculations
`lyceum_inference_requests_total`	Counter	`deployment_id`, `user_id`, `hf_model_id`, `status_code`	RPS calculations, error rates
`lyceum_inference_tokens_total`	Counter	`deployment_id`, `user_id`, `hf_model_id`, `token_type`	Token usage tracking
`lyceum_inference_instance_health`	Gauge	`deployment_id`, `replica_id`, `user_id`, `hf_model_id`	Per-replica health status (1=healthy, 0=unhealthy)
`lyceum_inference_instance_last_report_age_seconds`	Gauge	`deployment_id`, `replica_id`	Detect reporting delays
`lyceum_inference_kv_cache_usage_percent`	Gauge	`deployment_id`, `replica_id`	Monitor KV cache pressure
`lyceum_inference_gpu_utilization_percent`	Gauge	`deployment_id`, `replica_id`, `gpu_index`	GPU utilization tracking
`lyceum_inference_queue_depth`	Gauge	`deployment_id`, `replica_id`	Queue backlog monitoring

5. Health reporting to Gateway

Calls POST /api/v2/internal/inference/report_replica_status for worker health reports. The Gateway applies them to Execlet replicas. Kubernetes replica health comes from the Iris operator. MetricServer also reports an unhealthy transition when a worker stops sending metrics.

Configuration

Environment variables

Variable	Default	Description
`LYC_REDIS_ADDR`	`localhost:6379`	Redis server address
`LYC_REDIS_PASSWORD`	(empty)	Redis AUTH password
`LYC_API_ADDR`	`:8081`	HTTP listen address
`LYC_STALE_THRESHOLD`	`60s`	Duration after which non-reporting instance is unhealthy
`LYC_PROM_SCRAPE_INTERVAL`	`15s`	Prometheus scrape interval (for shutdown grace tuning)
`LYC_GRACEFUL_SCRAPE_DELAY`	`20s`	Duration to wait before shutting down API to allow final Prometheus scrape
`LYC_STALENESS_CHECK_INTERVAL`	`10m`	How often to check for stale instances
`LYC_EVICTION_THRESHOLD`	`1h`	Remove instances from Redis after being unhealthy this long
`METRIC_SERVER_SERVICE_TOKEN`	(empty)	Bearer token for `POST /metrics` auth and Gateway API calls. If unset, auth and Gateway reporting disabled
`API_BASE_URL`	(empty)	Gateway URL for health reporting. If unset, reporting disabled
`DEFAULT_TIMEOUT_S`	`60.0`	HTTP timeout for Gateway calls (in seconds)

Note: Gateway reporting requires both API_BASE_URL and METRIC_SERVER_SERVICE_TOKEN to be set. The same service token is used for both incoming metrics auth and outgoing Gateway calls.

Command-line flags

Flags take precedence over environment variables (for LYC_* variables only):

./metric_server \
  -redis-addr=redis:6379 \
  -redis-password=secret \
  -api-addr=:8081 \
  -stale-threshold=60s \
  -graceful-scrape-delay=20s \
  -prom-scrape-interval=15s

Operational considerations

High availability

Multiple MetricServer instances can run simultaneously
All state in Redis (no local state)
Prometheus scrapes all instances (aggregates automatically)

Failure modes

Redis Down: - Metrics ingestion fails (returns 500) - Workers retry on next flush interval - Prometheus scraping continues (serves cached metrics)

Gateway Down: - Health reporting fails - Logged but doesn't block metrics ingestion - Retries on next health change

Prometheus Down: - MetricServer unaffected - Metrics continue to accumulate - DeploymentScaler cannot scale (no metrics)

2026-06-042026-09-09