Skip to content

Gateway - Iris Integration

Location: /app/src/app/api/v2_streaming/external/compute/inference/lifecycle/dedicated.py ⧉ Language: Python (FastAPI) Role: Central orchestrator and source of truth for Iris deployments

Gateway is the existing Lyceum API server extended with Iris-specific endpoints. It manages the deployment lifecycle, maintains CloudSQL as the source of truth, provisions replicas via Streamer, and handles callbacks from Execlet nodes.

Responsibilities

  • Deployment Lifecycle: Create, stop, pause, and resume deployments
  • Replica Management: Provision, scale, and track replica status
  • Database: Maintain dedicated_deployments and deployment_replicas tables
  • Cache Seeding: Write initial deployment metadata to Redis
  • Docker Compose Job Rendering: Renders docker-compose template ⧉ with model container (vLLM) and InferenceWorker sidecar configuration
  • Streamer Communication: Submits rendered docker-compose jobs to Streamer, which assigns them to Execlet nodes for execution
  • Callback Handling: Receive status updates from Execlet nodes

API Endpoints

External (User-Facing)

Endpoint Method Caller Purpose
/api/v2/external/inference/create POST User (lyceum-cli or direct API) Create a new dedicated deployment
/api/v2/external/inference/stop DELETE User Stop a deployment and abort all replicas
/api/v2/external/inference/get GET User, InferenceProxy, DeploymentScaler Retrieve deployment details and replica status
/api/v2/external/inference/list GET User, DeploymentScaler List deployments (user's own or all for service tokens)
/api/v2/external/inference/replica/{replica_id}/logs GET User Get container error logs for failed replicas

Internal (Service-to-Service)

Endpoint Method Caller Purpose
/api/v2/internal/inference/scale POST DeploymentScaler Add or remove replicas from a deployment
/api/v2/internal/inference/report_replica_status POST MetricServer Report replica health status changes
/api/v2/internal/inference/pause POST DeploymentScaler Pause a deployment (scale to zero)
/api/v2/internal/inference/resume POST InferenceProxy, DeploymentScaler Resume a paused deployment
/api/v2/internal/inference/report_crashed_replicas POST DeploymentScaler Mark replicas as failed due to health check timeout

Callbacks (from Execlet)

Endpoint Method Caller Purpose
/api/internal/inference/replica/progress/{replica_id} POST Execlet Report replica status transitions (pulling → initing → running → failed)
/api/internal/inference/replica/complete/{replica_id} POST Streamer Handle VM-level crashes (when Execlet connection lost)

Configuration

Key Environment Variables

Variable Purpose
VLLM_IMAGE vLLM Docker image tag for model containers
INFERENCE_WORKER_IMAGE InferenceWorker Docker image tag
METRIC_SERVER_URL, METRIC_SERVER_SERVICE_TOKEN MetricServer connection and auth
DEPLOYMENT_SCALER_SERVICE_TOKEN Auth token for DeploymentScaler
INFERENCE_PROXY_SERVICE_TOKEN Auth token for InferenceProxy
EXECLET_HTTP_URL Streamer API URL for job submission
REDIS_URL, REDIS_PASSWORD Redis connection for cache seeding
RABBITMQ_* RabbitMQ connection details for docker-compose rendering
LYC_DB_IRIS_* CloudSQL database connection