Monitoring
Concrete values for our monitoring setup: Grafana Cloud stacks, the Alloy data pipeline, dashboards and the streamer smoke test. See the explanation for how these fit together.
Grafana Cloud Stacks
One stack per environment, each with its own subdomain, tokens and Terraform
state. Provisioned by infra/terraform/grafana.
| Environment | Stack slug | Grafana subdomain |
|---|---|---|
devel |
lycdevel |
lycdevel.grafana.net |
integration |
lycintegration |
lycintegration.grafana.net |
production |
lycprod |
lycprod.grafana.net |
Production stack naming
The production stack is lycprod rather than lycproduction: when we
renamed the deploy environment from prod to production we already had a
lycprod stack and could not create a fourth one, so we kept it.
The stacks, their service accounts and the access-policy tokens used to read and
write metrics and logs are all provisioned via Terraform in
infra/terraform/grafana. Tokens live in GCP Secret Manager with
environment-scoped access. Each stack has four access-policy tokens: logs:read,
logs:write, metrics:read, metrics:write.
Alloy Data Pipeline
Grafana Alloy ⧉ scrapes locally and forwards
metrics to hosted Prometheus (remote_write) and logs to hosted Loki, over
basic auth with the per-environment tokens.
| On VMs | On Cloud Run | |
|---|---|---|
| Deployed by | Ansible role infra/ansible/playbooks/roles/alloy |
Terraform module infra/terraform/_modules/gcp/cloud_run_grafana |
| Runs as | systemd service | container |
| Alloy HTTP port | 12345 |
— |
Metrics are scraped every 10 seconds by default. What Alloy collects is toggled through the Ansible role's variables:
| Source | What | Where |
|---|---|---|
| Node exporter | CPU, RAM, disk, network | all hosts |
| Nvidia DCGM | GPU metrics | GPU machines |
Streamer endpoint (:8721) |
streamer service metrics | streamer host, when enabled |
Hydra endpoint (:8080) |
autoscaler service metrics | autoscaler host, when enabled |
| systemd journal | service logs → Loki | all hosts |
| Docker Compose containers | container logs → Loki | opt-in via labels |
Containers opt into log collection and add labels through
technology.lyceum.observability.* container labels; their logs arrive in Loki
tagged with the service name.
Dashboards
Defined as Python scripts in infra/terraform/grafana/dashboards, each exposing
a dashboard() function built with the Grafana Foundation
SDK. Terraform runs each
through the _generate runner and applies it into the "Lyceum" folder in every
stack.
| Dashboard | Script | Shows |
|---|---|---|
| Build Info | build_info.py |
job, service, version and commit of each running binary |
| Streamer Status | streamer_status.py |
running/queued jobs, connected execlets by hardware profile, per-execlet ping latency (streamer_* metrics) |
| Instance Metrics | instance_metrics.py |
per-instance CPU, RAM, disk and network (node exporter), metrics are also shown on customer dashboard |
| Hydra Status | hydra_status.py |
state of the autoscaler's instance pools, i.e. the compute instances we host with different providers |
Streamer Smoke Test
An hourly end-to-end check defined in .github/workflows/api-monitoring.yml.
| Schedule | cron: '16 * * * *' (hourly), plus workflow_dispatch |
| Concurrency | single run at a time (cancel-in-progress: false) |
| Job script | scripts/api_calls_for_monitoring.py --hardware-profile <profile> --wait 3 |
| Pass condition | last output line contains completed |
| Slack script | scripts/slack_monitoring_messager.py → SLACK_MONITORING_WEBHOOK_URL |
| Slack channel | #monitoring |
A slack-status job runs if: always() after both checks and posts a summary
whose wording escalates with the number of failures:
| Monitored profiles | Message |
|---|---|
| All pass | 🎉 all systems operational |
| One fails | ⚠️ execution issues, @channel alert |
| Both fail | 🚨 both profiles failing, @channel alert |
The workflow is additionally marked as failed whenever either profile did not complete, surfacing the failure in GitHub alongside the Slack message.