Monitoring
We observe our services with Grafana Cloud ⧉. Rather than run our own Prometheus, Loki and Grafana instances, we push metrics and logs to Grafana Cloud and use its hosted Grafana to explore them. This keeps the storage, retention and availability of our observability data off our plate.
The concrete stacks, endpoints, ports and dashboards are listed in the monitoring reference; this page explains how the pieces fit together and why.
One Stack per Environment
Each deployment environment gets its own Grafana Cloud stack, which gives it an isolated Grafana instance under its own subdomain. Keeping the environments in separate stacks means data from one never mixes with another: each has its own read/write tokens and its own Terraform state.
How Data Gets There
The agent that ships data to Grafana Cloud is Grafana
Alloy. It scrapes metrics locally and forwards
them to the stack's hosted Prometheus via remote_write, and it collects logs
and pushes them to the stack's hosted Loki, authenticating with the
per-environment tokens.
Alloy runs both on our VMs (execlets, gateways, the autoscaler host), where it is installed by an Ansible role and runs as a systemd service, and on Cloud Run for the services that live there, where it is deployed as a container by Terraform.
What each Alloy instance collects is toggled through the role's variables, so a
plain host and a GPU execlet ship different data: host metrics from the node
exporter everywhere, GPU (DCGM) metrics only on machines with GPUs, and the
streamer's or Hydra's service metrics where those run. For logs, Alloy tails
systemd journal units and Docker Compose containers; a container opts in and
attaches labels through technology.lyceum.observability.* labels, so its logs
arrive in Loki tagged with its service name.
Declarative Dashboards
Our dashboards live in the repository as code, not as hand-edited JSON in the Grafana UI. This keeps them reviewable, diffable and reproducible across all three stacks.
Each dashboard is a Python script. And because dashboards are regular Python, a change is a normal pull request and gets deployed to every environment the same way.
Streamer Smoke Test
Dashboards tell us how the system looks from the inside, but they do not prove that a customer can actually run a job. That end-to-end check is a scheduled smoke test, defined as a GitHub Actions workflow.
Every hour the workflow submits a small job through the public API against a
couple of hardware profiles and waits for each to finish. This exercises the
whole path: the gateway, the streamer, an execlet actually picking up and running
the job, and the result coming back. A profile passes only if its job reaches the
completed state within the timeout.
A final job then always runs and posts a summary to the #monitoring Slack
channel, escalating its wording from an all-clear to a channel-wide alert
depending on how many profiles failed. The workflow itself is also marked as
failed when any profile did not complete, so the failure is visible in GitHub in
addition to Slack.