Skip to content

Daedalus

This document describes capabilities of our Daedalus Go library. In general, Daedalus standardizes a number of tasks common to most services, such as configuration, communication, observability, ...

Configurations

When loading config settings for a service, you should use lyceum.technology/daedalus/config to load them into a struct type that you have defined. Values can come from various sources (as defined below). Field tags are used to control how values are loaded.

Quickstart

Take the following minimal example, with a number of fields and a nested struct tagged with config. We refer to structs tagged with config as "config sections".

import "lyceum.technology/daedalus/config"

type Settings struct {
    HTTP struct {
        Host    string                  `config:"host" required:"true"`
        Port    int                     `config:"port" default:"8080"`
        Timeout config.PositiveDuration `config:"timeout" default:"5s"`
    } `config:"http"`
}

// ...

settings, err := config.Load[Settings]()

You should call Load at startup and handle its error (e.g. by panicking) before using the settings.

Without options, Load reads files in /secrets (if it exists), then environment variables. It considers files beginning with lyc_ and variables beginning with LYC_. Both sources turn these names into config keys (see below). Environment variables win if both sources provide the same config key. Files and environment variables of this form that do not match any config key cause an error by default.

E.g. for the struct above, you might just set LYC_HTTP__HOST=localhost with the port and timeout using their defaults. The PositiveDuration type causes an error if a non-positive timeout value is supplied. Call settings.HTTP.Timeout.Duration() to obtain a time.Duration.

config.Load[Settings](config.WithSources(mySource)) replaces the default sources with a mySource source. This source must implement config.Source with Name() string and Read() (map[string]string, error) methods. Read returns config keys and their string values. You can pass multiple sources, with later sources overriding earlier ones if both provide the same config key.

Config keys

Config keys have the form X.Y.Z with dots separating nested structs. E.g. HTTP.Host would correspond to the http.host config key. Config keys are matched to (nested) fields in the struct type passed to Load. Config sources turn whatever they read config values from into config keys according to predefined rules. For example, these all provide http.host:

  • /secrets/lyc_http__host as a file containing the value.
  • LYC_HTTP__HOST in the environment.

Every exported field in a config section needs a config tag. A source cannot provide both a section key and one of its children, such as http and http.host. Names that map to the same key also cause an error.

Field tags

In addition to the mandatory config tag, a number of optional tags can be supplied:

  • required:"true" on a leaf field requires a source value or a default. On a config section, it requires an explicit value for at least one child.

  • default:"value" defines a default value for a leaf field. A child's default does not satisfy required on its config section.

  • allowempty:"true" permits passing an empty string, slice, map or array. A string field with default:"" requires this tag.

  • format:"json" allows decoding a value into a map, slice or struct. Use json tags on fields inside of such a struct. A JSON default must be valid JSON, such as default:"[]" for a slice.

  • sensitive:"true" requires a trusted config source (see below). Put it on fields, not on a config section. Sensitive fields cannot have defaults.

Use a single JSON value for a list or an object:

type BackendSettings struct {
    Backends []string `config:"backends" format:"json" required:"true"`
}

Set LYC_BACKENDS='["api-1", "api-2"]' for this field. An empty string is not valid JSON. An empty JSON array needs allowempty:"true".

A map without format:"json" can read child keys. For example, LYC_LABELS__TEAM=infra sets labels.team in a map[string]string field tagged config:"labels".

Built-in types

  • config.PositiveDuration accepts positive durations (e.g. 5s). It has a Duration method via which a time.Duration can be obtained.

  • config.NonNegativeDuration is like config.PositiveDuration but accepts zero durations.

  • config.HTTPURL accepts an HTTP or HTTPS base URL with a host. It rejects credentials, queries, fragments and a trailing slash.

  • time.Time accepts an RFC3339 timestamp in UTC with a Z suffix.

Sensitive values

With the default sources, provide sensitive values through a file such as /secrets/lyc_database__password. A custom source passed to WithSources can also provide them if it implements config.SensitiveSource.

For the default file source, the directory and file must be owned by root. The directory cannot be writable by group or others. The resolved file must be regular and have no write, execute or special permission bits. The process must also be able to read it.

An environment variable cannot provide a sensitive key. Load checks every source that provides the key, even if a later source overrides its value. For example, LYC_DATABASE__PASSWORD causes an error even when the file exists.

The same rules apply to keys below a sensitive map field.

Validation and hooks

A Validate() error method can be implemented on the struct type passed to Load. It runs after loading all fields. Use it to validate relationships between fields, such as whether one value exceeds another.

WithHooks can convert or check a field before unmarshalling. Hooks run after JSON decoding.

Integration tests

lyceum.technology/daedalus/testutil can help run a service's integration tests against real Postgres, Redis and RabbitMQ etc. containers.

Quickstart

Each integration test package needs a TestMain that calls RunIntegrationTests. Name the service's component with WithComponent and pass a With*Pool option for every kind of container the package's tests use:

package testsintegration

import (
    "testing"

    dtest "lyceum.technology/daedalus/testutil"
)

func TestMain(m *testing.M) {
    dtest.RunIntegrationTests(
        m,
        dtest.WithComponent("croesus"),
        dtest.WithDBPool(),
        dtest.WithRedisPool(),
        dtest.WithRabbitMQPool(),
    )
}

A test then asks for its own isolated piece of a container, e.g.:

func TestSomething(t *testing.T) {
    t.Parallel()

    tx := dtest.NewPooledDBTx(t)
    redis := dtest.NewPooledRedisContainer(t)
    rabbitmq := dtest.NewPooledRabbitMQContainer(t)

    // ...
}

These are set up such that tests can safely run in parallel without seeing each other's data.

Containers

Each pool starts one container per component. Tests get their isolated piece via the following functions:

  • NewPooledDBTx(t) returns a transaction on a shared database that is rolled back when t finishes. It is cheap, so prefer it.

  • NewPooledDBContainer(t) returns a database of its own for t, copied from the migrated template. Use it when a transaction does not suffice, e.g. when the code under test commits, opens its own connections or uses the database from several goroutines at once.

  • NewPooledRedisContainer(t) returns an empty Redis database, flushed when t finishes. Connect to it via its URL().

  • NewPooledRabbitMQContainer(t) returns an empty RabbitMQ vhost, deleted when t finishes. Connect to it via its AMQPURL().

Calling one of these without the matching With*Pool option fails the test.

Migrations

The database pool applies the component's migrations from db/migrations/<component>/*.up.sql once and copies the result for every test. A test package can add SQL files to run before and after them in its own resources/migrations/before/ and resources/migrations/after/ directories, e.g. to create roles the migrations expect or to seed data. Changed migrations are picked up on the next run.

Other settings

Further options to RunIntegrationTests:

  • WithSetup(setup) runs any other shared setup once before the package's tests, and its cleanup once after them.

  • WithSetupTimeout(timeout) bounds how long all setups may take together. It defaults to three minutes.

In addition, some environment variables can be used to control test behavior:

  • Set LYC_TEST_LOG_LEVEL (e.g. to debug) to set log level. By default tests log nothing.

  • Set LYC_TEST_REUSE_CONTAINERS to true to reuse the pools' containers across runs, so only the first run pays for starting them. A container whose configuration changed is replaced automatically. By default every run starts fresh containers.

Integration tests are skipped entirely when running with go test -short.