Skip to content

Federated

This example runs sutura for one user over two data systems. The metric recurring_revenue is in Postgres, and the metric active_customers is in ClickHouse. DataHub is the catalog. One question reads both metrics and groups them by segment.

The example is in examples/federated:

Path Contents
conf/base.yaml The sutura settings
compose.yaml Postgres with TLS, ClickHouse, a live DataHub, and the catalog loader
postgres/init.sql The fact table fct_subscription_monthly
clickhouse/init.sql The dimension table dim_customer and metric table fct_customer_monthly
loader/load.py The catalog that the loader writes into DataHub
setup.sh Writes the secrets of the local services to .env and .local

The subscription rows are the CSV files of the single player example. The ClickHouse customer metric uses a four-row synthetic monthly fact table. All data is synthetic.

The settings

# sutura's settings for this example. One metric from Postgres, grouped by one dimension from
# ClickHouse. Both sources run as the deployment's own user, and DataHub says which platform is
# which source.
server:
  host: "127.0.0.1"
  port: 8080
security:
  identity: "single-user"
  single_user_because: "one operator reads both sources as the deployment's own user"
catalogs:
  # DataHub, read with one token for every caller.
  - name: "datahub"
    kind: "datahub"
    # Required for every catalog kind, and not read for DataHub.
    dir: "/unused-for-datahub"
    data_dir: "/unused-for-datahub"
    version: "example"
    endpoint: "http://127.0.0.1:8081"
    token_file: "/run/sutura/datahub-token"
    metric_property: "sutura"
    # A dataset on the `postgres` platform is read from the source `billing`, and one on the
    # `clickhouse` platform from the source `crm`.
    platforms:
      postgres: "billing"
      clickhouse: "crm"
sources:
  billing:
    kind: "postgres"
    host: "127.0.0.1"
    port: 5432
    database: "federated"
    user: "sutura"
    password_file: "/run/sutura/postgres-password"
    transport_mode: "verified"
    transport_anchors: "/run/sutura/postgres-ca.pem"
    posture: "shared-service-user"
  crm:
    kind: "clickhouse"
    host: "127.0.0.1"
    port: 8123
    user: "default"
    password_file: "/run/sutura/clickhouse-password"
    transport_mode: "plaintext"
    posture: "shared-service-user"

platforms maps the platform of each DataHub dataset to the source that serves it. The dataset fct_subscription_monthly is on the platform postgres, so sutura reads it from billing. The dataset dim_customer is on clickhouse, so sutura reads it from crm. Both sources run as the deployment's own user, and single_user_because records why.

The question

{
  "metrics": ["recurring_revenue", "active_customers"],
  "grain": "month",
  "range": { "start": "2026-06-01", "end": "2026-07-01" },
  "dimensions": ["segment"]
}

The metric recurring_revenue is on postgres__fct_subscription_monthly, while active_customers is on clickhouse__fct_customer_monthly. Both models expose segment; the first reaches it through subscription_customer. sutura reads one metric from each source and joins the two arms.

Run it

From examples/federated:

./setup.sh
docker compose up -d --wait postgres clickhouse datahub
docker compose run --rm federated-loader

The files in .local are the files that conf/base.yaml reads from /run/sutura: datahub-token, postgres-password, postgres-ca.pem and clickhouse-password. Every port of these services binds to 127.0.0.1 only.

Start sutura in a second terminal. It stays in the foreground. SUTURA_CONFIG_DIR names the directory that holds base.yaml, and the container reads the files in .local as its own user. It uses the host network, as in the single player example. The image tag ends in -musl: that image carries the Postgres driver, and the plain tag does not:

export SUTURA_VERSION=0.13.1
docker run --rm --network host --user "$(id -u):$(id -g)" \
  -v "$PWD/conf:/conf:ro" -v "$PWD/.local:/run/sutura:ro" -e SUTURA_CONFIG_DIR=/conf \
  ghcr.io/telekom/sutura:$SUTURA_VERSION-musl serve

Post the question:

curl -s -X POST http://127.0.0.1:8080/v1/query \
  -H 'content-type: application/json' \
  -d '{"metrics":["recurring_revenue","active_customers"],"grain":"month","range":{"start":"2026-06-01","end":"2026-07-01"},"dimensions":["segment"]}'

Distributed execution

server.nodes lets several sutura processes plan one combine together: the node that receives a question still answers it, but when the combine is large enough it spreads the work over the workers it declares. A question that reads one data system never goes across, even one asking for several metrics there - that is one statement at that system. A question over two or more systems, like this example's, goes across only once its legs hold more than server.nodes.across_above_bytes (or a working-set ceiling, if that key is unset). See Serving for the full server.nodes rule; the decision to spread a combine this way, and what it trades away, is summarized under Limits below.

This feature merged to main after the last tagged release, so ghcr.io/telekom/sutura:{{ sutura_version }}-musl - the image the rest of this page runs - predates it. Until a release ships it, build the image from this checkout instead:

nix build .#oci
docker load < result

That produces a local sutura:latest image with server.nodes and the worker role compiled in.

Run the coordinator and two workers

Nodes talk mutual TLS: every node's certificate must be signed by the one tls_node_authority this deployment declares, or a worker call is refused. conf/distributed/ holds three settings directories - coordinator, worker-1, worker-2 - and compose.distributed.yaml adds a one-shot service that writes a throwaway demo CA and one leaf certificate per node to .local/distributed. It is demo-only: the certificates last one day, .gitignore already covers .local/, and nothing here is fit to reuse.

With postgres, clickhouse and datahub already up from Run it:

docker compose -f compose.yaml -f compose.distributed.yaml run --rm distributed-certs

Start both workers, then the coordinator, each in its own terminal from examples/federated:

docker run --rm --network host --user "$(id -u):$(id -g)" \
  -v "$PWD/conf/distributed/worker-1:/conf:ro" -v "$PWD/.local/distributed:/run/sutura/distributed:ro" \
  -e SUTURA_CONFIG_DIR=/conf sutura:latest serve

docker run --rm --network host --user "$(id -u):$(id -g)" \
  -v "$PWD/conf/distributed/worker-2:/conf:ro" -v "$PWD/.local/distributed:/run/sutura/distributed:ro" \
  -e SUTURA_CONFIG_DIR=/conf sutura:latest serve

docker run --rm --network host --user "$(id -u):$(id -g)" \
  -v "$PWD/conf/distributed/coordinator:/conf:ro" -v "$PWD/.local:/run/sutura:ro" \
  -e SUTURA_CONFIG_DIR=/conf sutura:latest serve

The coordinator reads the same secrets as the single-node example from .local, plus the demo certificates nested under .local/distributed. Post the same question, now over TLS:

curl -sk -X POST https://127.0.0.1:8443/v1/query \
  -H 'content-type: application/json' \
  -d '{"metrics":["recurring_revenue","active_customers"],"grain":"month","range":{"start":"2026-06-01","end":"2026-07-01"},"dimensions":["segment"]}'

The answer is the same JSON as the single-node run - going across changes how the combine was computed, not its shape. To see that it actually crossed, watch the coordinator's own metrics while the request is in flight, in a third terminal:

curl -sk https://127.0.0.1:8443/metrics | grep sutura_worker_slots_wanted

sutura_worker_slots_wanted is a gauge the coordinator only moves while a combine is currently running across nodes; each worker's own /metrics carries sutura_worker_requests_in_flight, which moves only while that worker is answering a call. Because this example's combine finishes in well under a second, catching either gauge away from zero may take a couple of retries - there is no log line or answer field for this, by design (see Serving's limits).

Limits

  • The certificates are a demo CA generated fresh by distributed-certs, good for one day, with no use beyond this example. All three node certificates carry the same subjectAltName:IP:127.0.0.1, so on this loopback setup any one of them would verify for any of the three addresses - only the port tells the nodes apart here. A real deployment's nodes have distinct hostnames, each named in its own certificate.
  • across_above_bytes: 1 is set far below any realistic ceiling so this example's small legs cross at all; left unset, sutura's own working-set ceiling would keep a combine this size on one node.
  • A worker runs a task only for a call carrying the grant of the question it belongs to (Decision 5 of the ADR), so a node sent no stage of a question reads none of its output. Who may install a stage is still node-certificate identity alone: any certificate this demo CA signed can install a stage of its own on either worker, and one that learns a question's id before that question reaches a worker can have it refused there - a refusal, not a read.
  • One lost node costs the whole question (503 node_lost); nothing resumes a stage or re-plans onto the nodes that remain.
  • This two-worker setup has not been run end to end by anyone or anything - not CI, not by hand. The steps above are untested instructions, not a verified procedure.