Production & Maintenance

A production deployment of Telmoni separates the database roles, keeps audit partitions ahead of the calendar, sits behind a TLS reverse proxy, and can move sign-in to an OpenID Connect (OIDC) identity provider.

Identity & Authentication: Production OIDC HardeningCopy Anchor Link

When first launching Telmoni, you can seed an initial account with ADMIN_EMAIL and ADMIN_PASSWORD. It signs in through the login form (an email address and a password that Telmoni holds), which is on unless AUTH_DISABLE_USERNAME_PASSWORD=true.

In a production deployment, we strongly recommend connecting an OpenID Connect (OIDC) identity provider, one that publishes its discovery document at {issuer}/.well-known/openid-configuration and signs ID tokens with RS256 or the algorithm AUTH_CUSTOM_ID_TOKEN_SIGNED_RESPONSE_ALG names, and disabling the login form entirely.

Why use OIDC in production?Copy Anchor Link

1. Centralized lifecycle: When an employee leaves your company, deprovisioning them in your IdP stops their next sign-in to Telmoni. A session already open is Telmoni's own and lasts until it expires or is signed out, so remove them from your organizations as well. Telmoni reads no group or role claim, so their roles here change only in Telmoni.
2. Your provider's MFA: Telmoni has no second factor of its own and reads no acr or amr claim; a sign-in through the provider carries whatever factors the provider requires.
3. No password lanes: With AUTH_DISABLE_USERNAME_PASSWORD=true the server mounts none of the password lanes: sign-in, sign-up, verification, forgot and reset.

Step-by-step transition to OIDCCopy Anchor Link

# Identity Provider Discovery URL (must serve /.well-known/openid-configuration)
AUTH_CUSTOM_ISSUER=https://idp.example.com/realms/production
AUTH_CUSTOM_CLIENT_ID=telmoni-client-id
AUTH_CUSTOM_CLIENT_SECRET=telmoni-client-secret
AUTH_REDIRECT_URI=https://telmoni.example.com/auth/callback
AUTH_CUSTOM_NAME="Corporate SSO"

AUTH_PROVIDER_ORIGINS=https://idp.example.com

AUTH_DISABLE_USERNAME_PASSWORD=true
AUTH_DISABLE_SIGNUP=true

Closing new organizationsCopy Anchor Link

The signup flag is a row in the database, not an environment variable; the newest row for a key wins. The auth role may only read auth.feature_flags, so write the row as the table's owner — the migrator role, the login MIGRATOR_DATABASE_URL names — or as the Compose stack's single database user:

INSERT INTO auth.feature_flags (key, enabled, note, actor)
VALUES ('signup', false, 'invitation only', 'ops@example.com');

A new person who signs in then lands on their account alone, with their invitations. The console's switcher stops offering New organization too, to existing accounts as well, so nobody creates one. Insert a row with true to open sign-ups again.

The other feature flagsCopy Anchor Link

signup is one of five keys, all read the same way: the newest row per key wins, and a key with no row is on. A row in auth.organization_flags (the same columns, plus the organization's org_... id) overrides the global row for that organization, except for signup and connectors, which are read from the global rows alone.

INSERT INTO auth.organization_flags (organization_id, key, enabled, note, actor)
VALUES ('org_019a0b8c3f2e7d41', 'api_tokens', false, 'pending review', 'ops@example.com');

Database role hardeningCopy Anchor Link

In a production cluster, each module connects as its own role, granted its own schema, and none connects as a PostgreSQL superuser.

Role separationCopy Anchor Link

The database tier is split into five distinct roles:

┌─────────────────┬────────────────────────────────────────────────────────┐
│ Role            │ Granted Capabilities                                   │
├─────────────────┼────────────────────────────────────────────────────────┤
│ migrator        │ Owns every schema and table. Runs DDL.                 │
│ auth            │ CRUD on accounts, orgs, projects, and RBAC tables.     │
│ notifications   │ CRUD on delivery queues, feeds, and connector records. │
│ agent           │ CRUD on chunks, conversations, messages and cursors.   │
│ telemetry       │ Reads and writes each project's settings; no deletes.  │
└─────────────────┴────────────────────────────────────────────────────────┘

The audit chain is the one surface shared across roles: auth, notifications and telemetry may read and append audit.events, each to record its own changes (telemetry's are a project's content mode), and neither they nor the maintenance lanes may update or delete a row; agent has no audit grant at all. The spans are not in Postgres: ClickHouse keeps them, under users of its own (see Kubernetes). The owner role, migrator, keeps every privilege. Each module role is also kept from the writes the owner or a maintenance lane makes: auth may not write feature_flags or organization_flags, notifications may not write delivery_attempts, agent may not write erasures, and telemetry may not delete from project_settings.

Applying role hardeningCopy Anchor Link

To apply the production hardening policies, run the superuser setup script provided in the repository:

CREATE ROLE migrator WITH LOGIN PASSWORD '<migrator password>';
CREATE ROLE auth WITH LOGIN PASSWORD '<auth password>';
CREATE ROLE notifications WITH LOGIN PASSWORD '<notifications password>';
CREATE ROLE agent WITH LOGIN PASSWORD '<agent password>';
CREATE ROLE telemetry WITH LOGIN PASSWORD '<telemetry password>';
CREATE EXTENSION IF NOT EXISTS vector;

psql "$SUPERUSER_DATABASE_URL" -v ON_ERROR_STOP=1 -f crates/migrator/sql/role_hardening.sql

MIGRATOR_DATABASE_URL="postgres://migrator:<migrator password>@postgres:5432/telmoni" \
MIGRATOR_CLICKHOUSE_URL="http://migrator:<ClickHouse migrator password>@clickhouse:8123" \
  telmoni migrate

Partition rotationCopy Anchor Link

Telmoni's audit log uses range-partitioned tables partitioned monthly by timestamp (created_at). An audit row for a month with no partition fails its insert, and the change it records fails with it, so telmoni migrate creates the current month and the next three, and the rotation command keeps that runway ahead of the calendar. The same command drops ClickHouse's spans past retention:

telmoni rotate

Rotation behaviorCopy Anchor Link

• Forward Runway: telmoni rotate creates the current calendar month and the next three months of partitions ahead of time ({table}_YYYY_MM). It never drops audit partitions.
• Spans: It drops each day of spans in ClickHouse once every span in it arrived more than 360 days ago, and each month of hourly totals once it is more than 13 months behind. A night it misses is caught up by the next.
• Deleted projects: It rewrites away the rows a deleted project left. Deleting a project, or an organization with its projects, marks every span and every hour of totals of the project deleted the moment the purge runs, so that within moments, as the store applies the mark, no read answers them; the rotation applies that mark to the disk, so the rows leave it too. A rewrite that outlasts the statement's minute is left running, since a ClickHouse mutation finishes on its own, and noted in the log.
• Idempotency: Safe to run any number of times: a run creates only the audit partitions that are missing, drops only the days and months past retention and rewrites only what is marked, so a second run the same night changes nothing.

SchedulingCopy Anchor Link

The Helm chart runs it daily as the rotate CronJob (migrator.rotateSchedule, default "0 2 * * *"). On Compose, schedule it yourself, once a day, so spans go when their retention ends:

# /etc/cron.d/telmoni-rotate
0 3 * * * root docker compose -f /opt/telmoni/deploy/compose/docker-compose.yml run --rm migrate rotate

Operator CLI sweeps & disaster recoveryCopy Anchor Link

The telmoni binary includes administrative commands for cluster operators. No standalone binary is published: run them through the server image, whose entrypoint it is, as docker compose -f deploy/compose/docker-compose.yml run --rm server sweep deletion on Compose, or kubectl exec -n telmoni deploy/server -- /app/telmoni sweep deletion on Kubernetes.

Background maintenance sweepsCopy Anchor Link

Inside telmoni serve, the deletion sweep runs every ten minutes, the audit-export sweep every five, and the retention and audit-verification sweeps every 24 hours, each first a minute after boot. The deletion and audit-verification sweeps take a leader lock for each tick, so replicas on the same timers do not run one tick twice; the retention and audit-export sweeps take none, since their deletes are safe to repeat and each export they build is claimed first. Every step is idempotent, so a tick cut short by a rollout is finished by the next. RUN_SWEEPS=false leaves them to runs you schedule; schedule the audit-export sweep every few minutes, since it is the only retry for an export a restart cut short — without it one shows Preparing for good, and three such hold its owner off starting another — and what deletes a file past its 7 days. Telemetry's own sweep runs inside telmoni serve on every replica whatever RUN_SWEEPS says: every ten minutes, it finishes a deleted project's telemetry purge that its deletion could not. So does notifications' retention sweep, every hour and first a minute after boot: it deletes notices past 90 days, finished deliveries past 30 and expired connector handshakes, finishes a deleted project's connectors purge that its deletion could not, and asks Slack again, for a day, to uninstall an app a disconnect or a purge left unused that Slack did not answer. You can also trigger individual maintenance jobs on demand:

# Finish pending deletions: erase each organization past its wait (15 minutes
# after its deletion), then each person whose deletion is pending
telmoni sweep deletion

# Verify cryptographic hash chain across the audit ledger
telmoni sweep audit-verify

# Delete API keys revoked or expired over 7 days ago, invitations 30 days past
# expiry or withdrawal, sessions revoked over 90 days ago, and the sign-in
# tokens, codes and device authorizations nobody can use any more
telmoni sweep retention

# Re-embed every agent passage that a previous EMBEDDINGS_MODEL embedded
telmoni sweep agent-reindex

# Delete every audit log export past its 7 days, then finish building the
# exports a restart left unfinished
telmoni sweep audit-exports

Emergency organization managementCopy Anchor Link

If an organization needs to be closed by a platform administrator:

# Close an organization at once; the deletion sweep erases it after the same
# 15-minute wait as an owner's deletion
telmoni terminate org_019a0b8c3f2e7d41

terminate closes the organization without its owner's confirmation code, as service:operator in its audit log. There is no way back: the next deletion sweep past the wait erases it, as it erases one its owner deleted.

What the server refuses at bootCopy Anchor Link

telmoni serve reads its environment once and stops, with the reason in its log, rather than run with a setting that would fail later or weaken the deployment:

• AUTH_DISABLE_USERNAME_PASSWORD=true or AUTH_DOMAINS_WITH_SSO_ENFORCEMENT with no provider configured (no AUTH_CUSTOM_CLIENT_ID): nobody, or nobody at those domains, could sign in.
• ADMIN_EMAIL or ADMIN_PASSWORD with AUTH_DISABLE_USERNAME_PASSWORD=true, or an ADMIN_EMAIL at a domain AUTH_DOMAINS_WITH_SSO_ENFORCEMENT names: the seeded account could never sign in. The two are also refused one without the other, and at every boot an ADMIN_EMAIL that is not an address or an ADMIN_PASSWORD under 8 characters or over 256 bytes, even once the account exists.
• AUTH_CUSTOM_CLIENT_ID without AUTH_CUSTOM_ISSUER or AUTH_CUSTOM_CLIENT_SECRET, or either of those without it; an AUTH_CUSTOM_SCOPE without openid; an AUTH_CUSTOM_CHECKS without state, or none; and an AUTH_CUSTOM_ID_TOKEN_SIGNED_RESPONSE_ALG of ES512 or any HS*.
• A TELMONI_DEFAULT_ORG_ID that names no active organization, or a TELMONI_DEFAULT_PROJECT_ID that names no active project of it; either role as owner, Langfuse's VIEWER or NONE, or anything but member or admin; and a role or the project set without its id.
• SLACK_CLIENT_ID without SLACK_SIGNING_SECRET; either of SLACK_CLIENT_ID and SLACK_CLIENT_SECRET without the other; Slack credentials without CONNECTOR_KEK; and a CONNECTOR_KEK that is neither local: with 64 hex characters nor a Cloud KMS key name (a key version is refused too).
• TELEMETRY_KEK unset, or neither local: with 64 hex characters nor a Cloud KMS key name, as for CONNECTOR_KEK; REDIS_URL unset or not a Redis URL; and, with TELMONI_API_KEY set, a TELMONI_ENDPOINT that is not a URL.
• An SMTP_URL that is not a connection URL, or, with SMTP_URL set, a MAIL_FROM that is not an address.
• AGENT_MODEL_PROVIDER without AGENT_DATABASE_URL, or with a value other than anthropic and openai; openai without AGENT_MODEL; RERANK_URL without RERANK_MODEL; a hosted AGENT_MODEL_URL or EMBEDDINGS_URL without its key; AGENT_MODEL_TIMEOUT_SECS outside 1 to 100; an embeddings endpoint that answers with a width other than 768 (one not answering yet is allowed, and checked at its first answer).
• A set value that does not parse (PORT=eight), or a count below its minimum: the NOTIFICATIONS_DELIVERY_* settings, AGENT_MESSAGES_PER_HOUR and AGENT_MAX_TOKENS must be at least 1, except NOTIFICATIONS_DELIVERY_BACKOFF_SECS, which may be 0. A blank value reads as unset and takes the default.
• A TELEMETRY_CLICKHOUSE_URL that is not an http or https URL with a host, or that carries a query string or a fragment: settings are the server's to choose, never the URL's. The reason names the fault, never the URL, which holds a password. ClickHouse itself being down is no refusal: the server starts, and reaches it at the first read or write of a span.
• In a Kubernetes pod only: an AUTH_CUSTOM_ISSUER that is not https://, an SMTP_URL that is neither TLS nor loopback, a local: CONNECTOR_KEK or TELEMETRY_KEK, ALLOW_TEST_SESSION, a missing DATABASE_CA_CERT, and a ClickHouse URL over plain http to anything but loopback or a Service's name with its namespace (clickhouse.<namespace>.svc, ….svc.cluster.local), which telmoni migrate and telmoni rotate refuse too.

Reverse proxy configurationCopy Anchor Link

In production, place Telmoni behind a reverse proxy that terminates TLS and forwards the console's host to the web console on port 3000, and the machine lanes — ingest at /otlp/v1/traces, where the SDKs send; the read API under /v1; the CLI's sign-in under /cli/v1; and Slack's signed events at /webhooks/v1/slack — to the server on port 8082: on a host of their own (api.telmoni.example.com, which is then the SDKs' and the CLI's endpoint) or under their paths on the console's host (then the endpoint). Forward only those paths to 8082: on that port every other lane but the health probes (/health, /livez) requires the service secret and is the console's to call, and a person's every request enters through the console. Never expose either port directly to the public internet. The Compose file publishes 3000 on every interface of the host, so firewall it, or publish it on 127.0.0.1 alone with a Compose override; it publishes 8082 on 127.0.0.1 already, for the proxy on the same host. The server reads TRUSTED_PROXY_HOPS as the console does, so the machine lanes' per-address limits find the caller's address past the same proxies.

What the console expects of the proxy. The console's rate limiter takes each caller's address from X-Forwarded-For, counting TRUSTED_PROXY_HOPS entries back from the end: one for each entry the proxies in front append, so that entry is the first one no caller wrote. The Compose file sets TRUSTED_PROXY_HOPS=1 for the single reverse proxy below, which may set X-Forwarded-For to the client's address, as both configurations do, or append it to whatever the client sent; either way the last entry is the client's. Count every entry an outer hop appends as well: a plain load balancer in front of the proxy adds one (2), Google's load balancer adds two — the client's address and then its own, which is why the Helm chart's default is 2 with nothing else in front, and 3 with an appending proxy behind it. 0 is for nothing in front at all, and leaves the per-address limits to the caller's own header. X-Real-IP is not read.

The console's live updates (server-sent events) and the agent's streamed answers are marked X-Accel-Buffering: no, so Nginx passes them through unbuffered. The console uses no WebSockets.

The server speaks HTTP/1.1 alone, and gives each connection 620 seconds from its accept or its last answer to send a request's head, then closes it, so a client that never finishes one holds nothing for long. A proxy or load balancer that keeps idle connections to it must close them sooner, as Caddy, Nginx and Google's load balancer do by default, or a request it sends on one the server is closing fails.

# /etc/caddy/Caddyfile
telmoni.example.com {
    encode zstd gzip

    # The machine lanes, by path: ingest, where the SDKs send, the read
    # API, the CLI's sign-in and Slack's events. On a host of their own
    # instead, give that host a site block holding this handle alone, and
    # set it as the SDKs' and the CLI's endpoint.
    @machines path /otlp/* /v1/* /cli/* /webhooks/v1/slack
    handle @machines {
        reverse_proxy localhost:8082 {
            header_up X-Forwarded-For {remote_host}
        }
    }

    # Next.js Web Console
    handle {
        reverse_proxy localhost:3000 {
            header_up Host {host}
            header_up X-Real-IP {remote_host}
            header_up X-Forwarded-For {remote_host}
            header_up X-Forwarded-Proto {scheme}
        }
    }
}

Backups & disaster recoveryCopy Anchor Link

PostgreSQL backupsCopy Anchor Link

Telmoni keeps its persistent state in two stores: everything but the spans in PostgreSQL, and the spans with their hourly totals in ClickHouse (below). Create periodic logical backups of PostgreSQL with pg_dump. Every tenant table forces row-level security, so run it as a role that bypasses it: a superuser, such as the Compose stack's single database user, or migrator with separate roles.

# Logical database dump, from the Compose host (Postgres listens on 127.0.0.1)
pg_dump -h 127.0.0.1 -U telmoni -d telmoni -Fc -f "/backups/telmoni_$(date +%Y%m%d_%H%M%S).dump"

To restore from a backup file:

pg_restore -h 127.0.0.1 -U telmoni -d telmoni --clean --if-exists "/backups/telmoni_backup.dump"

ClickHouse backupsCopy Anchor Link

ClickHouse holds the telemetry database: every span, the hourly totals, and the digest of the statements its tables were made from. Losing it loses that history but not the server: telmoni migrate makes an empty store again. Back it up with ClickHouse's own BACKUP, as a user allowed to write backups, such as the Compose stack's single ClickHouse user. The image allows backups under backups/ in its data directory, so on the Compose host the archive lands in ClickHouse's volume, from where you copy it off:

# The container's own CLICKHOUSE_USER and CLICKHOUSE_PASSWORD sign the client in
docker compose -f deploy/compose/docker-compose.yml exec clickhouse \
  clickhouse-client --query "BACKUP DATABASE telemetry TO File('telemetry_20270101.zip')"
docker compose -f deploy/compose/docker-compose.yml cp \
  clickhouse:/var/lib/clickhouse/backups/telemetry_20270101.zip /backups/

To restore, copy the archive back into the same directory, drop what is there, and restore it. The row policies are not part of the database: they stay on the server, and telmoni migrate makes them again on a new one, finding its own digest in the restored store:

docker compose -f deploy/compose/docker-compose.yml cp \
  /backups/telemetry_20270101.zip clickhouse:/var/lib/clickhouse/backups/
docker compose -f deploy/compose/docker-compose.yml exec clickhouse \
  clickhouse-client --query "DROP DATABASE IF EXISTS telemetry SYNC"
docker compose -f deploy/compose/docker-compose.yml exec clickhouse \
  clickhouse-client --query "RESTORE DATABASE telemetry FROM File('telemetry_20270101.zip')"