Operations

Event retention

ingest_svc runs a background retention pass that deletes events older than the configured window. There are two paths, chosen automatically by whether the events table is partitioned:

• Partitioned table (the intended form — monthly events_YYYYMM partitions):

the pass drops whole partitions once all their rows are older than the window — an instant DDL DROP TABLE, no vacuum, no lock contention with ingest.

• Non-partitioned table (a deployment whose events table predates

partitioning, i.e. it was created before the partitioning DDL and CREATE TABLE IF NOT EXISTS therefore never converted it): the pass falls back to a batched DELETE of old rows (10k per iteration). Retention still runs, but DELETE is heavier than a partition drop and leaves dead tuples for autovacuum to reclaim. On every pass the log carries a WARNING that the table isn't partitioned.

Check which path you're on:

SELECT relkind FROM pg_class WHERE relname = 'events';
-- 'p' = partitioned (partition-drop path) | 'r' = plain table (DELETE fallback)

To move an existing plain table onto the efficient partition-drop path, run the opt-in offline migration (BACKUP + stop writers first; dry-run by default):

DATABASE_URL=postgres://... python scripts/migrate_events_to_partitioned.py        # dry run
DATABASE_URL=postgres://... python scripts/migrate_events_to_partitioned.py --yes  # apply

Configuration

# .env
EVENT_RETENTION_DAYS=90   # default; set to 0 to effectively disable pruning (nothing is ever old enough)

Restart ingest to pick up a change:

docker compose up -d --force-recreate ingest

Schedule

The retention pass runs once immediately at startup (covers a service that was down long enough for pruning to matter right away), then every 24 hours, as a background task inside ingest_svc's process lifetime — not a separate cron job or service. It shares the connection pool with the rest of ingest_svc and is deliberately co-located with partition creation (_ensure_event_partitions(), same file) rather than split into a separate maintenance service — both operate on the same table, the same pool, the same schema-ownership boundary.

A failed pass (e.g. a transient DB error) is logged and retried on the next scheduled tick — a single bad tick never kills the loop or the service.

Monitoring

Every pass that actually drops something logs at INFO:

Pruned event partition events_202501 (data before 2025-02-01, ~184023 rows)
Retention pass complete: 1 partition(s) dropped, ~184023 rows freed
Retention pass took 0.14s, dropped 1 partition(s)

Row counts are Postgres's own planner estimate (pg_class.reltuples, refreshed by autovacuum/ANALYZE) rather than an exact COUNT(*) — exact counts would mean a full scan of each partition just for a log line.

Startup staleness check: on every boot, ingest_svc checks whether any partition already exceeds EVENT_RETENTION_DAYS before the scheduled pass runs. If so, it logs a WARNING:

Retention check: a partition already exceeds EVENT_RETENTION_DAYS=90 at startup — either
this is the first startup after enabling retention on older data (harmless, the prune loop
below will catch up momentarily), or the retention loop has been silently failing across
restarts. Watch for 'Retention pass' log lines after startup to confirm it catches up.

There's no persisted "last successful prune" timestamp anywhere — in-memory state wouldn't survive a restart, which is exactly the failure mode this exists to catch. This check is a DB-state proxy instead: if a partition this old still exists, pruning hasn't kept up, whether that's because it's never run, is broken, or this is a legitimate one-time catch-up. Both cases look identical from here; if the "Retention pass" log line doesn't show up shortly after this warning, something is actually wrong.

Manual invocation

POST /admin/prune-events runs an out-of-band retention pass immediately, without waiting for the next scheduled tick — useful right after the startup staleness warning, or to reclaim space on demand. Admin-only, same pattern as POST /v1/keys:

curl -s -X POST "http://localhost:8001/admin/prune-events" \
  -H "Content-Type: application/json" \
  -d '{"admin_key": "<ADMIN_API_KEY>"}'

# Response:
{"partitions_dropped": 1, "signals_scrubbed": 412, "retention_days": 90}

Pass retention_days explicitly to override the configured default for this one invocation:

curl -s -X POST "http://localhost:8001/admin/prune-events" \
  -H "Content-Type: application/json" \
  -d '{"admin_key": "<ADMIN_API_KEY>", "retention_days": 30}'

ADMIN_API_KEY must be set in the environment — an unset or empty value rejects every request (closed by default), same as the key-creation endpoint.

Unlike the daily loop, a scrub failure here is not swallowed — the call returns a 500. The loop can afford to log and retry on the next tick; a manual invocation has no next tick, and reporting signals_scrubbed: 0 for a pass that actually errored is indistinguishable from "there was nothing to scrub".

What the SDK sends: redaction and content caps

Two controls decide what leaves your process at all. Both run in the SDK, in-path, before anything is buffered — which matters, because the ingest service never redacts. Whatever an SDK sends is what lands in Postgres, is rendered in the dashboard, is included in Slack alert payloads, and is sent to Anthropic, OpenAI or Mistral when someone clicks Explain. Server-side content caps exist only on the OpenTelemetry path (OTLP_MAX_ATTR_CHARS), not on the JSON path the SDKs use.

The Python and TypeScript SDKs implement the same rules and the same wire format, so a run reads identically whichever one produced it.

Redaction

Structured tool arguments are walked and any value under a secret-looking key is replaced with [REDACTED] before serialisation. Keys are matched on words: the name is lower-cased and -, ., whitespace and camelCase boundaries all become _, then it matches when the result equals a denylist entry or ends with _<entry>.

That word normalisation is load-bearing: matching on case and - alone caught access_token but missed accessToken, clientSecret, refreshToken and apiSecret — and camelCase is the dominant convention in the JSON tool arguments this actually sees.

Built-in entries are authorization, api_key, apikey, token, secret, password, cookie and set-cookie — the suffix rule is what lets eight entries cover every *_token and *_secret spelling without enumerating them.

What redaction does not cover. It is key-based, so a secret inside a value — a JSON blob passed as a string, a URL with a token in its query — is not detected. Plain-text fields such as LLM output, retrieval content and memory values are capped, not redacted, because there is no key to judge them by. The walk also stops at 32 levels of nesting.

Content caps

Every free-text field is cut at max_field_chars / maxFieldChars (default 8192, the same limit the OTLP path enforces): tool arguments and output, LLM output, retrieval query and content, memory values, input_text and system_prompt. A field that was cut carries <field>_truncated: true and <field>_original_length: <n> alongside it; an uncapped field carries neither, so a small payload is byte-identical to what it would have been.

Configuring it

dt = Dunetrace(
    max_field_chars=8192,                 # 0 disables the cap entirely
    redact_keys=["x_session_id"],         # extends the built-in denylist
    redact=lambda args: {...},            # runs BEFORE the denylist
)

const dt = new Dunetrace({
  maxFieldChars: 8192,
  redactKeys: ["x_session_id"],
  redact: (args) => ({ ... }),
});

redact_keys / redactKeys go through the same word normalisation, so "X-Session-Id", "xSessionId" and "x_session_id" are the same entry. The redact hook runs first and receives the structured arguments, so it can strip domain-specific fields the built-in list cannot know about; it runs on every tool call, so keep it cheap.

Neither control can raise: both are wrapped so a failure ships the original value rather than blocking your agent. A redaction bug therefore fails open, which is the right trade for instrumentation but is worth knowing.

Signal evidence scrub

failure_signals has no retention policy and deliberately keeps its rows: the dashboard compares the last 30 days of signals against a days-30-to-90 baseline to decide whether a fix worked (verified / likely_fixed / still_occurring), so deleting aged signals would permanently truncate the baseline arm of that comparison. The table is also not partitioned, so expiry would mean a batched DELETE with vacuum pressure rather than an instant partition drop, and five tables carry a bare signal_id with no foreign key — deleting would silently orphan them.

But a signal's evidence dict embeds excerpts of the same raw agent content the event retention pass exists to expire — in two cases untruncated:

So the rows stay and those keys are stripped, on the same EVENT_RETENTION_DAYS window as the event prune — one content horizon, so raw content leaves events and failure_signals.evidence at the same moment rather than on two schedules that can drift apart.

This costs no analytics. Every SQL consumer of evidence reads metadata (tool, count, consecutive_fails, args_identical, growth_factor, conversation_id); none reads a key in the table above. Detector-owned labels are deliberately kept — matched_marker and matched_patterns name which of Dunetrace's own constants matched, grounded_surfaces names surfaces, failure_source is a fixed literal, and every *_length field is an integer.

The scrub runs as a second pass in the same daily loop, kept separate from the prune so a partition-drop error can't skip it. It is idempotent — an already-scrubbed row is never rewritten, so repeat passes cost a scan and no writes — and batched so a large backlog doesn't hold one long transaction.

Evidence scrub complete: 412 signal(s) stripped of content keys (detected before 2026-05-21)
Evidence scrub took 0.31s, scrubbed 412 signal(s)

processed_runs retention

processed_runs is the detector's idempotency ledger — one row per run, recording that the run has been analysed. It is also the anti-join target in run discovery, so letting it grow without bound slows the detector's hottest query.

The detector worker prunes it daily. Only shard 0 prunes, since the table isn't shard-partitioned and extra replicas would only contend on the same rows. A pass keeps deleting while batches come back full, so a long-neglected table catches up in one pass rather than one batch per day.

Lowering PROCESSED_RUNS_RETENTION_DAYS below EVENT_RETENTION_DAYS isn't dangerous — the NOT EXISTS still refuses to delete rows whose events survive — it just wastes work re-examining rows that can't yet qualify. There is no manual trigger endpoint; the loop runs at worker startup and every 24h. To verify it, watch for Pruned N processed_runs row(s) in the detector logs.

Rate limiting

ingest_svc enforces a per-API-key sliding-window rate limit (60s window, rate_limit_rpm from the api_keys table) on POST /v1/ingest, POST /v1/deploy, and POST /v1/otlp/traces. Keys are org-scoped, not agent-scoped — one key can carry traffic for many agents. Without any further limiting, one runaway agent under a shared key can consume the entire key's budget and starve its siblings.

Per-agent sub-limits

Within a key's overall budget, each distinct agent_id gets its own sliding-window sub-limit — by default, 20% of the key's effective rpm. An agent hitting its own sub-limit gets a 429; other agents under the same key are unaffected. The key-level limit is still checked first and always applies regardless of agent — many agents each within their own sub-limit can still collectively exhaust the key.

Agent identity for sub-limiting comes from the request body's agent_id field for /v1/ingest and /v1/deploy (already parsed for auth), and from the X-Dunetrace-Agent-Id header for /v1/otlp/traces (cheap to read; the service.name resource-attribute fallback would require decoding the OTLP body, which the rate-limit middleware deliberately avoids — an OTLP trace relying on service.name alone only gets key-level limiting, not a per-agent sub-limit).

Important if a key genuinely has only one real agent: the 20% default still applies unless overridden — it does not detect "this key only has one agent" and skip sub-limiting. A single-agent-per-key deployment relying on the full key rpm for that one agent should set an explicit override (see below) closer to 1.0, or the effective throughput for that agent will be capped at 20% of what the key alone would otherwise allow.

Adjusting a per-agent quota

# View current quota (default or override) for an agent under a key
curl -s "http://localhost:8001/admin/keys/{key_id}/agents/{agent_id}/quota?admin_key=<ADMIN_API_KEY>"

# Response:
{"key_id": 42, "agent_id": "worker-1", "quota_pct": 0.20, "is_override": false}

# Set an override — e.g. give this agent 50% of the key's rpm
curl -s -X PUT "http://localhost:8001/admin/keys/{key_id}/agents/{agent_id}/quota" \
  -H "Content-Type: application/json" \
  -d '{"admin_key": "<ADMIN_API_KEY>", "quota_pct": 0.5}'

key_id is the numeric id from the key-creation response or GET /v1/keys (api_svc) — never the raw secret key string, which shouldn't appear in a URL path. Quota changes take effect within a few minutes (cached the same way rate_limit_rpm is, not read fresh on every request) — not instantly.

429 response headers

HTTP/1.1 429 Too Many Requests
Retry-After: 12
X-RateLimit-Key-Remaining: 0
X-RateLimit-Agent-Remaining: 0

X-RateLimit-Agent-Remaining is only present when an agent_id was resolvable for the request (see above) — its absence means only the key-level limit was checked, not that the agent has unlimited quota.