Data Drains

Data Drains let organization owners and admins on Enterprise plans continuously export Agent Serve data to a destination they control — a customer-owned S3 bucket, Google Cloud Storage bucket, Azure Blob container, Microsoft Sentinel workspace, BigQuery table, Snowflake table, Datadog logs intake, or an HTTPS webhook. A drain runs on a schedule, picks up only new rows since its last successful run, and writes them to the destination. Viewing drain configuration and run history is restricted to owners and admins as well, since destinations expose internal bucket names, table identifiers, and webhook URLs.

Drains are independent of Data Retention but designed to compose with it — see Pairing with Data Retention below.

Setup

Go to Settings → Organization → Data drains in your workspace, then click New drain.

Each drain has four pieces:

1. A source — the category of data to export
2. A destination — where the data goes
3. A schedule — how often it runs
4. A name — unique within your organization

Sources

A drain exports exactly one source. To export multiple sources, create multiple drains.

Each row is delivered as a single line of NDJSON. The shape of each row is part of the public schema and stable across versions; every row carries an id field that downstream consumers can use to dedupe.

Drains select new rows by their creation cursor and deliver them at least once. A failed run can deliver a row again, so consumers should deduplicate by its id. Mutable fields on Chats (messages, title, lastSeenAt) are a point-in-time snapshot and won't be re-emitted if the chat is later updated. Treat the export as append-only and reconstitute current state from your own system of record if you need it.

Destinations

Amazon S3 (or any S3-compatible store)

Writes one NDJSON object per delivered chunk to your bucket.

• Bucket — the bucket name. Must already exist; Agent Serve does not create buckets.
• Region — AWS region (e.g. us-east-1).
• Prefix (optional) — folder path inside the bucket. Trailing slash optional.
• Access key ID / Secret access key — IAM credentials with s3:PutObject on the bucket. The "Test connection" button performs a real write probe to verify, then deletes it.
• Endpoint (optional) — for non-AWS stores like MinIO, Cloudflare R2, or GCS S3-interop. Leave blank for AWS S3.
• Force path-style (optional) — required for MinIO/Ceph, must be off for AWS S3 and R2.

Object keys are deterministic:

{prefix}/{source}/{drainId}/{yyyy}/{mm}/{dd}/{runId}-{seq}.ndjson

Objects are written with AES256 server-side encryption.

Google Cloud Storage

Writes one NDJSON object per delivered chunk to your GCS bucket.

• Bucket — the bucket name. Must already exist; Agent Serve does not create buckets.
• Prefix (optional) — folder path inside the bucket. Trailing slash optional.
• Service account JSON key — paste the full JSON key for a service account with storage.objects.create (and storage.objects.delete if you want "Test connection" to clean up its probe). Agent Serve authenticates via OAuth2 service-account JWT and uploads through the GCS JSON API.

Object names follow the same {prefix}/{source}/{drainId}/{yyyy}/{mm}/{dd}/{runId}-{seq}.ndjson layout as S3. Object metadata mirrors the S3 destination's agent-serve-* keys via x-goog-meta-* headers.

Azure Blob Storage

Writes one NDJSON block blob per delivered chunk to your container.

• Account name — your storage account (3–24 lowercase chars).
• Container — must already exist; Agent Serve does not create containers.
• Prefix (optional) — folder path inside the container.
• Account key — a storage account access key with write access to the container.

Blob names follow the same {prefix}/{source}/{drainId}/{yyyy}/{mm}/{dd}/{runId}-{seq}.ndjson layout. The same metadata is exposed as Azure blob metadata, collapsed to lowercase identifiers (agentservedrainid, agentserverunid, …) per Azure's rules.

For sovereign clouds, set Endpoint suffix to blob.core.usgovcloudapi.net (US Gov), blob.core.chinacloudapi.cn (China), or blob.core.cloudapi.de (Germany).

Microsoft Sentinel

Sends rows through the Azure Monitor Logs Ingestion API into a custom table in the Log Analytics workspace behind Microsoft Sentinel. It supports Azure public cloud with a Microsoft Entra app and client secret. It does not use the legacy shared-key Data Collector API.

Prepare Azure (Agent Serve does not create these resources):

1. Create a custom table, for example AgentServeLogs_CL, in the workspace Sentinel uses.
2. Create a data collection rule (DCR) with an input stream such as Custom-AgentServeLogs that declares the columns below, and route it to that table.
3. Copy the DCR's immutable ID (dcr-…) and its logs ingestion endpoint.
4. Register a Microsoft Entra app, create a client secret, and give the app the Monitoring Metrics Publisher role on the DCR. Role assignments can take up to 30 minutes to apply.

{
  "Custom-AgentServeLogs": {
    "columns": [
      { "name": "TimeGenerated", "type": "datetime" },
      { "name": "Source", "type": "string" },
      { "name": "RecordId", "type": "string" },
      { "name": "DrainId", "type": "string" },
      { "name": "RunId", "type": "string" },
      { "name": "Data", "type": "dynamic" }
    ]
  }
}

Configure the drain with the logs ingestion endpoint (origin only, for example https://my-dcr.eastus-1.ingest.monitor.azure.com), the data collection rule ID, the stream name, the Microsoft Entra tenant ID, the application (client) ID, and the client secret. The endpoint must be HTTPS on *.ingest.monitor.azure.com. Agent Serve validates DNS, pins the resolved address, and refuses redirects. The client secret is stored encrypted like other drain credentials; access tokens stay in memory for the run.

Test connection writes one probe event with Source: "connection_test". A passing test means Azure accepted the request; rows can take several minutes to become queryable.

Each row is wrapped in an envelope. TimeGenerated is the drain run start, so the original event time stays in Data.createdAt, Data.startedAt, Data.endedAt, or Data.completedAt. The full source row is kept under Data.

Limits. Requests are split at 1,000,000 UTF-8 bytes. A row whose serialized Data exceeds 64,000 bytes fails the run before anything in that chunk is sent; rows are never truncated, so large Chat transcripts or workflow outputs may need an object-storage drain. Network errors, 408, 429, and 5xx are retried up to three times, honoring Retry-After up to 60 seconds. 400 and 403 fail immediately. A 401 refreshes the token once. Error messages never include provider response bodies or tokens.

Delivery is at least once: a failed run is retried from the last committed cursor, so batches Azure already accepted can be sent again. Deduplicate on Source and RecordId.

Google BigQuery

Streams each row into a target table via the tabledata.insertAll API, with per-row insertId dedup.

• Project ID — your GCP project (supports domain-scoped IDs like example.com:my-project).
• Dataset ID / Table ID — must already exist; Agent Serve does not create tables. The table schema must match the source row shape, with a compatible column for each top-level field. Agent Serve sends those fields directly and sets ignoreUnknownValues to false; it does not wrap the row in a single JSON or STRING column.
• Service account JSON key — needs roles/bigquery.dataEditor (insert) and roles/bigquery.metadataViewer (for the tables.get probe used by "Test connection"). Agent Serve authenticates via OAuth2 service-account JWT.

Each row is sent with an insertId of {drainId}-{runId}-{sequence}-{index}. BigQuery dedupes inserts with the same insertId for ~60 seconds, so retries inside that window won't duplicate. If a chunk reports partial failure (insertErrors), the run fails with the offending row indices and an outer-driver retry may duplicate rows that already succeeded — the dispatcher's bounded retry minimizes this risk. Enforced per-request limits: 10 MB body, 50,000 rows.

Snowflake

Inserts each row into a target VARIANT column via the Snowflake SQL API v2 with key-pair JWT auth.

• Account — the Snowflake account identifier. Preferred form is <orgname>-<acctname> (no dots). Legacy <locator>.<region>.<cloud> is also accepted.
• User / Warehouse / Database / Schema / Table — must already exist. The user needs INSERT privilege on the table and USAGE on the warehouse, database, and schema.
• Column (optional) — target VARIANT column name. Defaults to DATA (matches Snowflake's unquoted identifier folding).
• Role (optional) — Snowflake role to assume.
• Private key (PEM) — PKCS8-encoded RSA private key. Register the matching public key on the Snowflake user via ALTER USER ... SET RSA_PUBLIC_KEY = '...'.

Each chunk becomes a single INSERT INTO "DB"."SCHEMA"."TABLE" ("col") VALUES (PARSE_JSON(?)), ... with one TEXT binding per row. Identifiers are quoted to preserve case. The destination handles Snowflake's async 202-then-poll pattern transparently. Per-row JSON payloads are capped at 16 MB to match Snowflake's VARIANT limit.

Datadog Logs

POSTs each row as a log entry to Datadog's v2 logs intake.

• Site — your Datadog site: us1, us3, us5, eu1, ap1, ap2, or gov.
• Service (optional) — value for the reserved service field. Defaults to agentserve.
• Tags (optional) — comma-separated ddtags appended to every entry alongside auto-injected agent_serve_drain_id:, agent_serve_run_id:, and agent_serve_source: tags.
• API key — a Datadog API key (not an Application key) with logs-write permission.

Top-level row fields are auto-indexed as Datadog log attributes. The reserved fields ddsource, service, ddtags, and message are always set by Agent Serve and override anything in the row. Payloads above 1 KB are gzip-compressed. Enforced limits match Datadog's intake: 5 MB per request (post-compression), 1000 entries per request, 1 MB per entry.

HTTPS Webhook

POSTs each chunk as NDJSON to your endpoint.

• URL — must be HTTPS. Agent Serve resolves the hostname and refuses to deliver to private, loopback, or cloud-metadata IPs. The resolved IP is pinned for the duration of a run to prevent DNS rebinding.
• Signing secret — shared secret used for HMAC-SHA256 signing.
• Bearer token (optional) — sent as Authorization: Bearer <token>.
• Signature header name (optional) — defaults to X-Agent-Serve-Signature.

Each request includes:

Content-Type: application/x-ndjson
User-Agent: AgentServe-DataDrain/1.0
X-Agent-Serve-Timestamp: <unix-seconds>
X-Agent-Serve-Signature-Version: v1
X-Agent-Serve-Signature: t=<unix-seconds>,v1=<hex(hmac-sha256)>
X-Agent-Serve-Drain-Id: <drain id>
X-Agent-Serve-Run-Id: <run id>
X-Agent-Serve-Source: <source name>
X-Agent-Serve-Sequence: <chunk index>
X-Agent-Serve-Row-Count: <rows in this chunk>
Idempotency-Key: <runId>-<sequence>

The signature is computed as HMAC-SHA256(secret, "${timestamp}.${body}") and serialized as t=<timestamp>,v1=<hex>. Verify by recomputing over the same string and rejecting timestamps older than ~5 minutes — this defends against captured-request replay attacks.

Failed deliveries retry up to 3 times with exponential backoff (500ms, 1s, 2s with ±20% jitter), respecting Retry-After on 429/503. Non-retryable 4xx responses fail the run immediately.

Schedule

You can also disable a drain with the Enabled toggle (it stops running but is preserved), or trigger an out-of-schedule run with Run now on any drain row.

Delivery semantics

Drains use an opaque cursor that advances only on full success. If a delivery fails partway through a run, the cursor is unchanged and the next run replays from the last successful position.

This is at-least-once delivery. Combined with the id field on every row and the Idempotency-Key header on every webhook chunk, downstream systems can dedupe deterministically.

The last 10 runs for each drain are visible by expanding its row in the settings page, with status, row count, bytes written, destination locator (s3://... or webhook URL), and the error message if it failed.

Security

• Destination credentials are encrypted at rest using the same key-rotation–aware encryption that protects OAuth tokens.
• Credentials are never returned by the Agent Serve API after creation. Updates accept new credentials; omitting them leaves the existing encrypted blob in place.
• Webhook URLs are SSRF-validated: HTTPS-only, no private/loopback/metadata IPs, with the resolved IP pinned to defeat DNS rebinding.
• Every create, update, delete, manual run, and test-connection call is recorded in the Audit Log.

Pairing with Data Retention

Drains and Data Retention are independent modules. Agent Serve does not gate retention on drain progress — if a drain is failing, retention will still purge data on its own schedule. Keep the two configurations independent and pair them deliberately.

To safely use both together, set the drain cadence shorter than the retention period for the same data category:

For example, with Log retention set to 30 days, set the workflow-logs drain to Hourly or Daily so every row is exported well before retention purges it from Agent Serve. Monitor recent drain runs in the settings page; if a drain has been failing for longer than your retention window, you may lose rows that retention purges before they are exported.

After data lands in your bucket or webhook system, archive lifecycle (transitions to Glacier, expiration, GDPR right-to-erasure propagation) is governed by your own infrastructure — Agent Serve has no further visibility into that data once delivery succeeds.

Self-hosted setup

Environment variables

DATA_DRAINS_ENABLED=true
NEXT_PUBLIC_DATA_DRAINS_ENABLED=true

On a self-hosted deployment the feature resolves on when DATA_DRAINS_ENABLED is true, or when it is unset and ENTERPRISE_ENABLED is true. An explicit DATA_DRAINS_ENABLED=false wins over the master switch. That resolved value — not the bare variable — gates the Settings → Organization → Data drains page, the server-side mutating endpoints, and the cron dispatcher. When it resolves off the page is hidden, drain create/update/delete/run requests return 404, and the dispatcher is a no-op.

Inside a workspace the browser reads that resolved value from the server, so the page follows the server whether or not NEXT_PUBLIC_DATA_DRAINS_ENABLED is set. Set it to the same value anyway: it backs the client fallback on surfaces rendered outside a workspace, and on a Compose install or source checkout npx @agentserve/setup doctor reports the pair disagreeing.

Scheduling the dispatcher

The dispatcher is an HTTP endpoint, not a self-scheduling job — something has to call it:

GET /api/cron/run-data-drains

It authenticates with a bearer token equal to CRON_SECRET and returns 401 when that variable is unset, so a self-hosted deployment must set it:

openssl rand -hex 32

Set the same value as CRON_SECRET on both the app and whatever invokes the endpoint. Generating it in your shell does not configure either one.

Both shipped deployments schedule this endpoint hourly for you — Helm through cronjobs.jobs.runDataDrains, Docker Compose through its cron service. A deployment that runs neither schedules it itself:

curl -H "Authorization: Bearer $CRON_SECRET" \
  https://agentserve.example.com/api/cron/run-data-drains

Each due drain is then fanned out as a run-data-drain background job, so the deployment also needs TRIGGER_DEV_ENABLED and a configured Trigger.dev project. Without it the dispatcher still claims the work and enqueues it to the database, but nothing drains that queue for this job type, so the runs stay pending and never execute. See background jobs.