Queues and Notifications

Publish one metadata-only message per processed file to Amazon SQS, and connect Kafka or other brokers without a built-in client.

SQS notification sink

Any file-drop connector watch, whatever its backend (local, SFTP or an object store), can publish one message per processed file to an Amazon SQS queue. Add a sink block to the watch:

"sink": {
  "type": "sqs",
  "queueUrl": "https://sqs.eu-west-1.amazonaws.com/123456789012/transmute-events",
  "region": "eu-west-1",
  "credentials": { "webIdentity": true }
}

Field	Required	Meaning
`type`	Yes	`sqs`, the only supported sink type
`queueUrl`	Yes	The queue URL. A name ending in `.fifo` makes it a FIFO queue
`region`	Yes	The queue's region
`credentials`	Yes	Same two forms as the `s3` block: `accessKeyIdEnv` + `secretAccessKeyEnv` (+ optional `sessionTokenEnv`) naming environment variables, or `"webIdentity": true`. See credentials
`messageGroupId`	FIFO only	Required for a FIFO queue (1 to 128 characters); refused on a standard queue
`allowInsecure`	No	Permits a plain-HTTP or private-address endpoint, for local testing only. Same network rules as object stores

The role or key needs sqs:SendMessage on the queue.

Message body: metadata only

{
  "watch": "s3-statements",
  "file": "stmt-2026-06-01.fin",
  "status": "succeeded",
  "warningCodes": ["TRUNC_NARRATIVE"],
  "outputKey": "statements/out/stmt-2026-06-01.xml",
  "sidecarKey": "statements/out/stmt-2026-06-01.fin.warnings.json",
  "durationMs": 3
}

Field	Meaning
`watch`	The watch's name (or `from->to`)
`file`	The input file name
`status`	`succeeded` or `failed`
`errorCode`	On failure only: the problem slug, for example `invalid-input`
`warningCodes`	Deduplicated, sorted warning codes; `[]` when none
`outputKey`	The output's key or path (the first output when there are several)
`outputKeys`	All output keys, present only when one input produced several outputs
`sidecarKey`	The `.warnings.json` sidecar on success, or the `.reason.json` on failure
`durationMs`	Conversion time

That is the complete list. The message never carries file content, parsed field values or content hashes. Consumers that need the result read the output object or file themselves.

Delivery guarantees

The message is sent after the outputs and sidecar are written and before the input is moved out of in/. If SQS refuses it, the input stays in in/ and the whole file is retried on the next poll. Delivery is therefore at-least-once:

On a standard queue, deduplicate on file plus sidecarKey.
On a FIFO queue, the connector sets a deduplication id derived from the watch, file name, status and sidecar key, so SQS drops a repeat sent within its five-minute deduplication window.

A file can also be notified twice when an object-store commit is interrupted between copy and delete (see commit ordering).

Kafka and NATS

There is no built-in Kafka or NATS client. Only sqs is accepted as a sink type; any other value stops the connector at startup.

For Kafka, either:

run Kafka Connect's S3 source connector against the watch's out/ prefix, or
bridge the SQS queue into Kafka with an SQS source connector.

Example: S3 watch with a FIFO sink

{
  "watches": [
    {
      "name": "s3-statements",
      "backend": "s3",
      "in": "in", "out": "out", "archive": "archive", "error": "error",
      "from": "mt940", "to": "camt.053",
      "s3": {
        "region": "eu-west-1",
        "bucket": "acme-drop",
        "prefix": "statements",
        "credentials": { "webIdentity": true }
      },
      "sink": {
        "type": "sqs",
        "queueUrl": "https://sqs.eu-west-1.amazonaws.com/123456789012/transmute-events.fifo",
        "region": "eu-west-1",
        "messageGroupId": "statements",
        "credentials": { "webIdentity": true }
      }
    }
  ]
}

Was this page helpful?