Standalone Worker

Embedded versus standalone

Batch jobs are queued in Postgres and processed by a worker. By default the API process runs one itself (the embedded worker), processing one item at a time so batch work never competes hard with synchronous requests. The /worker binary in the same image is the standalone alternative: it runs the same claim loop as its own process, so you can give batch work dedicated CPU and scale it independently.

Workers claim jobs with Postgres row locks, so any number of standalone workers, with or without the embedded one, can run side by side without coordination and without Redis.

The standalone worker:

• claims queued items, converts them and stores the encrypted results for the job's TTL;
• requeues items whose claim went stale (a worker that died mid-item) and deletes expired jobs;
• pushes results to an accounting connection when the batch asked for delivery;
• writes one metadata-only usage row per item to Postgres.

What it does not do today:

• It sends no webhooks. job.completed and job.failed webhooks, and key.expiring notifications, are sent only by the API's embedded worker. If you turn the embedded worker off, these notifications stop. Keep the embedded worker on if you rely on webhooks, and poll GET /v1/jobs/{id} otherwise.
• It does not load mapping packs. Batch jobs use the releases built into the image.

Run it

Turn the embedded worker off on the API (TRANSMUTE_EMBEDDED_WORKER=false) and start one or more workers from the same image:

services:
  worker:
    image: ghcr.io/403ent/transmute:<release>
    entrypoint: ["/worker"]
    env_file: /opt/transmute/worker.env   # DATABASE_URL, TRANSMUTE_JOB_KEY; mode 600
    environment:
      TRANSMUTE_WORKER_CONCURRENCY: "4"

The worker's HTTP surface has two routes and reads nothing from requests: GET /healthz answers ok (use it as a liveness probe), and /wake makes the worker claim immediately instead of waiting for its one-minute fallback poll. On a normal host or pod the worker drains the queue without ever receiving /wake; a newly queued job may simply wait up to a minute.

Shutdown on SIGTERM finishes the items in progress, flushes usage records and exits.

Helm. worker.enabled: true deploys worker.replicas worker pods (with worker.resources) and sets TRANSMUTE_EMBEDDED_WORKER=false on the API. It requires externalDatabase.existingSecret and jobKey.existingSecret, and the render fails without them. The chart passes only DATABASE_URL, TRANSMUTE_JOB_KEY and LOG_LEVEL to worker pods; it has no extraEnv for the worker, so concurrency, provider credentials and the posting switch cannot be set there today. If your batches use delivery, leave worker.enabled off and let the API's embedded worker, which receives extraEnv, do the work.

Delivering to a connection

A batch can ask for each item's result to be pushed to an accounting connection once it is converted:

{
  "items": [
    { "id": "stmt-0601", "from": "mt940", "to": "ofx", "message": "..." }
  ],
  "delivery": { "connection": "conn_...", "deliverAs": "file" }
}

transactions needs the transactions:post scope, every item converting to: "accounting-transaction", a connection granted with posting, and a mapping that has passed a dry run. All of this is checked when you enqueue. A posting-only connection (QuickBooks Online, Sage) accepts only transactions. The full setup is in Accounting connections.

On a self-hosted install, posting also requires the transactions licence feature, TRANSMUTE_EXPERIMENTAL_ACCOUNTING set on the API and on any process that performs delivery, and posting enabled for the tenant:

/admin enable-accounting <tenant-uuid>

A delivery failure never fails the item: the result stays retrievable from GET /v1/jobs/{id} until the job expires, and the item carries a deliveryError code (connection-failed, provider-error, dry-run-required, post-quota-exceeded, connection-reconsent-required, post-incomplete or invalid-input). Posting records each transaction before sending it, so delivering the same item again never double-books.

Delivered file names

With deliverAs: file, each result is uploaded as <item id>.<ext>. The item id is sanitised to A-Z a-z 0-9 . _ - (other characters become -), and an item without an id uses its position in the batch. An item with several outputs uploads <item id>-1.<ext>, <item id>-2.<ext> and so on.

The extension follows the target format:

The file-drop connector names its out/ files with the same table.

From the file-drop connector

A file-drop watch can push its outputs to a connection too, in addition to writing them to out/:

"tenant": "11111111-1111-1111-1111-111111111111",
"deliver": { "connection": "conn_...", "deliverAs": "file" }

deliver takes the same connection, deliverAs and bankAccountRef keys, and deliverAs: "transactions" requires the watch to convert to: "accounting-transaction". The watch must have a tenant (connections belong to a tenant), and the connector process needs DATABASE_URL, TRANSMUTE_JOB_KEY and at least one provider's credentials; without them it refuses to start rather than silently skipping delivery. A failed push leaves the file lifecycle untouched (outputs in out/, input in archive/) and writes out/<output name>.delivery-error.json with the problem code.