Building a Low-Volume Batch Worker Without a Database: Using S3 as Task Storage

I recently needed to build a low volume internal workflow: a user uploads a CSV, a backend processes it asynchronously, and the user downloads the results.

The standard approach job processing stack would be a database for task metadata, a queue for dispatch, and worker deployments. But for an occasional internal tool, that’s often more infrastructure than the problem deserves.

Since the workflow was entirely file-oriented (large input and output CSVs), S3 was already the natural fit.

So the question became: could S3 also hold the task state and handle concurrent workers safely for a workflow that only runs occasionally?

🥁 … Yes, while everyone knows S3 is great for storing files, what makes this pattern interesting is how far you can push it. Using a tiny metadata file status.json as the task record, S3 conditional writes for safe concurrency, and an inverted ULID for newest-first listing, you can get a lightweight task queue for free.

Disclaimer: this isn’t “S3 as a replacement for queue”. It’s a highly targeted pattern for low-volume, file-oriented workflows where the main goal is keeping your infrastructure footprint minimal while keeping the workflow durable and recoverable.

When Does This Pattern Make Sense?

“Low volume” is subjective, so let’s put numbers on it. This design is a great fit when:

• You process roughly 1 to 100 tasks per day.
• Seconds-level latency for task pickup is acceptable.
• You only need to list the latest tasks (no complex querying, filtering or reporting).

If your workload requires high throughput, complex filtering, or strict ordering, you absolutely should reach for a database and queue.

But for small, file-oriented workflows, the standard infrastructure overhead (database schemas, queue configuration, worker deployments, and monitoring runbooks) often becomes larger than the feature itself.

How It Fits Together

Before the details, here is the whole system in one view. It is four moving parts on top of S3:

• Artifacts live in a per-task prefix - the input and output files.
• State lives in a tiny status.json next to them - the task record.
• Workers find work by listing a small set of marker objects, then claim tasks using S3 conditional writes (compare and set - CAS).
• Ordering comes for free from the task ID (inverted ULID), so the newest tasks list first.

sequenceDiagram
    actor C as Client
    participant API as Control API
    participant S3
    participant W as Worker

    Note over C,S3: Submit
    C->>API: POST /tasks
    API->>S3: write status.json (created)
    API-->>C: presigned upload URL
    C->>S3: upload input.csv
    C->>API: POST /tasks/{id}/submit
    API->>S3: status.json (queued) + active marker

    Note over W,S3: Process
    W->>S3: list active/
    W->>S3: claim via If-Match ETag
    W->>S3: write output + status.json (completed)

    Note over C,S3: Download
    C->>API: GET /tasks/{id}/output
    API-->>C: presigned download URL
    C->>S3: download output.csv

The rest of this post walks through each part: the layout, the state record, the worker, the ordering trick, the API, and where the pattern stops fitting.

S3 Layout

Each task gets its own prefix:

batch-tasks/
  <task-id>/
    input.csv
    status.json
    output-attempt-1.csv
  active/
    <task-id>

Per-Task Artifacts

The task prefix holds the durable artifacts:

• input.csv — uploaded by the user via a presigned URL.
• status.json — the task state and progress.
• output-attempt-N.csv — written by the worker, one per attempt.

Active Marker

active/<task-id> is a zero-byte object that acts as a worker index: its presence means “this task may need attention”. It is not the source of truth for the task’s state - status.json is. Markers are cheap to list for polling, but correctness always comes from the state document.

Task Lifecycle

A task moves through a small set of states:

stateDiagram-v2
    [*] --> created
    created --> queued: submit
    queued --> running: worker claims lease
    running --> completed: output written
    running --> failed: error
    failed --> queued: retry if attempt < max
    completed --> [*]

    created --> abandoned: cancel
    queued --> abandoned: cancel
    running --> abandoned: cancel
    failed --> abandoned: cancel
    abandoned --> [*]
    failed --> [*]: max attempt reached

• created: record exists and an upload URL was issued but the input is not there yet.
• queued: input uploaded, ready to be processed.
• running: a worker holds the lease and is processing it.
• completed: output written.
• failed: an attempt failed; retryable until attempt_id reaches the max, at which point the active marker is removed and the task stays failed permanently.
• abandoned: terminal - cancelled by an operator.

The created state exists because uploads go directly to S3: the API creates the task and returns a presigned URL before the file exists, then a separate submit step confirms the upload and moves the task to queued.

Task Record

status.json is the entire task record:

{
  "schema_version": 1,
  "task_id": "7ZZZZZZZY1F801XTH00MY7XF09K",
  "task_name": "June batch import",
  "status": "queued",
  "attempt_id": 0,
  "row_count": 1000,
  "processed_count": 250,
  "success_count": 245,
  "error_count": 5,
  "input_object_path": "batch-tasks/7ZZZZZZZY1F801XTH00MY7XF09K/input.csv",
  "output_object_path": null,
  "worker_id": "worker-8",
  "lease_expires_at": "2026-06-21T10:15:00Z",
  "last_error": null,
  "created_at": "2026-06-21T10:00:00Z",
  "updated_at": "2026-06-21T10:05:00Z",
  "created_by": "john.doe"
}

The exact fields depend on the workflow, but the useful categories are stable:

Treat the categories as the skeleton of any task record. The workflow-specific parts will vary - progress counters can be different per job, and the lifecycle can carry a task_type and payload that the worker dispatches on (more on that later). But identity, lifecycle, ownership, and audit are what make a task trackable, claimable, and debuggable regardless of what it actually does.

Worker

Polling for Work

The worker can run inside the existing service if the workload is small enough. It periodically lists batch-tasks/active/, reads each listed task’s status.json, and decides whether the task can be acquired. Listing the active markers keeps polling cheap, the worker scans only the handful of tasks that may need attention, not every task ever submitted.

When a task reaches a terminal state (completed or abandoned), the worker removes its active marker. The task’s status.json and output stay in S3 for audit but drop out of the polling set. Because the marker is only an index, neither failure mode corrupts the task: a missing marker on a pending task is recreated the next time it is retried or submitted, and a stale marker on a terminal task is harmless - the worker ignores or removes it on the next poll.

Acquiring a Task Safely

The hardest part is making sure two workers never process the same task. S3 conditional writes give us a compare and set (CAS) on a single object:

• Read status.json and capture its ETag
• Write the update only If-Match the same ETag

Worker A reads status.json → queued, ETag "abc"
Worker B reads status.json → queued, ETag "abc"

Worker A writes running, If-Match "abc" → success, new ETag "def"
Worker B writes running, If-Match "abc" → conflict

Only one worker wins the lease. The same primitive protects heartbeats, retries, cancellation, and terminal writes: a stale worker’s ETag no longer matches, so it cannot clobber newer state.

Leases and Retries

On acquisition the worker writes status=running, its worker_id, and a lease_expires_at, extending the lease as it makes progress. If it crashes, the lease expires and a retry path moves the task back to queued.

Retries write to attempt-specific paths so earlier output is never overwritten:

output-attempt-1.csv
output-attempt-2.csv

Cap the number of attempts. Without a limit, a task that fails deterministically due to bad input, a permanent downstream error will be re-queued forever, and workers will keep burning cycles re-running work that can never succeed. Once attempt_id hits the max, remove the active marker and leave the task in failed, it stays in S3 for audit but drops out of the polling set. Surface last_error so an operator can see why without digging through logs.

Newest-First Listing With Inverted ULIDs

S3 lists object keys lexicographically in ascending order, but an operator UI almost always wants “show me the newest tasks” — the equivalent of ORDER BY created_at DESC LIMIT 50. We get that for free from the task ID.

A ULID is a 26-character identifier: a 48-bit millisecond timestamp followed by 80 bits of randomness.

 0JWHDX3P8Q    KYGTVF3NE2RFMB
|----------|  |--------------|
 timestamp       randomness
 (10 chars)      (16 chars)
 48-bit ms       80 bits

ULID uses Crockford base32 (5 bits per character), so 10 chars hold 50 bits, the timestamp only occupies 48 of them. That makes normal ULIDs sortable by creation time (oldest first), which is the opposite of what we want. So we invert the timestamp:

inverted_timestamp = (2^48 - 1) - current_unix_ms

Then keep the normal random suffix.

The result is still a fixed-width, sortable, collision-resistant ID, but newer tasks produce smaller key prefixes. Since S3 lists keys ascending, listing the task prefix naturally returns newest-first.

Example:

batch-tasks/
  7ZZZZZZZY1F801XTH00MY7XF09K/   ← newest
  7ZZZZZZZY0HR51M5BZ9QY1Q8P0C/
  7ZZZZZZZ08S8B2Y2HXEBYB0C3PR/   ← oldest

The tradeoff: the key format now encodes an ordering decision. That is fine when documented clearly, but it is not a substitute for rich filtering or arbitrary sort orders, that is where a database wins.

API Layer

Control API Shape

A small control API is enough:

POST /tasks
POST /tasks/{id}/submit
GET  /tasks
GET  /tasks/{id}
POST /tasks/{id}/retry
POST /tasks/{id}/cancel
GET  /tasks/{id}/output

The key boundary is that the API owns state transitions. The UI can display a backend-computed retryable flag, but the retry endpoint re-checks state authoritatively, clients never decide transitions from their own clock or assumptions.

Presigned URLs Keep the Service Thin

The service never proxies file bytes; uploads and downloads go direct to S3.

Upload: POST /tasks writes status.json (created) and returns a presigned upload URL → the client uploads to S3 → POST /tasks/{id}/submit verifies the object exists, marks the task queued, and writes the active marker.

Download: GET /tasks/{id}/output verifies the task is completed and returns a short-lived presigned download URL.

This keeps the service focused on task control and authorization; S3 handles transfer.

Pagination

If history outgrows one screen, GET /tasks takes an opaque cursor:

GET /tasks?limit=50&cursor=...

Internally the cursor maps to an S3 start-after key (last task ID seen), which is passed to the next ListObjectsV2 call to resume from that position. Keep it opaque to clients, it is a pagination mechanism, not a query language.

Tradeoffs and Limits

Where It Fits

This pattern is a good fit when:

• Workflow is file-oriented and artifacts already belong in S3
• Task volume is low and not high-throughput
• Seconds-level pickup latency is acceptable
• UI only needs recent task history
• Tasks fail cleanly: errors are retryable or cancellable, not partial/unrecoverable
• Keeping infrastructure footprint minimal matters more than query flexibility

Where It Breaks Down

S3 is not a queue: there is no built-in message locking, no dead-letter queue, no consumer groups, and no dispatch policy, you build the state transitions yourself. Listing is cheap for small pages but it is not a query engine, and rendering a list costs one list call plus one status.json read per task shown (fine for 20–50 rows, bad for thousands).

Reach for the right tool once the workflow earns it:

Final Thought: Typed Tasks As A Future Extension

Start with one concrete workflow to keep the model honest and avoids prematurely building a generic task platform. But if the service later owns several similar low-volume jobs, status.json can grow a task_type and a type-specific payload:

{
  "task_id": "7ZZZZZZZY1F801XTH00MY7XF09K",
  "task_type": "email_enrichment",
  "status": "queued",
  "payload": {
    "input_object": "batch-tasks/7ZZZZZZZY1F801XTH00MY7XF09K/input.csv",
    "output_format": "csv"
  }
}

The worker can then dispatch by type:

email_enrichment     → read CSV, enrich rows, write output
report_export        → generate report artifact
bulk_reconciliation  → run reconciliation and emit result file

This works only while the types share one lifecycle, retry model, permissions, and operational profile. Once they need different SLAs, worker pools, retention, or query patterns, stop stretching the pattern, that is the signal for a database, queue, or workflow engine.

The goal was never to avoid infrastructure forever. It is to add it once the workflow has earned it.