[{"content":"I recently needed to build a low volume internal workflow: a user uploads a CSV, a backend processes it asynchronously, and the user downloads the results.\nThe standard approach job processing stack would be a database for task metadata, a queue for dispatch, and worker deployments. But for an occasional internal tool, that\u0026rsquo;s often more infrastructure than the problem deserves.\nSince the workflow was entirely file-oriented (large input and output CSVs), S3 was already the natural fit.\nSo the question became: could S3 also hold the task state and handle concurrent workers safely for a workflow that only runs occasionally?\n🥁 \u0026hellip; Yes, while everyone knows S3 is great for storing files, what makes this pattern interesting is how far you can push it. Using a tiny metadata file status.json as the task record, S3 conditional writes for safe concurrency, and an inverted ULID for newest-first listing, you can get a lightweight task queue for free.\nDisclaimer: this isn\u0026rsquo;t \u0026ldquo;S3 as a replacement for queue\u0026rdquo;. It\u0026rsquo;s a highly targeted pattern for low-volume, file-oriented workflows where the main goal is keeping your infrastructure footprint minimal while keeping the workflow durable and recoverable.\nWhen Does This Pattern Make Sense? \u0026ldquo;Low volume\u0026rdquo; is subjective, so let\u0026rsquo;s put numbers on it. This design is a great fit when:\nYou process roughly 1 to 100 tasks per day. Seconds-level latency for task pickup is acceptable. You only need to list the latest tasks (no complex querying, filtering or reporting). If your workload requires high throughput, complex filtering, or strict ordering, you absolutely should reach for a database and queue.\nBut for small, file-oriented workflows, the standard infrastructure overhead (database schemas, queue configuration, worker deployments, and monitoring runbooks) often becomes larger than the feature itself.\nHow It Fits Together Before the details, here is the whole system in one view. It is four moving parts on top of S3:\nArtifacts live in a per-task prefix - the input and output files. State lives in a tiny status.json next to them - the task record. Workers find work by listing a small set of marker objects, then claim tasks using S3 conditional writes (compare and set - CAS). Ordering comes for free from the task ID (inverted ULID), so the newest tasks list first. sequenceDiagram actor C as Client participant API as Control API participant S3 participant W as Worker Note over C,S3: Submit C-\u003e\u003eAPI: POST /tasks API-\u003e\u003eS3: write status.json (created) API--\u003e\u003eC: presigned upload URL C-\u003e\u003eS3: upload input.csv C-\u003e\u003eAPI: POST /tasks/{id}/submit API-\u003e\u003eS3: status.json (queued) + active marker Note over W,S3: Process W-\u003e\u003eS3: list active/ W-\u003e\u003eS3: claim via If-Match ETag W-\u003e\u003eS3: write output + status.json (completed) Note over C,S3: Download C-\u003e\u003eAPI: GET /tasks/{id}/output API--\u003e\u003eC: presigned download URL C-\u003e\u003eS3: download output.csv The rest of this post walks through each part: the layout, the state record, the worker, the ordering trick, the API, and where the pattern stops fitting.\nS3 Layout Each task gets its own prefix:\nbatch-tasks/ \u0026lt;task-id\u0026gt;/ input.csv status.json output-attempt-1.csv active/ \u0026lt;task-id\u0026gt; Per-Task Artifacts The task prefix holds the durable artifacts:\ninput.csv — uploaded by the user via a presigned URL. status.json — the task state and progress. output-attempt-N.csv — written by the worker, one per attempt. Active Marker active/\u0026lt;task-id\u0026gt; is a zero-byte object that acts as a worker index: its presence means \u0026ldquo;this task may need attention\u0026rdquo;. It is not the source of truth for the task\u0026rsquo;s state - status.json is. Markers are cheap to list for polling, but correctness always comes from the state document.\nTask Lifecycle A task moves through a small set of states:\nstateDiagram-v2 [*] --\u003e created created --\u003e queued: submit queued --\u003e running: worker claims lease running --\u003e completed: output written running --\u003e failed: error failed --\u003e queued: retry if attempt \u003c max completed --\u003e [*] created --\u003e abandoned: cancel queued --\u003e abandoned: cancel running --\u003e abandoned: cancel failed --\u003e abandoned: cancel abandoned --\u003e [*] failed --\u003e [*]: max attempt reached created: record exists and an upload URL was issued but the input is not there yet. queued: input uploaded, ready to be processed. running: a worker holds the lease and is processing it. completed: output written. failed: an attempt failed; retryable until attempt_id reaches the max, at which point the active marker is removed and the task stays failed permanently. abandoned: terminal - cancelled by an operator. The created state exists because uploads go directly to S3: the API creates the task and returns a presigned URL before the file exists, then a separate submit step confirms the upload and moves the task to queued.\nTask Record status.json is the entire task record:\n{ \u0026#34;schema_version\u0026#34;: 1, \u0026#34;task_id\u0026#34;: \u0026#34;7ZZZZZZZY1F801XTH00MY7XF09K\u0026#34;, \u0026#34;task_name\u0026#34;: \u0026#34;June batch import\u0026#34;, \u0026#34;status\u0026#34;: \u0026#34;queued\u0026#34;, \u0026#34;attempt_id\u0026#34;: 0, \u0026#34;row_count\u0026#34;: 1000, \u0026#34;processed_count\u0026#34;: 250, \u0026#34;success_count\u0026#34;: 245, \u0026#34;error_count\u0026#34;: 5, \u0026#34;input_object_path\u0026#34;: \u0026#34;batch-tasks/7ZZZZZZZY1F801XTH00MY7XF09K/input.csv\u0026#34;, \u0026#34;output_object_path\u0026#34;: null, \u0026#34;worker_id\u0026#34;: \u0026#34;worker-8\u0026#34;, \u0026#34;lease_expires_at\u0026#34;: \u0026#34;2026-06-21T10:15:00Z\u0026#34;, \u0026#34;last_error\u0026#34;: null, \u0026#34;created_at\u0026#34;: \u0026#34;2026-06-21T10:00:00Z\u0026#34;, \u0026#34;updated_at\u0026#34;: \u0026#34;2026-06-21T10:05:00Z\u0026#34;, \u0026#34;created_by\u0026#34;: \u0026#34;john.doe\u0026#34; } The exact fields depend on the workflow, but the useful categories are stable:\nCategory Fields Notes identity task_id, task_name, created_by task_id is the inverted ULID (see ordering).\ncreated_by is whoever submitted it, useful for audit and access checks. lifecycle schema_version, status, attempt_id schema_version lets you evolve the record format safely.\nattempt_id increments on each retry and names the output file (output-attempt-2.csv). progress row_count, processed_count, success_count, error_count Drives the operator UI\u0026rsquo;s progress bar.\nThe worker updates these as it streams through the input. storage input_object_path, output_object_path Full S3 keys.\noutput_object_path stays null until the run completes. ownership worker_id, lease_expires_at worker_id identifies the current holder - a pod name, hostname, or instance ID is fine.\nlease_expires_at is how long that worker\u0026rsquo;s claim is valid. The worker extends it while making progress; if it lapses, the task is treated as not picked up (the worker likely crashed) and becomes available for another worker to retry. audit created_at, updated_at, last_error last_error captures the most recent failure message so an operator can see why a task failed without digging through logs. Treat the categories as the skeleton of any task record. The workflow-specific parts will vary - progress counters can be different per job, and the lifecycle can carry a task_type and payload that the worker dispatches on (more on that later). But identity, lifecycle, ownership, and audit are what make a task trackable, claimable, and debuggable regardless of what it actually does.\nWorker Polling for Work The worker can run inside the existing service if the workload is small enough. It periodically lists batch-tasks/active/, reads each listed task\u0026rsquo;s status.json, and decides whether the task can be acquired. Listing the active markers keeps polling cheap, the worker scans only the handful of tasks that may need attention, not every task ever submitted.\nWhen a task reaches a terminal state (completed or abandoned), the worker removes its active marker. The task\u0026rsquo;s status.json and output stay in S3 for audit but drop out of the polling set. Because the marker is only an index, neither failure mode corrupts the task: a missing marker on a pending task is recreated the next time it is retried or submitted, and a stale marker on a terminal task is harmless - the worker ignores or removes it on the next poll.\nAcquiring a Task Safely The hardest part is making sure two workers never process the same task. S3 conditional writes give us a compare and set (CAS) on a single object:\nRead status.json and capture its ETag Write the update only If-Match the same ETag Worker A reads status.json → queued, ETag \u0026#34;abc\u0026#34; Worker B reads status.json → queued, ETag \u0026#34;abc\u0026#34; Worker A writes running, If-Match \u0026#34;abc\u0026#34; → success, new ETag \u0026#34;def\u0026#34; Worker B writes running, If-Match \u0026#34;abc\u0026#34; → conflict Only one worker wins the lease. The same primitive protects heartbeats, retries, cancellation, and terminal writes: a stale worker\u0026rsquo;s ETag no longer matches, so it cannot clobber newer state.\nLeases and Retries On acquisition the worker writes status=running, its worker_id, and a lease_expires_at, extending the lease as it makes progress. If it crashes, the lease expires and a retry path moves the task back to queued.\nRetries write to attempt-specific paths so earlier output is never overwritten:\noutput-attempt-1.csv output-attempt-2.csv Cap the number of attempts. Without a limit, a task that fails deterministically due to bad input, a permanent downstream error will be re-queued forever, and workers will keep burning cycles re-running work that can never succeed. Once attempt_id hits the max, remove the active marker and leave the task in failed, it stays in S3 for audit but drops out of the polling set. Surface last_error so an operator can see why without digging through logs.\nNewest-First Listing With Inverted ULIDs S3 lists object keys lexicographically in ascending order, but an operator UI almost always wants \u0026ldquo;show me the newest tasks\u0026rdquo; — the equivalent of ORDER BY created_at DESC LIMIT 50. We get that for free from the task ID.\nA ULID is a 26-character identifier: a 48-bit millisecond timestamp followed by 80 bits of randomness.\n0JWHDX3P8Q KYGTVF3NE2RFMB |----------| |--------------| timestamp randomness (10 chars) (16 chars) 48-bit ms 80 bits ULID uses Crockford base32 (5 bits per character), so 10 chars hold 50 bits, the timestamp only occupies 48 of them. That makes normal ULIDs sortable by creation time (oldest first), which is the opposite of what we want. So we invert the timestamp:\ninverted_timestamp = (2^48 - 1) - current_unix_ms Then keep the normal random suffix.\nThe result is still a fixed-width, sortable, collision-resistant ID, but newer tasks produce smaller key prefixes. Since S3 lists keys ascending, listing the task prefix naturally returns newest-first.\nExample:\nbatch-tasks/ 7ZZZZZZZY1F801XTH00MY7XF09K/ ← newest 7ZZZZZZZY0HR51M5BZ9QY1Q8P0C/ 7ZZZZZZZ08S8B2Y2HXEBYB0C3PR/ ← oldest The tradeoff: the key format now encodes an ordering decision. That is fine when documented clearly, but it is not a substitute for rich filtering or arbitrary sort orders, that is where a database wins.\nAPI Layer Control API Shape A small control API is enough:\nPOST /tasks POST /tasks/{id}/submit GET /tasks GET /tasks/{id} POST /tasks/{id}/retry POST /tasks/{id}/cancel GET /tasks/{id}/output The key boundary is that the API owns state transitions. The UI can display a backend-computed retryable flag, but the retry endpoint re-checks state authoritatively, clients never decide transitions from their own clock or assumptions.\nPresigned URLs Keep the Service Thin The service never proxies file bytes; uploads and downloads go direct to S3.\nUpload: POST /tasks writes status.json (created) and returns a presigned upload URL → the client uploads to S3 → POST /tasks/{id}/submit verifies the object exists, marks the task queued, and writes the active marker.\nDownload: GET /tasks/{id}/output verifies the task is completed and returns a short-lived presigned download URL.\nThis keeps the service focused on task control and authorization; S3 handles transfer.\nPagination If history outgrows one screen, GET /tasks takes an opaque cursor:\nGET /tasks?limit=50\u0026amp;cursor=... Internally the cursor maps to an S3 start-after key (last task ID seen), which is passed to the next ListObjectsV2 call to resume from that position. Keep it opaque to clients, it is a pagination mechanism, not a query language.\nTradeoffs and Limits Where It Fits This pattern is a good fit when:\nWorkflow is file-oriented and artifacts already belong in S3 Task volume is low and not high-throughput Seconds-level pickup latency is acceptable UI only needs recent task history Tasks fail cleanly: errors are retryable or cancellable, not partial/unrecoverable Keeping infrastructure footprint minimal matters more than query flexibility Where It Breaks Down S3 is not a queue: there is no built-in message locking, no dead-letter queue, no consumer groups, and no dispatch policy, you build the state transitions yourself. Listing is cheap for small pages but it is not a query engine, and rendering a list costs one list call plus one status.json read per task shown (fine for 20–50 rows, bad for thousands).\nReach for the right tool once the workflow earns it:\nTool When Database Filtering, joins, arbitrary ordering, or pagination across many records Queue High throughput, backpressure, fair distribution, delayed retries, or dead-lettering Workflow engine Multi-step orchestration: dependencies, approvals, branching, cross-step recovery Final Thought: Typed Tasks As A Future Extension Start with one concrete workflow to keep the model honest and avoids prematurely building a generic task platform. But if the service later owns several similar low-volume jobs, status.json can grow a task_type and a type-specific payload:\n{ \u0026#34;task_id\u0026#34;: \u0026#34;7ZZZZZZZY1F801XTH00MY7XF09K\u0026#34;, \u0026#34;task_type\u0026#34;: \u0026#34;email_enrichment\u0026#34;, \u0026#34;status\u0026#34;: \u0026#34;queued\u0026#34;, \u0026#34;payload\u0026#34;: { \u0026#34;input_object\u0026#34;: \u0026#34;batch-tasks/7ZZZZZZZY1F801XTH00MY7XF09K/input.csv\u0026#34;, \u0026#34;output_format\u0026#34;: \u0026#34;csv\u0026#34; } } The worker can then dispatch by type:\nemail_enrichment → read CSV, enrich rows, write output report_export → generate report artifact bulk_reconciliation → run reconciliation and emit result file This works only while the types share one lifecycle, retry model, permissions, and operational profile. Once they need different SLAs, worker pools, retention, or query patterns, stop stretching the pattern, that is the signal for a database, queue, or workflow engine.\nThe goal was never to avoid infrastructure forever. It is to add it once the workflow has earned it.\n","permalink":"https://tayts.vercel.app/posts/s3-task-store-queue/","summary":"ETags for compare-and-set, an inverted ULID for newest-first ordering, a zero-byte marker as a polling index — turns out that\u0026rsquo;s enough for a real task queue, no database or queue service needed.","title":"Building a Low-Volume Batch Worker Without a Database: Using S3 as Task Storage"},{"content":"Hi, I’m TS. And NO, it doesn’t stand for TypeScript. I’m a software engineer who is just trying to understand how things work under the hood and get better at building reliable, scalable software in the midst of the AI hype.\nI am a strong believer that the foundations of software engineering like algorithms, data structures, design patterns, and clean code are still relevant in the AI era. AI may change the way we build software, but it doesn\u0026rsquo;t change the fundamental principles of software engineering. I hope to share my thoughts and experiences on these topics, as well as my journey in the world of software engineering.\nP.S. I am fueled by cat and kitten videos. Please say hello with a meow~\n⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣀⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀ ⠀⠀⠀⢀⣴⠖⠚⠛⠒⠶⠦⣤⡾⠋⠙⣦⣤⣤⣤⡴⠚⢳⡀⠀ ⠀⠀⢠⠟⠀⠀⠀⠀⠀⠀⠀⠘⠃⠀⠀⠀⠀⠀⠀⠀⠀⠘⣇⠀ ⠀⣠⡟⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⡆ ⣸⠋⠟⡀⠀⠀⢀⡀⠀⠀⠀⠀⠀⠀⢧⣤⠆⢀⡀⢠⣤⡤⢸⡇ ⠹⣄⠀⠙⠛⠛⠉⠉⠹⣧⠀⠚⠓⢦⡀⠀⠀⠈⠉⠀⠀⢠⣿⡄ ⠀⠙⢳⢦⣤⣤⣤⢴⡿⡳⠶⢤⣤⡼⣷⠦⢤⣤⣤⡤⡴⣿⠞⠃ ⠀⠀⠁⠊⠁⠈⠈⠁⠀⠀⠈⠁⠉⠀⠁⠀⠉⠐⠁⠉⠀⠀⠀⠀ ","permalink":"https://tayts.vercel.app/about/","summary":"About me","title":"About"}]