Skip to content

Complete Queue lifecycle safety, durability, and Laravel parity - #457

Merged
binaryfire merged 31 commits into
0.4from
audit/queue-lifecycles-parity
Jul 28, 2026
Merged

Complete Queue lifecycle safety, durability, and Laravel parity#457
binaryfire merged 31 commits into
0.4from
audit/queue-lifecycles-parity

Conversation

@binaryfire

Copy link
Copy Markdown
Collaborator

Summary

This PR completes the Queue lifecycle, durability, and current Laravel parity work.

The central change is ownership. Deferred work no longer retains a borrowed queue object across a database transaction. The prepared payload survives until commit, but the physical queue lease does not. A successful commit borrows a fresh queue for the operation, while the immediate path continues to use the current lease. This keeps size-one pools live and prevents one coroutine from publishing through an object another coroutine has borrowed.

The rest of the work follows that same rule: every queue, connection, payload, transaction callback, temporary file, and cache entry has one clear owner and one terminal cleanup path.

This PR also:

  • ports current SQS batching, credential providers, FIFO retry options, and overflow payload storage;
  • makes malformed external payloads terminal, observable, and durable instead of repeatedly releasing or losing them;
  • reserves malformed Redis jobs atomically without rewriting their raw bytes;
  • completes worker timeout, idle, quiet-stop, pause, and stop metadata;
  • adds current Queue middleware, inspection, command output, and QueueFake behavior;
  • publishes file failed-job records atomically and preserves malformed failed-job evidence until retry succeeds;
  • corrects Redis scan boundaries and case-insensitive command metadata used by Queue inspection;
  • restores current Laravel-facing contracts where Hypervel had drifted; and
  • updates configuration ownership, package metadata, documentation, and audit records.

For more details, see: docs/plans/2026-07-28-0353-queue-pooling-payload-durability-and-current-laravel-parity.md

Pooled dispatch

After-commit dispatch now captures a prepared operation rather than a borrowed queue. The pool proxy supplies an owner-first dispatcher when a queue is checked out and clears it before the object returns to the pool.

This distinction matters:

  • no active transaction means the current lease performs the operation directly;
  • an active transaction registers the prepared operation and releases the current lease;
  • commit borrows exactly one fresh lease for the operation;
  • rollback publishes nothing and releases the resolved unique and debounce locks; and
  • shared physical pools overwrite logical connection identity on every borrow.

Synchronous, background, and deferred queues resolve one transaction manager and use that same owner for registration and rollback cleanup.

SQS

Bulk publication uses SendMessageBatch with ordered chunks bounded by SQS count and byte limits. After-commit bulk work uses one callback and one lease for the batch rather than one checkout per job.

Oversized message bodies are written only for the chunk about to be sent. The cleanup rules distinguish the outcomes that matter:

  • local publication failure removes entries for the unsent chunk;
  • explicit SQS rejection removes that entry;
  • an ambiguous transport failure retains the body because SQS may have accepted the message;
  • later unsent chunks never create cache entries; and
  • terminal deletion releases the queue lease and then removes the overflow body without allowing cleanup to replace an earlier transport failure.

Credential providers now support the current static, callable, ECS, and instance forms. FIFO group and deduplication identifiers preserve exact values, including "0".

Payload durability

Queue payloads are decoded and validated once per job. The job caches either the validated payload or the exact InvalidPayloadException, so every later accessor observes the same result.

Malformed jobs now follow a terminal path:

  • emit the normal exception and failure lifecycle events;
  • persist the failed record;
  • delete the transport job;
  • skip payload-dependent hooks that cannot be resolved safely;
  • report the original payload failure once; and
  • never calculate backoff or release the poison job.

Redis reservation performs attempts validation inside the existing Lua round trip. Invalid JSON, scalar payloads, missing or nonnumeric attempts, and raw "0" values are reserved with their exact bytes intact. That prevents loss between LPOP and reservation and preserves the real member needed for removal.

Horizon and Telescope contain only the named invalid-payload telemetry failure after the queue action succeeds. Ordinary application, listener, database, and transport exceptions continue to propagate.

Worker lifecycle

Workers now distinguish admitted jobs from completed jobs and use one monotonic clock for runtime deadlines, timeout monitoring, and quiet-period stopping.

The worker surface includes:

  • explicit non-expiring behavior for --timeout=0;
  • WorkerIdle events;
  • --stop-when-empty-for;
  • truthful timeout and stop reasons;
  • completion time and memory metadata;
  • one batched paused-queue cache read;
  • in-flight draining across every stop path; and
  • boot-only controls for reporting job exceptions and stopping on lost connections.

The implementation does not add process-global current-job state. Concurrent jobs remain daemon-owned entries, which is safe for Hypervel's coroutine worker.

Queue APIs and storage

This ports or completes current Laravel-shaped behavior for:

  • release middleware;
  • callable exception-throttle backoff;
  • enum-backed queue, connection, limiter, and overlap identifiers;
  • DateInterval releases;
  • eager pending, delayed, and reserved inspection;
  • JSON failed-job and monitor output;
  • QueueFake delayed, reserved, and before/after push behavior; and
  • current database failed-job provider construction and filtering.

Inspection remains a concrete capability rather than being added to the core Queue contract. It intentionally returns eager collections and is documented as an operational API that should not be used against very large backlogs in latency-sensitive paths.

The file failed-job provider now writes a complete same-directory temporary file and atomically renames it under the existing lock. It preserves the current mode where available, removes unpublished temporary files, and never lets cleanup replace the primary persistence failure.

Redis and contracts

Queue inspection consumes Redis scans through a raw phpredis connection. Transformed scan results have a different shape and are rejected before scanning or deleting anything.

Redis command metadata now gives case-insensitive method names the same truthful signature, preserves native score-range argument order, and separates Redis transaction discard from pooled connection lifecycle discard.

The contract updates remove optional channel enumeration from the core broadcaster contract and allow Notification factories to accept every notifiable shape their implementations already support. Queue package metadata now declares the dependencies required by its split package.

Performance

The ordinary dispatch path remains one pool checkout and does not allocate a dispatcher closure per operation. The retained dispatcher is one bounded closure per worker-cached logical proxy.

Real after-commit work adds one necessary checkout at commit instead of retaining a lease for the duration of the transaction. SQS bulk reduces network requests by sending bounded batches. Valid payloads decode once. Redis poison handling stays inside the existing Lua round trip.

Paused-queue lookup uses one cache many() call. This replaces sequential reads for empty or later-priority queues; a busy first-priority queue transfers a few additional boolean values but does not add another network round trip. The read size depends on configured queue names, not backlog size.

Eager inspection is not on the worker or request hot path. It is explicitly documented as memory-sensitive operational behavior.

Compatibility

Public APIs follow current Laravel naming and shapes unless Hypervel's pooling or coroutine runtime requires a deliberate difference.

The protected enqueueUsing() callback receives the operation's queue owner first. This is documented in the Queue package README because it is the boundary that prevents deferred pooled work from retaining a borrowed object.

Hypervel's concurrent worker, pooling, queue-route, debounce, payload-context, SafeScan, and Redis transformation behavior remain intact.

Validation

Focused Queue, Redis, Horizon, Telescope, database, SQS, worker, middleware, failed-job, facade, package metadata, and integration coverage was run throughout the work.

The final repository gate was run through composer fix, covering formatting, static analysis, and the full parallel test suite. The final diff also passes whitespace validation.

Record the settled ownership model for pooled queues, deferred dispatch, SQS overflow storage, poison payloads, worker lifecycle, and eager inspection.

Capture the approved Laravel parity boundaries, anti-overengineering constraints, performance analysis, regression matrix, and full validation workflow so the implementation remains auditable across compactions.
Document that framework configuration is shallow-merged, with mergeable options reserved for named groups whose individual entries replace matching defaults.

Align the configuration bootstrap and package provider docblocks with that contract, and add regressions proving named queue connections merge independently while other nested settings replace wholesale.
Give deferred queue operations an explicit owner-first dispatcher so transaction callbacks never retain a borrowed queue beyond its lease.

Configure and clear the dispatcher at pool boundaries, preserve the immediate path for size-one pools, and cover logical connection reuse, rollback cleanup, callback ownership, and end-to-end transaction behavior.
Resolve one transaction manager for each synchronous, background, and deferred after-commit operation and pass that exact owner through rollback and commit registration.

Keep execution-time payload behavior intact, return a predictable null for deferred work, and expand the fixtures to model a real active transaction without permissive cache behavior.
Make Beanstalk size include ready, delayed, and reserved jobs from one tube-stats read, matching the active workload represented by other drivers.

Retain buried jobs as dead-letter state, correct the contract wording, and verify the total and individual metrics without adding another network request.
Expose eager pending, delayed, and reserved inspection for database-backed queues while preserving queue, attempt, and record identifiers.

Validate inspected payloads through the shared Queue boundary and cover status classification, exact queue filtering, ordering, and invalid-payload diagnostics.
Accept static credentials, callable or object providers, and the current ECS and instance provider names while memoizing provider resolution.

Keep pool fingerprints explicit for dynamic configuration, omit non-client options, and fail unsupported provider names with a descriptive exception.
Publish SQS jobs through ordered SendMessageBatch chunks while retaining exact queue events, FIFO options, and one deferred lease per batch.

Store oversized bodies only for the chunk being sent, preserve ambiguous deliveries, clean explicit rejections, and enforce terminal delete, lease release, and overflow cleanup precedence.
Decode each consumed payload once, cache either the validated array or the exact InvalidPayloadException, and reject malformed or incomplete job envelopes consistently.

Preserve raw evidence in diagnostics and extend the database rollback contract to the level-aware behavior already required by timeout cleanup.
Move Redis attempts validation into the existing Lua reservation round trip so malformed JSON, scalar payloads, invalid attempts, and raw zero values are reserved without mutation or loss.

Carry the reservation attempt count into RedisJob, make identifiers lazy and non-throwing, add eager raw-mode inspection, and verify exact removal members against real Redis.
Translate malformed queue payloads into the shared Queue exception and contain only that named failure after the underlying queue action succeeds.

Preserve valid entries in mixed migrations, skip unavailable timer identifiers, and keep unrelated Horizon listener and transport failures observable.
Stop Telescope failed-job recording when the shared Queue payload validator rejects external data, avoiding a second unsafe payload read.

Keep all ordinary watcher failures and valid payload behavior unchanged, with focused coverage for malformed job telemetry.
Add truthful timeout, idle, quiet-stop, completion, pause, and stop metadata while keeping concurrent jobs in daemon-owned state rather than worker singletons.

Use one monotonic clock, batch paused-queue reads, preserve in-flight work across stop paths, reset boot-only controls between tests, and distinguish admitted from completed jobs.
Add quiet-period stopping, explicit pause capability errors, current monitor formatting, and complete verbose and JSON worker output.

Preserve exact zero values and command option precedence while carrying the real connection, queue, elapsed time, memory, and lifecycle context through each operational path.
Align both database failed-job providers with the current constructor order and expose their configured table.

Preserve exact non-null connection and queue filters, including string zero, and cover lookup, pruning, counting, and construction behavior for both identifier strategies.
Write failed-job snapshots through a same-directory temporary file under the existing lock, require complete I/O, preserve file modes where available, and rename atomically.

Clean unpublished temporary files without replacing the primary failure and cover false reads, short writes, rename failures, visibility, mode preservation, and cleanup.
Make failed-driver selection explicit and fail unsupported values rather than silently constructing an invalid provider.

Keep file path and limit defaults at the provider that consumes them, preserve replace-whole failed and batching configuration, and verify every supported driver and optional file setting.
Add current JSON listing output and preserve malformed failed-job evidence until a retry is successfully re-enqueued.

Decode retry payloads strictly, retain arbitrary and FIFO fields, support direct and pooled SQS options, and cover valid, missing, malformed, and failed requeue paths.
Pass the actual framework job through unique-until-processing cleanup and accept enum-backed queue, connection, limiter, and overlap identifiers.

Normalize identifiers once, preserve integer-backed zero and explicit zero delays, and retain serialized rate-limiter behavior across both cache implementations.
Add Laravel-shaped Release middleware and callable exception backoff for both cache-backed throttling implementations.

Keep conditional release APIs, exact backoff resolution, and Redis behavior aligned with current upstream while covering each public branch through integration tests.
Route DateInterval releases through the existing seconds-until helper already used for date-time values.

Keep integer and absolute-date behavior intact and make fake assertions observe the same normalized delay contract.
Add concrete pending, delayed, and reserved inspection to inspectable queues, delegate through failover and facade surfaces, and return empty collections for unsupported drivers.

Keep the optional capability out of the core contract and verify eager result behavior and facade metadata without adding a second abstraction.
Record immediate, delayed, and reserved fake jobs as disjoint state while preserving one ordered push history.

Carry delays through partial fakes, route push and later through one operation boundary, and add instance-owned before and after hooks whose state publishes only after successful operations.
Reject transformed Redis connections before SafeScan begins because transformed tuples cannot preserve native scan cursors or deletion members.

Keep FlushByPattern on the same raw boundary, update the public usage guidance, and verify no scan or deletion is attempted with an incompatible connection.
Reconcile case-insensitive phpredis method signatures, preserve mixed SETNX and HSETNX values, and keep reverse score bounds in native order.

Separate pooled connection lifecycle discard from Redis transaction discard, document held-connection ownership, regenerate facade metadata, and cover transformed, raw, and integration behavior.
Remove channel enumeration from the core broadcaster contract because conforming broadcasters are not required to implement it.

Keep the concrete command capability through one narrowly documented dynamic call instead of introducing a one-consumer capability contract.
Widen the notification factory boundary to the mixed notifiable shapes already supported by the dispatcher and implementations.

Keep sendNow at the factory's two-argument contract and remove the false dependency on a particular collection representation.
Declare Queue's direct context and filesystem requirements so its split package is complete in isolation.

Use the shared PHP and Artisan binary helpers in the listener and verify the package manifest rather than relying on root-monorepo dependency leakage.
Document SQS overflow storage, inspection, release middleware, worker lifecycle controls, failed-job output, and malformed-payload handling in the existing task-first style.

Record the deliberate owner-first protected callback difference needed for pooled deferred work and keep operational memory and dedicated-store warnings at the user-facing surfaces.
Mark the Queue package complete and record the verified lifecycle, durability, parity, performance, and cross-package findings in the audit ledger.

Update dependency routing and completed-package revalidation so future audits can trace every contract and source change made by this work unit.
@coderabbitai

coderabbitai Bot commented Jul 28, 2026

Copy link
Copy Markdown

Important

Review skipped

Too many files!

This PR contains 127 files, which is 27 over the limit of 100.

To get a review, narrow the scope:
• coderabbit review --committed # exclude uncommitted changes
• coderabbit review --dir # limit to a subdirectory
• coderabbit review --base # compare against a closer base

Upgrade to a paid plan to raise the limit.

This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 75f0cb87-2109-40ba-b621-2f9a813ea92b

📥 Commits

Reviewing files that changed from the base of the PR and between 86c4fbe and ff49060.

📒 Files selected for processing (127)
  • AGENTS.md
  • docs/plans/2026-07-12-0900-framework-coroutine-state-lifecycle-audit.md
  • docs/plans/2026-07-12-0915-framework-coroutine-state-lifecycle-audit-ledger.md
  • docs/plans/2026-07-28-0353-queue-pooling-payload-durability-and-current-laravel-parity.md
  • src/boost/docs/queues.md
  • src/boost/docs/redis.md
  • src/contracts/src/Broadcasting/Broadcaster.php
  • src/contracts/src/Notifications/Factory.php
  • src/contracts/src/Queue/Queue.php
  • src/database/src/ConnectionInterface.php
  • src/foundation/config/queue.php
  • src/foundation/src/Bootstrap/LoadConfiguration.php
  • src/foundation/src/Console/ChannelListCommand.php
  • src/horizon/src/Events/JobsMigrated.php
  • src/horizon/src/JobPayload.php
  • src/horizon/src/Listeners/ForgetJobTimer.php
  • src/horizon/src/Listeners/MarshalFailedEvent.php
  • src/horizon/src/RedisQueue.php
  • src/queue/README.md
  • src/queue/composer.json
  • src/queue/src/Attributes/Connection.php
  • src/queue/src/Attributes/Queue.php
  • src/queue/src/BackgroundQueue.php
  • src/queue/src/BeanstalkdQueue.php
  • src/queue/src/CallQueuedHandler.php
  • src/queue/src/Connectors/SqsConnector.php
  • src/queue/src/Console/ListFailedCommand.php
  • src/queue/src/Console/MonitorCommand.php
  • src/queue/src/Console/PauseCommand.php
  • src/queue/src/Console/RetryCommand.php
  • src/queue/src/Console/WorkCommand.php
  • src/queue/src/DatabaseQueue.php
  • src/queue/src/DeferredQueue.php
  • src/queue/src/Events/JobReleasedAfterException.php
  • src/queue/src/Events/Looping.php
  • src/queue/src/Events/WorkerIdle.php
  • src/queue/src/Events/WorkerStopping.php
  • src/queue/src/Failed/DatabaseFailedJobProvider.php
  • src/queue/src/Failed/DatabaseUuidFailedJobProvider.php
  • src/queue/src/Failed/FileFailedJobProvider.php
  • src/queue/src/FailoverQueue.php
  • src/queue/src/InteractsWithQueue.php
  • src/queue/src/InvalidPayloadException.php
  • src/queue/src/Jobs/InspectedJob.php
  • src/queue/src/Jobs/Job.php
  • src/queue/src/Jobs/RedisJob.php
  • src/queue/src/Jobs/SqsJob.php
  • src/queue/src/Listener.php
  • src/queue/src/ListenerOptions.php
  • src/queue/src/LuaScripts.php
  • src/queue/src/Middleware/RateLimited.php
  • src/queue/src/Middleware/RateLimitedWithRedis.php
  • src/queue/src/Middleware/Release.php
  • src/queue/src/Middleware/ThrottlesExceptions.php
  • src/queue/src/Middleware/ThrottlesExceptionsWithRedis.php
  • src/queue/src/Middleware/WithoutOverlapping.php
  • src/queue/src/NullQueue.php
  • src/queue/src/Queue.php
  • src/queue/src/QueueManager.php
  • src/queue/src/QueuePoolProxy.php
  • src/queue/src/QueueServiceProvider.php
  • src/queue/src/RedisQueue.php
  • src/queue/src/SqsQueue.php
  • src/queue/src/SyncQueue.php
  • src/queue/src/Worker.php
  • src/queue/src/WorkerOptions.php
  • src/queue/src/WorkerStopReason.php
  • src/redis/src/Operations/FlushByPattern.php
  • src/redis/src/Operations/SafeScan.php
  • src/redis/src/RedisConnection.php
  • src/redis/src/RedisProxy.php
  • src/support/src/Facades/Queue.php
  • src/support/src/Facades/Redis.php
  • src/support/src/ServiceProvider.php
  • src/support/src/Testing/Fakes/QueueFake.php
  • src/telescope/src/Watchers/JobWatcher.php
  • tests/Bus/UniqueJobPayloadContextTest.php
  • tests/Foundation/Fixtures/config/queue.php
  • tests/Foundation/FoundationApplicationTest.php
  • tests/Integration/Horizon/Feature/QueueProcessingTest.php
  • tests/Integration/Horizon/Feature/RedisPayloadTest.php
  • tests/Integration/Queue/QueueConnectionTest.php
  • tests/Integration/Queue/RateLimitedTest.php
  • tests/Integration/Queue/RateLimitedWithRedisTest.php
  • tests/Integration/Queue/RedisQueueTest.php
  • tests/Integration/Queue/ReleaseMiddlewareTest.php
  • tests/Integration/Queue/ThrottlesExceptionsTest.php
  • tests/Integration/Queue/ThrottlesExceptionsWithRedisTest.php
  • tests/Integration/Queue/WithoutOverlappingJobsTest.php
  • tests/Integration/Queue/WorkCommandTest.php
  • tests/Integration/Redis/RedisProxyIntegrationTest.php
  • tests/Queue/DatabaseFailedJobProviderTest.php
  • tests/Queue/DatabaseUuidFailedJobProviderTest.php
  • tests/Queue/FailoverQueueTest.php
  • tests/Queue/FileFailedJobProviderTest.php
  • tests/Queue/InteractsWithQueueTest.php
  • tests/Queue/ListFailedCommandTest.php
  • tests/Queue/MonitorCommandTest.php
  • tests/Queue/PackageMetadataTest.php
  • tests/Queue/QueueAttributesTest.php
  • tests/Queue/QueueBackgroundQueueTest.php
  • tests/Queue/QueueBeanstalkdQueueTest.php
  • tests/Queue/QueueConfigTest.php
  • tests/Queue/QueueDatabaseQueueUnitTest.php
  • tests/Queue/QueueDeferredQueueTest.php
  • tests/Queue/QueueExceptionTest.php
  • tests/Queue/QueueJobTest.php
  • tests/Queue/QueueListFailedCommandTest.php
  • tests/Queue/QueueManagerTest.php
  • tests/Queue/QueueNullQueueTest.php
  • tests/Queue/QueuePauseResumeTest.php
  • tests/Queue/QueuePoolProxyTest.php
  • tests/Queue/QueueRedisJobTest.php
  • tests/Queue/QueueRedisQueueTest.php
  • tests/Queue/QueueServiceProviderTest.php
  • tests/Queue/QueueSqsConnectorTest.php
  • tests/Queue/QueueSqsJobTest.php
  • tests/Queue/QueueSqsQueueTest.php
  • tests/Queue/QueueSyncQueueTest.php
  • tests/Queue/QueueWorkerTest.php
  • tests/Queue/RetryCommandTest.php
  • tests/Redis/Operations/FlushByPatternTest.php
  • tests/Redis/Operations/SafeScanTest.php
  • tests/Redis/PackageMetadataTest.php
  • tests/Redis/RedisConnectionTest.php
  • tests/Support/SupportTestingQueueFakeTest.php
  • tests/Telescope/Watchers/JobWatcherTest.php

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch audit/queue-lifecycles-parity

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Comment thread src/queue/src/SqsQueue.php
@greptile-apps

greptile-apps Bot commented Jul 28, 2026

Copy link
Copy Markdown

Greptile Summary

The PR completes queue lifecycle ownership, payload durability, worker behavior, SQS batching and overflow storage, Redis inspection, failed-job persistence, and Laravel-facing API parity.

  • Defers after-commit publication through freshly borrowed queue owners rather than retaining pooled leases.
  • Adds bounded SQS batch publication and explicit overflow-payload ownership rules.
  • Makes malformed payload handling terminal and durable across queue transports and telemetry integrations.
  • Expands worker lifecycle controls, queue middleware, inspection APIs, command output, and QueueFake behavior.
  • Hardens failed-job storage, Redis scanning, configuration, contracts, package metadata, tests, and documentation.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
src/queue/src/SqsQueue.php Adds SQS batching and overflow-payload lifecycle handling; the prior single-message cleanup concern is invalid because a thrown SDK request cannot establish that no retry attempt was accepted.
src/queue/src/Queue.php Refactors enqueue ownership and prepared after-commit operations so deferred work does not retain borrowed queue leases.
src/queue/src/QueuePoolProxy.php Supplies owner-first dispatch through fresh pooled leases and clears lease-specific state before returning objects.
src/queue/src/Worker.php Extends timeout, idle, quiet-stop, pause, draining, and completion metadata behavior for concurrent workers.
src/queue/src/RedisQueue.php Adds durable malformed-payload reservation and eager operational inspection behavior.
src/queue/src/Failed/FileFailedJobProvider.php Publishes failed-job records through checked same-directory temporary files and atomic replacement.
src/redis/src/Operations/SafeScan.php Rejects transformed Redis connections before scan-based operations can inspect or delete data.
src/support/src/Testing/Fakes/QueueFake.php Expands fake queue history and hook behavior to match the updated queue-facing API.

Reviews (2): Last reviewed commit: "docs(queue): explain ambiguous SQS overf..." | Re-trigger Greptile

@binaryfire

binaryfire commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator Author

@greptileai I reviewed the SQS overflow finding against the queue implementation and the installed AWS SDK.

No runtime change is appropriate. The finding assumes a thrown single-message request is definitively rejected, but the SDK retries connection and transient failures. A surfaced error can therefore follow an earlier attempt that SQS accepted after its response was lost. There is no supported signal proving that no attempt published the pointer, so deleting its body on failure risks losing a live job.

The existing batch cleanup remains correct: explicit per-entry Failed results are exact rejection evidence and are removed; a thrown batch retains the current chunk.

I pushed ff49060 to document both call-site invariants and add concise operator guidance for dedicated, monitored, non-evicting overflow storage. Focused SQS coverage, formatting, static analysis, and whitespace validation are green.

Document why an AWS SDK failure cannot prove a single-message publication was rejected, and why batch cleanup is limited to explicit per-entry failures.

Warn operators that non-expiring overflow bodies may outlive failed, expired, or cleared messages, and require a dedicated monitored store that does not evict live payloads. Runtime queue behavior remains unchanged.
@binaryfire
binaryfire force-pushed the audit/queue-lifecycles-parity branch from e5ebe5b to ff49060 Compare July 28, 2026 14:39
@greptile-apps

greptile-apps Bot commented Jul 28, 2026

Copy link
Copy Markdown

The reasoning is correct and the documentation approach is the right call.

The key distinction holds: the AWS SDK's built-in retry behavior means a surfaced exception from sendMessage is not proof of rejection — an earlier attempt may have succeeded and had its response dropped. Deleting the overflow body in that case would silently discard a live job's payload, leaving SQS pointing at a cache miss.

The batch path's cleanup of explicit Failed entries is correctly scoped — those are SQS-level per-entry rejection signals, not SDK transport failures, so they are safe to remove.

The inline comments in pushRaw and sendBatchedMessages now make both invariants visible at the call sites, and the operator warning in queues.md about non-expiring, monitored storage is the right surface for that guidance. The fix is complete.

@greptile-apps

greptile-apps Bot commented Jul 28, 2026

Copy link
Copy Markdown

The reasoning is sound and the documentation commit is the right resolution.

The core invariant holds: an AWS SDK exception on sendMessage does not prove SQS never accepted the message. The SDK exhausts its retry budget — which includes connection failures and transient errors — before surfacing an exception. Any of those retried attempts may have been accepted by SQS with the response lost in transit. Deleting the overflow body at that point risks orphaning a live job whose payload the worker needs to execute.

The asymmetry between the two call sites is correctly handled:

  • pushRaw / thrown sendMessage — outcome is ambiguous; retain the body.
  • sendMessageBatch / explicit Failed entries — per-entry rejection is deterministic; those paths can be cleaned up safely.

The inline comment in pushRaw now documents that invariant at the call site, and the operator guidance in the docs addresses the observable consequence: non-expiring overflow bodies may outlive the messages they belong to. Pointing operators toward a dedicated, monitored, non-evicting store is the right mitigation — it makes the retention safe rather than pretending the ambiguity doesn't exist.

The finding is resolved as documented. No runtime change is warranted.

@binaryfire
binaryfire merged commit 2756e17 into 0.4 Jul 28, 2026
35 of 38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant