Skip to content

Rate-limit PEXPIRE on entity event streams - #8032

Open
nicholaspcr wants to merge 2 commits into
v3.37from
fix/events-redis-expire-limiter
Open

nicholaspcr wants to merge 2 commits into
v3.37from
fix/events-redis-expire-limiter

Conversation

@nicholaspcr

Copy link
Copy Markdown
Contributor

Summary

Backport of https://github.com/TheThingsIndustries/lorawan-stack/pull/4960

The events Redis store issued a PEXPIRE per entity event stream per published event, dominating write load on the events Redis. Streams are now refreshed at most once per events.redis.store.entity-ttl / 2. Retention is unchanged.

Changes

  • New pkg/events/redis/expire_limiter.go: a fixed-size, pointer-free table that allows one PEXPIRE per stream per interval. Anything uncertain allows a PEXPIRE, never skips one.
  • publish now gates PExpire on the limiter, built in NewPubSub.
  • New events.redis.store.expire-limiter-size: streams tracked, 8 bytes each, defaulting to 4194304 (32 MiB). Negative refreshes the TTL on every event.

Testing

Steps
  1. go test -race -run TestExpireLimiter ./pkg/events/redis/
  2. go test -run XXX -bench BenchmarkExpireLimiter ./pkg/events/redis/
  3. tools/bin/mage dev:dbStart && TEST_REDIS=1 go test ./pkg/events/redis/
  4. golangci-lint run ./pkg/events/redis/...
Results

Limiter tests pass under -race; allow is 10.28 ns/op, zero allocations (Apple M4); lint reports no new issues.

Regressions

Affects every component publishing events with events.redis.store.enable set, most visibly the Application Server and Network Server. Covered by the existing suite plus limiter tests for counter wrap, eviction, concurrency, and refresh landing inside the jittered TTL.

Checklist

  • Scope: The referenced issue is addressed, there are no unrelated changes.
  • Compatibility: The changes are backwards compatible with existing API, storage, configuration and CLI, according to the compatibility commitments in README.md for the chosen target branch.
  • Documentation: Relevant documentation is added or updated.
  • Testing: The steps/process to test this feature are clearly explained including testing for regressions.
  • Infrastructure: If infrastructural changes (e.g., new RPC, configuration) are needed, a separate issue is created in the infrastructural repositories.
  • Changelog: Significant features, behavior changes, deprecations and fixes are added to CHANGELOG.md.
  • Commits: Commit messages follow guidelines in CONTRIBUTING.md, there are no fixup commits left.

@nicholaspcr nicholaspcr self-assigned this Sep 18, 2026
@github-actions github-actions Bot added compat/db This could affect Database compatibility compat/config This could affect Configuration compatibility labels Sep 18, 2026
@nicholaspcr
nicholaspcr marked this pull request as ready for review September 21, 2026 13:11
@nicholaspcr
nicholaspcr requested a review from a team as a code owner September 21, 2026 13:11

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

compat/config This could affect Configuration compatibility compat/db This could affect Database compatibility

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant