Skip to content

feat(usage): persist replayable receipts without resetting core counters - #94

Open
dr-hoseyn wants to merge 2 commits into
PasarGuard:devfrom
dr-hoseyn:codex/durable-usage-receipts
Open

dr-hoseyn wants to merge 2 commits into
PasarGuard:devfrom
dr-hoseyn:codex/durable-usage-receipts

Conversation

@dr-hoseyn

@dr-hoseyn dr-hoseyn commented Sep 26, 2026 •

Copy link
Copy Markdown

The current destructive stats RPC resets core counters before the panel can durably store its response. This PR adds non-destructive collection with a durable node-owned receipt: Xray/WireGuard counters are read without reset, and the cumulative baseline plus receipt commit together in a synchronous bbolt transaction before the response is published. Until ACK, all collectors receive the same ID and payload, including after a node process restart.

CollectUsage / AcknowledgeUsage are exposed through authenticated REST and gRPC handlers. Replay and ACK work even with a stopped backend. ACK only clears the matching pending receipt, so a delayed ACK cannot remove a newer sample. After a stream is activated, legacy destructive reads for that stream (including individual-user/outbound reads) fail with HTTP 412 / gRPC FailedPrecondition. Activation survives restart. Empty polls persist their baseline without creating receipts that would grow the panel ledger.

Xray snapshots are fenced by a core-generation ID and lifecycle lock. WireGuard keeps cumulative interface traffic across observed kernel counter resets and preserves its legacy accounting baseline during activation. Invalid counter decreases without a new epoch fail closed. Disk/open/read/transaction errors do not reset the core. Each stream retains one pending payload; historical counter identities are retained to avoid charging a reappearing counter twice.

Measured write reduction

Unchanged operations now roll back the read-only transaction instead of committing bbolt metadata: replaying a pending receipt, a stale/duplicate ACK, and polling unchanged cumulative counters. Activation, changed baselines, new receipts and matching ACKs still require synchronous commits.

A controlled comparison against the preceding PR revision 093903d measured bbolt transaction-ID advancement for 100 calls of each of those four unchanged cases: 400 journal commits before, 0 after (100% fewer unnecessary commits). A subsequent changed counter still persisted and replayed after reopening the journal. This measures journal commits for unchanged calls, not overall throughput or writes for real new usage.

Deployment and durability boundary

Companion panel: PasarGuard/panel#946. Deploy/release this node and the companion Python Bridge (PasarGuard/node_bridge_py#22) first, drain all legacy panel collectors, then activate the updated panel. Do not resume a legacy collector after activation.

USAGE_JOURNAL_PATH defaults to /var/lib/pg-node/usage/receipts.db, covered by the existing compose volume. Keep that file on persistent local storage; deleting/restoring an old journal independently of the panel can lose deduplication history. bbolt file locking serializes access across processes, commits use fsync, and initial directory entries are synchronized on Unix. No asynchronous/NoSync mode is used.

This removes the destructive reset-before-storage gap and provides replay for committed samples. It does not make the underlying Xray/kernel counters durable: unobserved traffic, or a core/host crash that destroys volatile counters before the first journal commit, still requires persistence in the traffic-producing core itself. No end-to-end guarantee against arbitrary source/hardware failure is claimed.

Validation

Current-head upstream Linux CI passed, including the regular suite and WireGuard kernel integration: https://github.com/PasarGuard/node/actions/runs/36256290794 .

  • Full go test -count=1 -p 1 ./... passed locally using a Go overlay solely to supply the Windows Xray path and external regression checks. No tracked tests were modified. Linux-only kernel integration is left to upstream CI.
  • go vet ./... passed.
  • Ten external Go regressions cover journal reopen/replay, delayed/duplicate ACK, snapshot failure/transaction abort, concurrent collectors, activation fencing, rollback/overflow rejection, unavailable storage, cancellation, and cumulative interface resets.
  • Actual production Go REST/gRPC handlers tested over TLS from the Python Bridge, including forced process kill/restart and cross-transport replay: both scenarios passed. Additional tests recovered and acknowledged a persisted receipt with the backend completely absent over both transports.
  • Go → Bridge → panel/SQLite recovery passed for both transports. The controlled 1,111-byte sample was recovered exactly once after staging failure and restart (zero missing or duplicate bytes in these scenarios; not a throughput benchmark).

Related open node PRs #58 and #78 were checked; neither implements a durable usage receipt protocol. No test, documentation, or benchmark files are included.

@coderabbitai

coderabbitai Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: fbf685f2-6e8a-4fbb-a4ed-94375fe30ce8

📥 Commits

Reviewing files that changed from the base of the PR and between 4a9d0cf and 093903d.

⛔ Files ignored due to path filters (3)
  • common/service.pb.go is excluded by !**/*.pb.go
  • common/service_grpc.pb.go is excluded by !**/*.pb.go
  • go.sum is excluded by !**/*.sum
📒 Files selected for processing (15)
  • backend/wireguard/stats.go
  • backend/wireguard/wireguard.go
  • backend/xray/stats.go
  • backend/xray/xray.go
  • common/service.proto
  • config/config.go
  • controller/controller.go
  • controller/rest/service.go
  • controller/rest/stats.go
  • controller/rest/usage.go
  • controller/rpc/stats.go
  • controller/usage.go
  • go.mod
  • pkg/stats/interface_counters.go
  • pkg/usage/store.go

Included review availability: This review used your included allowance. Your plan provides up to 2 included reviews per hour; 1 remain after this review.


Walkthrough

The change adds journal-backed usage receipts for user and outbound statistics. WireGuard and Xray provide non-resetting snapshots with usage epochs. Controller, REST, and RPC methods collect and acknowledge receipts, while stats reset handling uses the journal for supported streams.

Changes

Usage Receipt Collection

Layer / File(s) Summary
Usage API and journal
common/service.proto, config/config.go, controller/controller.go, pkg/usage/store.go, go.mod
Adds usage request, receipt, and acknowledgement messages and RPCs. Configures a bbolt journal path. The store persists stream baselines and pending receipts, replays pending receipts, acknowledges matching receipt IDs, and rejects legacy resets for initialized streams.
Backend usage snapshots
backend/wireguard/*, backend/xray/*, pkg/stats/interface_counters.go
Adds non-resetting snapshots and usage epochs to WireGuard and Xray. WireGuard tracks cumulative interface counters. Xray changes its epoch after a successful core restart.
Controller collection and stats
controller/usage.go, controller/controller.go
Adds controller collection and acknowledgement methods backed by the journal. Routes reset stats through stream-aware handling and maps supported singular reset types to plural streams.
HTTP and RPC endpoints
controller/rest/*, controller/rpc/stats.go
Adds REST and RPC collection and acknowledgement handlers. Existing stats handlers call the controller, and REST usage routes are registered outside backend-availability middleware.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant Panel
  participant NodeService
  participant Controller
  participant UsageStore as usage.Store
  participant UsageBackend as usageBackend
  Panel->>NodeService: CollectUsage request
  NodeService->>Controller: CollectUsage
  Controller->>UsageStore: Collect with snapshot callback
  UsageStore->>Controller: Invoke snapshot callback
  Controller->>UsageBackend: UsageSnapshot
  UsageBackend-->>Controller: Epoch and stats
  Controller-->>UsageStore: Epoch and stats
  UsageStore-->>Controller: UsageReceipt
  Controller-->>NodeService: UsageReceipt
  NodeService-->>Panel: UsageReceipt
  Panel->>NodeService: AcknowledgeUsage request
  NodeService->>Controller: AcknowledgeUsage
  Controller->>UsageStore: Acknowledge receipt ID
  UsageStore-->>Controller: Acknowledgement result
  Controller-->>NodeService: Empty response
  NodeService-->>Panel: Empty response
Loading

Suggested reviewers: m03ed

Merge Risk: ⚪ Minimal · up to 09390

No confirmed issue prevents merging after normal checks.

Security Architecture Review

Security architecture risk: 🟡 Moderate · up to 09390

The receipt design protects against repeated collection and delayed acknowledgements, but its accounting safeguards depend on preserving the journal throughout deployment and rollback. Losing or replacing that file could reopen destructive reads and lose the history needed to avoid duplicate accounting.

Retained concerns

  • Medium · security · inferred: Activation and deduplication are tied to one configured journal file. If deployment or recovery replaces it with an empty or stale file, the activation fence can disappear and legacy destructive reads can resume; subsequent receipts may no longer reconcile with previously delivered usage.
Security review details

Security Blast Radius

  • inferred — A lost activation record affects accounting for its node and stream: a fresh journal can permit destructive reads, while a stale baseline can make later receipts inconsistent with previously delivered usage. The evidence does not establish cross-node access.

Security Findings and Attack Paths

  • inferred — The material failure path is operational rather than an evidenced unauthenticated attack: changing, deleting, or restoring the journal can remove the state on which the new destructive-read fence relies. Ordinary journal open or transaction failure instead returns an error.

Trust Boundaries and Controls

  • observed — The REST API-key check rejects missing, malformed, or mismatched keys before usage handlers run. Collection also restricts the accepted journal stream types; this evidence does not establish a tenant-specific identity model.

Resilience and Maintainability Implications

  • observed — Receipt replay, transactional baseline persistence, exact-ID acknowledgement, and epoch checks protect normal retries, stale ACKs, and counter decreases. They do not themselves recover activation state from a missing journal.

Hardening Proposals

  • proposed — Make journal preservation and restoration an explicit deployment and rollback precondition, and consider a separate durable activation marker or recovery procedure so an absent journal cannot silently re-enable destructive collection.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 22.22% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 9 functions across 13 files. (2 skipped: … Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: persistent, replayable usage receipts collected without resetting core counters.
Full details: Docstring Coverage

Explanation

Docstring coverage is 22.22% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 9 functions across 13 files. (2 skipped: 2 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

A rabbit checks the counter’s flow,
Then saves each receipt safe below.
The epochs mark a fresh new start,
Ack clears the matching, careful part.
Soft paws hop where the stats now go.

Comment @coderabbitai help to get the list of available commands.

@dr-hoseyn

Copy link
Copy Markdown
Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant