Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 12 additions & 0 deletions crates/datadog-agent-trace-sampler/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Copyright 2025-Present Datadog, Inc. https://www.datadoghq.com/
# SPDX-License-Identifier: Apache-2.0

[package]
name = "datadog-agent-trace-sampler"
version = "0.1.0"
edition.workspace = true
license.workspace = true
homepage.workspace = true
repository.workspace = true

[dependencies]
78 changes: 78 additions & 0 deletions crates/datadog-agent-trace-sampler/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# Datadog Agent Trace Sampler

Agent-side trace sampling shared across the serverless agents (bottlecap and the
Serverless Compatibility Layer).

This crate implements the Go trace agent's **error sampler** as a *rescue*
sampler: after an agent decides to drop a trace, the trace gets a second look
via `ErrorsSampler::sample`, and if it contains an error it may be kept. Two
rescue strategies are available, selected by `ErrorSamplerMode`:

- **`AlwaysKeep`** — keep every error chunk unconditionally, no budget/state/
clock; stamps `_dd.errors_sr = 1.0`. Suits low-volume, freeze/thaw
environments like Lambda (bottlecap's default).
- **`RateLimited`** — a dependency-free 1:1 port of the Go agent's
`ScoreSampler` (`ScoreSampler` targeting `ErrorTPS`, from `pkg/trace/sampler/`
in `DataDog/datadog-agent`): keep up to `target_tps` error traces per second,
distributed fairly across distinct trace signatures. Suits continuous
processes that can hit error storms (the Serverless Compatibility Layer's
default).

Per-platform defaults are chosen by each consumer's config layer (e.g. via a
`DD_APM_ERROR_SAMPLER_MODE` env var), not this crate — it has no notion of
which platform it runs on.

## Why dependency-free

The public API takes primitives in (`SpanView` / `TraceView`) and returns a
`SampleDecision` out; it never exposes a protobuf `Span` type. This lets
consumers that pin different `libdatadog` revisions share the crate without
compiling incompatible `pb::Span` types into their build graphs.

## Usage

```rust
use datadog_agent_trace_sampler::{
ErrorSamplerConfig, ErrorsSampler, SampleDecision, SpanView, TraceView,
};

// `ErrorSamplerConfig::default()` uses `ErrorSamplerMode::RateLimited`
// (Go-parity default); set `mode: ErrorSamplerMode::AlwaysKeep` for the
// simpler unconditional-rescue strategy.
let mut sampler = ErrorsSampler::new(ErrorSamplerConfig::default());

let spans = [SpanView {
service: "web",
name: "web.request",
resource: "GET /",
error: true,
http_status_code: Some("500"),
error_type: None,
}];
let trace = TraceView {
env: "prod",
trace_id: 0xdead_beef,
root_index: 0,
root_global_sample_rate: 1.0,
spans: &spans,
};

// `now_unix_secs` drives the rolling window and is passed in (not read from a
// clock) so the crate stays dependency-free and deterministically testable.
match sampler.sample(1_700_000_000, &trace) {
SampleDecision::Keep { errors_sr } => {
// caller stamps `_dd.errors_sr = errors_sr` on the root span
}
SampleDecision::Drop => {
// the pending agent-side drop proceeds
}
}
```

`ErrorsSampler::sample` takes `&mut self` (the rolling buffer and rate map mutate
on every call). Consumers that share one sampler across threads wrap it in
`Arc<Mutex<ErrorsSampler>>`.

Setting `target_tps` to `0.0` (or negative) disables the sampler in both modes:
every candidate returns `SampleDecision::Drop` (i.e. nothing is rescued).
`extra_sample_rate` is only meaningful in `RateLimited` mode.
154 changes: 154 additions & 0 deletions crates/datadog-agent-trace-sampler/src/lib.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
// Copyright 2025-Present Datadog, Inc. https://www.datadoghq.com/
// SPDX-License-Identifier: Apache-2.0

#![cfg_attr(not(test), deny(clippy::panic))]
#![cfg_attr(not(test), deny(clippy::unwrap_used))]
#![cfg_attr(not(test), deny(clippy::expect_used))]
#![cfg_attr(not(test), deny(clippy::todo))]
#![cfg_attr(not(test), deny(clippy::unimplemented))]

//! Agent-side trace sampling shared across serverless agents (bottlecap and the
//! Serverless Compatibility Layer).
//!
//! This crate implements a dual-mode *rescue* sampler for the Go trace agent's
//! error sampler (`ScoreSampler` targeting `ErrorTPS`): after an agent decides to
//! drop a trace, the trace gets a second look via [`ErrorsSampler::sample`], and
//! if it contains an error it may be rescued. Two strategies are available,
//! selected by [`ErrorSamplerMode`]:
//!
//! - [`ErrorSamplerMode::AlwaysKeep`]: keep every error chunk unconditionally, no
//! budget/state/clock; stamps `_dd.errors_sr = 1.0`. Suits low-volume,
//! freeze/thaw environments like Lambda (bottlecap's default).
//! - [`ErrorSamplerMode::RateLimited`]: a dependency-free 1:1 port of the Go
//! agent's `ScoreSampler`, keeping up to `target_tps` error traces per second
//! distributed fairly across distinct trace signatures. Suits continuous
//! processes that can hit error storms (the Serverless Compatibility Layer's
//! default).
//!
//! Per-platform defaults are chosen by each consumer's config layer, not this
//! crate.
//!
//! The public API takes primitives in and returns a decision out (no protobuf
//! `Span` type), so consumers pinning different `libdatadog` revisions can share
//! it without compiling incompatible span types into their build graphs.
//!
//! # Example
//!
//! ```
//! use datadog_agent_trace_sampler::{
//! ErrorSamplerConfig, ErrorsSampler, SampleDecision, SpanView, TraceView,
//! };
//!
//! let mut sampler = ErrorsSampler::new(ErrorSamplerConfig::default());
//! let spans = [SpanView {
//! service: "web",
//! name: "web.request",
//! resource: "GET /",
//! error: true,
//! http_status_code: Some("500"),
//! error_type: None,
//! }];
//! let trace = TraceView {
//! env: "prod",
//! trace_id: 0xdead_beef,
//! root_index: 0,
//! root_global_sample_rate: 1.0,
//! spans: &spans,
//! };
//! match sampler.sample(/* now_unix_secs */ 1_700_000_000, &trace) {
//! SampleDecision::Keep { errors_sr } => {
//! // caller stamps `_dd.errors_sr = errors_sr` on the root span
//! let _ = errors_sr;
//! }
//! SampleDecision::Drop => { /* the pending drop proceeds */ }
//! }
//! ```

mod score_sampler;
mod signature;

pub use score_sampler::ErrorsSampler;
pub use signature::Signature;

/// A read-only view of a single span, holding only the fields the sampler needs.
///
/// `http_status_code` and `error_type` come from the span's `meta` map keys
/// `http.status_code` and `error.type` respectively.
#[derive(Debug, Clone, Copy)]
pub struct SpanView<'a> {
pub service: &'a str,
pub name: &'a str,
pub resource: &'a str,
/// Whether the span is an error. The Go agent's span `Error` field is an
/// `int32`, and its v0 signature hash folds the raw low byte into the hash,
/// so a span with `error > 1` would hash differently there. Normalizing to
/// a bool matches Go's newer `computeSpanHashV1`; in practice tracers only
/// ever emit 0 or 1, so the two agree on real traffic.
pub error: bool,
pub http_status_code: Option<&'a str>,
pub error_type: Option<&'a str>,
}

/// A read-only view of a trace chunk to be sampled.
#[derive(Debug, Clone, Copy)]
pub struct TraceView<'a> {
pub env: &'a str,
pub trace_id: u64,
/// Index of the root span within `spans`.
pub root_index: usize,
/// The root span's global sample rate (`metrics["_sample_rate"]`), default 1.0.
/// Callers may pass the raw wire value: the error sampler falls back to 1.0 for
/// anything non-finite or outside `(0, 1]`.
pub root_global_sample_rate: f64,
pub spans: &'a [SpanView<'a>],
}

/// Selects which rescue strategy [`ErrorsSampler`] uses.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ErrorSamplerMode {
/// Keep every error chunk. No budget, no rolling window, no clock. Stamps
/// `_dd.errors_sr = 1.0`. `target_tps` still gates whether the sampler is
/// disabled (`<= 0.0` drops everything, matching `RateLimited`);
/// `extra_sample_rate` is unused.
AlwaysKeep,
/// Full 1:1 Go `ScoreSampler` port: keep up to `target_tps` error traces
/// per second, distributed fairly across distinct trace signatures. Stamps
/// the computed `errors_sr`.
RateLimited,
}

/// Configuration for the error sampler.
#[derive(Debug, Clone, Copy)]
pub struct ErrorSamplerConfig {
/// Which rescue strategy to use.
pub mode: ErrorSamplerMode,
/// Target error traces per second (`ErrorTPS`). `0.0` (or negative)
/// disables the sampler in both modes (every candidate is dropped, i.e.
/// never rescued). Only meaningful for rate computation in `RateLimited`.
pub target_tps: f64,
/// Extra raw sampling rate applied on top of the computed rate. Only
/// meaningful in `RateLimited`.
pub extra_sample_rate: f64,
}

impl Default for ErrorSamplerConfig {
/// Matches the Go agent defaults: `ErrorTPS = 10`, `ExtraSampleRate = 1.0`,
/// mode `RateLimited`.
fn default() -> Self {
ErrorSamplerConfig {
mode: ErrorSamplerMode::RateLimited,
target_tps: 10.0,
extra_sample_rate: 1.0,
}
}
}

/// The outcome of sampling a trace.
#[derive(Debug, PartialEq)]
pub enum SampleDecision {
/// Keep (rescue) the trace. The caller should stamp `_dd.errors_sr` on the
/// root span with `errors_sr`.
Keep { errors_sr: f64 },
/// Drop the trace (do not rescue it); the pending agent-side drop proceeds.
Drop,
}
Loading
Loading