Skip to content

Determinism Tier 1 follow-up: markdown audit reports embed a wall-clock timestamp #436

Description

@mlieberman85

Follow-up to #418 (Determinism Tier 1), which this gap escaped.

Summary

format_results_markdown in
packages/darnit/src/darnit/tools/audit.py embeds a wall-clock stamp
in every report:

**Generated At:** 2026-09-08T15:47:47.180106Z

Consequence: a markdown audit report can never be byte-for-byte
reproducible, even for an identical commit with an identical verdict
set. Two runs one second apart produce different bytes.

A second, smaller instance: pass_history[].duration_ms is
load-dependent. It reads 0 for fast file-existence checks in practice,
but nothing guarantees that.

How it surfaced

Feature 036's SC-002 required capturing a pre-feature output baseline
and diffing post-feature output against it byte-for-byte. Two
consecutive captures of the same commit did not match. Both fields had
to be scrubbed for the baseline to be stable at all -- see the scrub()
helper in tests/darnit/fixtures/error_class_baseline/capture_baseline.py
and its docstring, which documents both as out of scope for 036 and
points here.

#418's survey covered filesystem iteration order, wall-clock injection
into remediation templates, list ordering, and non-atomic writes. It
did not look at the report formatter, which is why this was missed.

Why it matters

Same argument #418 makes: a compliance tool whose report output drifts
run-to-run undermines the "same commit -> same result" premise that
makes an audit trustworthy. Concretely it also:

  • Blocks any golden-file regression test of markdown output. Any future
    feature wanting to prove "my change is output-neutral" has to
    reinvent the same scrubbing.
  • Makes report diffs noisy in review when an operator commits reports
    to a repo (some fleet operators do).

Scope

Decide between:

  1. Drop the field. Simplest. The information is recoverable from
    the attestation timestamp or the file mtime for anyone who needs it.
  2. Make it opt-in via a formatter argument, default off.
  3. Make it injectable -- feature 036 already established this
    pattern for exactly this problem: RemediationExecutor gained a
    now_provider kwarg (defaulting to datetime.now) so tests can
    pin a fixed clock. Applying the same shape to
    format_results_markdown keeps production behavior while making
    the output testable.

Recommend (3) for consistency with the precedent, or (1) if nobody is
actually consuming the field.

Also decide whether duration_ms should be omitted from serialized
pass_history (it is diagnostic, not evidence) or kept and documented
as non-reproducible.

Non-goals

  • Re-auditing the other formatters. JSON and SARIF do not embed
    timestamps at the per-result level; the attestation predicate has its
    own timestamp, which is legitimate and should stay.
  • Any change to what the audit concludes. This is purely about output
    byte-stability.

Constitution alignment

  • II. Conservative-by-Default: reproducible reports are a
    precondition for the report being usable as evidence.

Related

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions