testkit: type cgroup memory.max refusal as ErrLimitUnreachable

[team-humaki]

Fixes #64.

docker update --memory-swap=N with swap disabled can fail when unreclaimable RSS already exceeds the cap (runc: failed to write memory.max). That is the injector asking for an impossible limit, not a product fault.

• DockerTarget.SetMemoryLimit returns ErrLimitUnreachable on that refusal
• orchestrator records fault_skipped_limit_unreachable instead of fault_apply_failed
• tests for the matcher and the soak event

[postgresql007]

Thanks for picking this up — the shape is right, and fault_skipped_limit_unreachable sitting alongside fault_skipped_cell_down is exactly what #64 was asking for. Builds clean here, and go test ./internal/testkit/inject/... ./internal/testkit/validate/... is green.

One thing needs changing before this can merge: isCgroupLimitUnreachable keys on the path, not on the reason.

if strings.Contains(s, "memory.max") {
    return true
}

Any daemon output that mentions memory.max becomes ErrLimitUnreachable, whatever actually went wrong. Dropping the real runc strings through it:

daemon output	got	want
write /sys/fs/cgroup/docker/<id>/memory.max: permission denied	true	false
write /sys/fs/cgroup/docker/<id>/memory.max: operation not permitted	true	false
cannot open /sys/fs/cgroup/docker/<id>/memory.max: no such file or directory	true	false

The {out: "permission denied", want: false} case in TestIsCgroupLimitUnreachable does pass — but only because that bare string contains no memory.max. The message docker actually returns does contain it, so the assertion you intended isn't being exercised.

Why this matters beyond tidiness. On a host that cannot do cgroup writes at all — rootless docker, a runner without cgroup delegation, cgroup v1 — every cgroup_squeeze becomes a skip. fault_skipped_* is not counted into FaultStats, and no pass criterion asserts a minimum number of applied faults, so the soak reports PASS having never applied the fault it exists to exercise. That is the failure mode v1.4 shipped with the WAL-stream sidecar: the measurement quietly stopped and the report still said green. Typing the refusal is the right call; it just has to stay narrow enough that a host which can never squeeze still pages someone.

Suggested shape — require the errno that actually means "the kernel could not reclaim to this cap", and let everything else stay loud:

func isCgroupLimitUnreachable(out string) bool {
	s := strings.ToLower(out)
	if !strings.Contains(s, "memory.max") &&
		!(strings.Contains(s, "failed to write") && strings.Contains(s, "cgroup")) {
		return false
	}
	// EINVAL / EBUSY: the write reached the kernel and the kernel
	// refused it, because the live unreclaimable footprint is already
	// above the cap. That is the injector asking for the impossible,
	// and the case this sentinel is for.
	//
	// EACCES / EPERM / ENOENT are a different animal: this host cannot
	// squeeze at all. Skipping those silently would hide every fault in
	// the run rather than the 0.7% timing race.
	return strings.Contains(s, "invalid argument") ||
		strings.Contains(s, "device or resource busy") ||
		strings.Contains(s, "cannot allocate memory")
}

plus a case carrying the full text, so the intent of the existing one is actually pinned:

{
	out:  `runc did not terminate successfully: failed to write "33554432": write /sys/fs/cgroup/docker/abc/memory.max: permission denied`,
	want: false,
},

Everything else in the PR — the sentinel, the orchestrator arm, TestRun_FaultLimitUnreachable_RecordsSkipNotFailure — looks good to me as-is.

Separately, and not something this PR needs to solve: fault_skipped_* events being invisible to FaultStats and to the pass criteria is its own gap, since a run that skipped every squeeze is currently indistinguishable from one that applied them all. Worth its own issue.