OLS-3794: Operator batch sandbox execution model

.ai/spec/how/project-structure.md

@@ -9,7 +9,7 @@
 | `api/v1alpha1/` | `AgenticRun`, `Agent`, `LLMProvider`, `ApprovalPolicy`, `AgenticRunApproval`, result types, `DerivePhase` | CRD type definitions, phase derivation, CEL markers, deepcopy |
 | `cmd/main.go` | `main`, `scheme` | Operator binary entry point |
 | `cmd/oc-agentic/main.go` | `main` | CLI binary entry point |
-| `controller/agenticrun/` | `AgenticRunReconciler`, `SandboxAgentCaller`, `SandboxManager`, `SandboxLifecycle`, `PodSpecBuilder` | AgenticRun reconciler, unified sandbox management, agent HTTP client, RBAC, results |
+| `controller/agenticrun/` | `AgenticRunReconciler`, `SandboxAgentCaller`, `SandboxManager`, `SandboxLifecycle`, `PodSpecBuilder`, `PodEventHandler` | AgenticRun reconciler, unified sandbox management (SA, RBAC, ConfigMap, pod), pod event handler, timeout loop, results |
 | `controller/console/` | `EnsureAgenticConsole`, `AgenticConsoleConfig` | Console plugin deployment (Deployment, Service, ConfigMap, ConsolePlugin CR) |
 | `controller/sandbox/` | Legacy bootstrap helpers | SA creation inlined into `cmd/main.go` |
 | `pkg/configuration/` | `Config`, `Cache`, `OnConfigMapChange` | ConfigMap-driven config cache (sandbox mode, PodSpec, OTEL, MCP) |
@@ -32,7 +32,7 @@
 - Parses flags (`--namespace`, `--metrics-bind-address`, `--health-probe-bind-address`, `--agentic-console-image`)
 - Creates `configuration.Cache` and registers ConfigMap watcher for `lightspeed-agentic-configuration`
 - Wires `SandboxManager` → `SandboxAgentCaller` → `AgenticRunReconciler` directly (no `controller/setup.go`)
-- Ensures `lightspeed-agent` ServiceAccount unconditionally
+- Ensures `lightspeed-agent` ServiceAccount unconditionally (discovery seed for reader CRBs)
 - Registers console plugin, health/readiness probes, and webhook
 - Starts manager with signal handler
 

.ai/spec/what/audit-logging.md

@@ -116,7 +116,7 @@ Implementation spec for compliance audit logging in the agentic operator. Parent
 
 ### Templog Finalizer
 
-32. When a new AgenticRun CR is created and templog is enabled (read from an environment variable set by the lightspeed-operator), the operator MUST add the finalizer `agentic.openshift.io/templog-cleanup` to the AgenticRun.
+32. When a new AgenticRun CR is created and templog is enabled (read from an environment variable set by the lightspeed-operator), the operator MUST add the finalizer `agentic.openshift.io/templog-cleanup` to the AgenticRun. Both RBAC and templog finalizers are processed in a **single reconcile pass** on deletion (RBAC first, then templog).
 
 33. On AgenticRun deletion, if the `agentic.openshift.io/templog-cleanup` finalizer is present, the operator MUST call the Collector admin API: `DELETE /api/v1/logs?agentic_run_id=<uid>` passing the raw Kubernetes UID (with hyphens; collector normalizes internally). On success, remove the finalizer. On failure, block deletion and requeue with exponential backoff.
 

.ai/spec/what/system-config.md

@@ -17,7 +17,7 @@ Jira tracking: OLS-3018 (base kill switch), OLS-3267 (hardening).
 ### Emergency Suspension (`spec.suspended`)
 
 5. **Activation**: Setting `spec.suspended` to `true` MUST immediately prevent the run reconciler from starting any new workflow steps (analysis, execution, verification, escalation) for any run cluster-wide.
-6. **In-flight termination**: When `spec.suspended` becomes `true`, all non-terminal runs MUST be terminated: sandbox pods MUST be deleted (best-effort), execution RBAC MUST be cleaned up, and the `EmergencyStopped` condition MUST be set on each run.
+6. **In-flight termination**: When `spec.suspended` becomes `true`, all non-terminal runs MUST be terminated: sandbox pods MUST be released via `Agent.ReleaseSandboxes` (which handles pod deletion, SA cleanup, reader CRB subject removal, and execution RBAC cleanup via `SandboxManager.Release`), and the `EmergencyStopped` condition MUST be set on each run.
 7. **EmergencyStopped condition**: The operator MUST set condition type `EmergencyStopped` with status `True`, reason `SystemSuspended`, and message `"Terminated by system kill switch (AgenticOLSConfig.spec.suspended=true)"`.
 8. **EmergencyStopped is terminal — no automatic restart**: `EmergencyStopped` is a terminal phase. Runs in this state MUST NOT resume when `spec.suspended` is set back to `false`. To retry work, the admin creates new runs. This is a safety invariant: the kill switch exists for emergencies where agent behavior is harmful, so automatically restarting the same runs that caused the emergency would re-introduce the exact problem the admin stopped. Resumption MUST always require explicit human action (creating new runs).
 9. **DerivePhase precedence**: `EmergencyStopped=True` MUST be checked **before** all other conditions in `DerivePhase()`. It takes precedence over `Escalated`, `Denied`, and all progress conditions.
@@ -55,7 +55,7 @@ Primary enforcement so new runs never persist while the system is suspended. Spi
 
 12. **Watch and re-queue**: The run reconciler MUST watch `AgenticOLSConfig` and re-queue all non-terminal runs when the CR changes (same pattern as the existing `ApprovalPolicy` watch).
 13. **Reconcile guard**: The suspension check MUST execute after the deletion handler but before finalizer addition, terminal phase routing, approval resolution, and phase dispatch.
-14. **Order of operations on termination**: For each non-terminal run when suspended: (a) release sandbox claims via `Agent.ReleaseSandboxes` (best-effort, log errors), (b) clean up execution RBAC via `cleanupExecutionRBAC` (best-effort, log errors), (c) set `EmergencyStopped` condition, (d) status patch. Errors in (a) or (b) MUST NOT prevent (c) and (d).
+14. **Order of operations on termination**: For each non-terminal run when suspended: (a) release sandboxes via `Agent.ReleaseSandboxes` (best-effort, log errors — `Release` handles pod deletion, SA GC, reader CRB cleanup, and execution RBAC cleanup for each step), (b) set `EmergencyStopped` condition, (c) status patch. Errors in (a) MUST NOT prevent (b) and (c).
 15. **Config fetch failure**: If the `AgenticOLSConfig` CR cannot be fetched and the error is not `NotFound`, the reconciler MUST return the error for retry. `NotFound` MUST be treated as `suspended=false`.
 
 ### Console Visibility

.ai/spec/what/templog.md

@@ -33,7 +33,7 @@ Implementation details for the agentic-operator's role in the templog feature.
 
 ### AgenticRun Finalizer
 
-13. The `agentic.openshift.io/templog-cleanup` (and RBAC cleanup) finalizers are added the first time the controller reconciles any non-deleting AgenticRun — including already-terminal runs — so TTL or manual delete always runs Collector log cleanup.
+13. The `agentic.openshift.io/templog-cleanup` (and RBAC cleanup) finalizers are added the first time the controller reconciles any non-deleting AgenticRun — including already-terminal runs — so TTL or manual delete always runs Collector log cleanup. Both finalizers are processed in a **single reconcile pass** on deletion: RBAC cleanup first (via `ReleaseSandboxes`), then templog cleanup.
 14. When an AgenticRun CR is deleted and the finalizer is present:
     a. The operator calls the Collector admin API: `DELETE /api/v1/logs?agentic_run_id=<uid>` over HTTPS using the CA cert from the ConfigMap. The raw Kubernetes UID (with hyphens) is passed; the collector normalizes internally.
     b. On success, removes the finalizer — CR deletion proceeds.

cmd/main.go

@@ -130,13 +130,12 @@ func main() {
 	auditLogger := agenticrun.NewProductionAuditLogger(zapLogger, telemetryProvider)
 
 	// --- Create sandbox manager and agent caller ---
-	sandboxMgr := agenticrun.NewSandboxManager(mgr.GetClient(), cfgCache, namespace)
+	sandboxMgr := agenticrun.NewSandboxManager(mgr.GetClient(), cfgCache, namespace, auditLogger)
 	agentCaller := &agenticrun.SandboxAgentCaller{
-		Sandbox:       sandboxMgr,
-		K8sClient:     mgr.GetClient(),
-		ClientFactory: agenticrun.NewAgentHTTPClient,
-		Namespace:     namespace,
-		Audit:         auditLogger,
+		Sandbox:   sandboxMgr,
+		K8sClient: mgr.GetClient(),
+		Namespace: namespace,
+		Audit:     auditLogger,
 	}
 
 	// --- Register controllers ---

controller/agenticrun/agent.go

@@ -44,78 +44,42 @@ type EscalationOutput struct {
 	Content string
 }
 
-// AgentCaller abstracts the agent invocation path. The reconciler
-// passes structured data; the implementation decides how to format
-// it for the LLM (text-only prompt vs multimodal with binary
-// attachments). In production this manages sandbox lifecycle + HTTP
-// calls; in tests a stub returns canned results.
-//
-// HTTP implementations POST to /v1/agent/run — a step-agnostic
-// endpoint where all workflow context is in the request payload.
+// AgentCaller abstracts the agent invocation path. Each method
+// launches a sandbox pod with the step's input ConfigMap. The pod
+// runs autonomously, creates the Result CR, and exits. The pod
+// handler watches for completion and patches the step condition.
 type AgentCaller interface {
-	Analyze(ctx context.Context, run *agenticv1alpha1.AgenticRun, step resolvedStep, requestText string, serviceAccount string) (*AnalysisOutput, error)
-	Execute(ctx context.Context, run *agenticv1alpha1.AgenticRun, step resolvedStep, option *agenticv1alpha1.RemediationOption, serviceAccount string) (*ExecutionOutput, error)
-	Verify(ctx context.Context, run *agenticv1alpha1.AgenticRun, step resolvedStep, option *agenticv1alpha1.RemediationOption, exec *ExecutionOutput, serviceAccount string) (*VerificationOutput, error)
-	Escalate(ctx context.Context, run *agenticv1alpha1.AgenticRun, step resolvedStep, requestText string, serviceAccount string) (*EscalationOutput, error)
+	Analyze(ctx context.Context, run *agenticv1alpha1.AgenticRun, step resolvedStep, requestText string) error
+	Execute(ctx context.Context, run *agenticv1alpha1.AgenticRun, step resolvedStep, option *agenticv1alpha1.RemediationOption) error
+	Verify(ctx context.Context, run *agenticv1alpha1.AgenticRun, step resolvedStep, option *agenticv1alpha1.RemediationOption, exec *ExecutionOutput) error
+	Escalate(ctx context.Context, run *agenticv1alpha1.AgenticRun, step resolvedStep, requestText string) error
 	ReleaseSandboxes(ctx context.Context, run *agenticv1alpha1.AgenticRun) error
+	ReleaseSandbox(ctx context.Context, run *agenticv1alpha1.AgenticRun, step string) error
 }
 
-// StubAgentCaller returns canned success results. Wire in a real
-// implementation (sandbox + HTTP) when the agent infrastructure is ready.
+// StubAgentCaller is a no-op implementation for testing.
 type StubAgentCaller struct{}
 
-func (s *StubAgentCaller) Analyze(_ context.Context, _ *agenticv1alpha1.AgenticRun, _ resolvedStep, _ string, _ string) (*AnalysisOutput, error) {
-	actionRequired := true
-	return &AnalysisOutput{
-		Success:        true,
-		ActionRequired: &actionRequired,
-		Options: []agenticv1alpha1.RemediationOption{{
-			Title: "Stub remediation",
-			Diagnosis: agenticv1alpha1.DiagnosisResult{
-				Summary:   "Stub diagnosis",
-				RootCause: "Stub root cause",
-			},
-			RemediationPlan: agenticv1alpha1.RemediationPlan{
-				Description: "Stub remediation plan",
-				Actions:     []agenticv1alpha1.ProposedAction{{Command: "kubectl get pods -n default", Type: "pre-check", Description: "Stub action"}},
-				Reversible:  agenticv1alpha1.ReversibilityReversible,
-			},
-		}},
-	}, nil
+func (s *StubAgentCaller) Analyze(_ context.Context, _ *agenticv1alpha1.AgenticRun, _ resolvedStep, _ string) error {
+	return nil
 }
 
-func (s *StubAgentCaller) Execute(_ context.Context, _ *agenticv1alpha1.AgenticRun, _ resolvedStep, _ *agenticv1alpha1.RemediationOption, _ string) (*ExecutionOutput, error) {
-	return &ExecutionOutput{
-		Success: true,
-		ActionsTaken: []agenticv1alpha1.ExecutionAction{{
-			Type:        "stub",
-			Description: "Stub execution action",
-			Outcome:     agenticv1alpha1.ActionOutcomeSucceeded,
-		}},
-	}, nil
+func (s *StubAgentCaller) Execute(_ context.Context, _ *agenticv1alpha1.AgenticRun, _ resolvedStep, _ *agenticv1alpha1.RemediationOption) error {
+	return nil
 }
 
-func (s *StubAgentCaller) Escalate(_ context.Context, _ *agenticv1alpha1.AgenticRun, _ resolvedStep, _ string, _ string) (*EscalationOutput, error) {
-	return &EscalationOutput{
-		Success: true,
-		Summary: "Stub escalation summary",
-		Content: "Stub escalation content",
-	}, nil
+func (s *StubAgentCaller) Verify(_ context.Context, _ *agenticv1alpha1.AgenticRun, _ resolvedStep, _ *agenticv1alpha1.RemediationOption, _ *ExecutionOutput) error {
+	return nil
+}
+
+func (s *StubAgentCaller) Escalate(_ context.Context, _ *agenticv1alpha1.AgenticRun, _ resolvedStep, _ string) error {
+	return nil
 }
 
 func (s *StubAgentCaller) ReleaseSandboxes(_ context.Context, _ *agenticv1alpha1.AgenticRun) error {
 	return nil
 }
 
-func (s *StubAgentCaller) Verify(_ context.Context, _ *agenticv1alpha1.AgenticRun, _ resolvedStep, _ *agenticv1alpha1.RemediationOption, _ *ExecutionOutput, _ string) (*VerificationOutput, error) {
-	return &VerificationOutput{
-		Success: true,
-		Checks: []agenticv1alpha1.VerifyCheck{{
-			Name:   "stub-check",
-			Source: "stub",
-			Value:  "ok",
-			Result: agenticv1alpha1.CheckResultPassed,
-		}},
-		Summary: "Stub verification passed",
-	}, nil
+func (s *StubAgentCaller) ReleaseSandbox(_ context.Context, _ *agenticv1alpha1.AgenticRun, _ string) error {
+	return nil
 }

controller/agenticrun/approval.go

@@ -31,9 +31,9 @@ func getApprovalPolicy(ctx context.Context, c client.Client) (*agenticv1alpha1.A
 	return policy, nil
 }
 
-func getAgenticRunApproval(ctx context.Context, c client.Client, run *agenticv1alpha1.AgenticRun) (*agenticv1alpha1.AgenticRunApproval, error) {
+func getAgenticRunApproval(ctx context.Context, c client.Client, run *agenticv1alpha1.AgenticRun, namespace string) (*agenticv1alpha1.AgenticRunApproval, error) {
 	approval := &agenticv1alpha1.AgenticRunApproval{}
-	err := c.Get(ctx, types.NamespacedName{Name: run.Name, Namespace: run.Namespace}, approval)
+	err := c.Get(ctx, types.NamespacedName{Name: run.Name, Namespace: namespace}, approval)
 	if err != nil {
 		return nil, err
 	}
@@ -45,8 +45,9 @@ func ensureAgenticRunApproval(
 	c client.Client,
 	run *agenticv1alpha1.AgenticRun,
 	policy *agenticv1alpha1.ApprovalPolicy,
+	namespace string,
 ) (*agenticv1alpha1.AgenticRunApproval, error) {
-	existing, err := getAgenticRunApproval(ctx, c, run)
+	existing, err := getAgenticRunApproval(ctx, c, run, namespace)
 	if err == nil {
 		return existing, nil
 	}
@@ -85,7 +86,7 @@ func ensureAgenticRunApproval(
 	approval := &agenticv1alpha1.AgenticRunApproval{
 		ObjectMeta: metav1.ObjectMeta{
 			Name:      run.Name,
-			Namespace: run.Namespace,
+			Namespace: namespace,
 			OwnerReferences: []metav1.OwnerReference{{
 				APIVersion:         "agentic.openshift.io/v1alpha1",
 				Kind:               "AgenticRun",
@@ -102,7 +103,7 @@ func ensureAgenticRunApproval(
 
 	if err := c.Create(ctx, approval); err != nil {
 		if apierrors.IsAlreadyExists(err) {
-			return getAgenticRunApproval(ctx, c, run)
+			return getAgenticRunApproval(ctx, c, run, namespace)
 		}
 		return nil, fmt.Errorf("%s: %w", ErrCreateAgenticRunApproval, err)
 	}
@@ -181,7 +182,7 @@ func getStageOverrideAgent(approval *agenticv1alpha1.AgenticRunApproval, stage a
 	return ""
 }
 
-func getStageOption(approval *agenticv1alpha1.AgenticRunApproval, _ *agenticv1alpha1.ApprovalPolicy) *int32 {
+func getStageOption(approval *agenticv1alpha1.AgenticRunApproval) *int32 {
 	if approval != nil {
 		for _, s := range approval.Spec.Stages {
 			if s.Type == agenticv1alpha1.ApprovalStageExecution && s.Execution != nil && s.Execution.Option != nil {

controller/agenticrun/approval_test.go

@@ -15,7 +15,7 @@ func TestEnsureAgenticRunApproval_OwnerReference(t *testing.T) {
 	run.UID = "test-uid-123"
 	fc := fake.NewClientBuilder().WithScheme(testScheme()).WithObjects(run).Build()
 
-	approval, err := ensureAgenticRunApproval(context.Background(), fc, run, nil)
+	approval, err := ensureAgenticRunApproval(context.Background(), fc, run, nil, run.Namespace)
 	if err != nil {
 		t.Fatalf("ensureAgenticRunApproval: %v", err)
 	}
@@ -50,7 +50,7 @@ func TestEnsureAgenticRunApproval_AutoApproveStages(t *testing.T) {
 	fc := fake.NewClientBuilder().WithScheme(testScheme()).WithObjects(run).Build()
 	policy := testAutoApprovePolicy()
 
-	approval, err := ensureAgenticRunApproval(context.Background(), fc, run, policy)
+	approval, err := ensureAgenticRunApproval(context.Background(), fc, run, policy, run.Namespace)
 	if err != nil {
 		t.Fatalf("ensureAgenticRunApproval: %v", err)
 	}
@@ -85,7 +85,7 @@ func TestEnsureAgenticRunApproval_AnalysisOnly_SkipsAbsentStages(t *testing.T) {
 	fc := fake.NewClientBuilder().WithScheme(testScheme()).WithObjects(run).Build()
 	policy := testAutoApprovePolicy() // auto-approves Analysis + Verification
 
-	approval, err := ensureAgenticRunApproval(context.Background(), fc, run, policy)
+	approval, err := ensureAgenticRunApproval(context.Background(), fc, run, policy, run.Namespace)
 	if err != nil {
 		t.Fatalf("ensureAgenticRunApproval: %v", err)
 	}
@@ -114,7 +114,7 @@ func TestEnsureAgenticRunApproval_NoPolicy(t *testing.T) {
 	run := testAgenticRun()
 	fc := fake.NewClientBuilder().WithScheme(testScheme()).WithObjects(run).Build()
 
-	approval, err := ensureAgenticRunApproval(context.Background(), fc, run, nil)
+	approval, err := ensureAgenticRunApproval(context.Background(), fc, run, nil, run.Namespace)
 	if err != nil {
 		t.Fatalf("ensureAgenticRunApproval: %v", err)
 	}
@@ -152,7 +152,7 @@ func TestGetStageOption_FromApproval(t *testing.T) {
 			},
 		},
 	}
-	got := getStageOption(approval, nil)
+	got := getStageOption(approval)
 	if got == nil || *got != 2 {
 		t.Errorf("expected option 2 from approval, got %v", got)
 	}
@@ -170,14 +170,14 @@ func TestGetStageOption_ApprovalTakesPrecedence(t *testing.T) {
 			},
 		},
 	}
-	got := getStageOption(approval, nil)
+	got := getStageOption(approval)
 	if got == nil || *got != 2 {
 		t.Errorf("expected option from approval, expected 2, got %v", got)
 	}
 }
 
 func TestGetStageOption_FallbackToZero(t *testing.T) {
-	got := getStageOption(nil, nil)
+	got := getStageOption(nil)
 	if got == nil || *got != 0 {
 		t.Errorf("expected fallback to 0, got %v", got)
 	}
@@ -197,12 +197,12 @@ func TestEnsureAgenticRunApproval_Idempotent(t *testing.T) {
 		},
 	}
 
-	first, err := ensureAgenticRunApproval(context.Background(), fc, run, policy)
+	first, err := ensureAgenticRunApproval(context.Background(), fc, run, policy, run.Namespace)
 	if err != nil {
 		t.Fatalf("first call: %v", err)
 	}
 
-	second, err := ensureAgenticRunApproval(context.Background(), fc, run, policy)
+	second, err := ensureAgenticRunApproval(context.Background(), fc, run, policy, run.Namespace)
 	if err != nil {
 		t.Fatalf("second call: %v", err)
 	}