feat: Goe rewrite#5
Open
jacobseunglee wants to merge 19 commits into
Open
Conversation
Implements the GoE v2 foundation: Pydantic v2 models for the full entity graph + procedure DSL, a step-by-step procedure executor with interpolation/ assertions/output capture, and a thin TestEnvironment adapter over v1 TestEnvironmentTool. 74 tests passing (3 browser/listen xfailed with documented root causes). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…g, and SUID privesc Signed-off-by: Akshay Rohatgi <52616034+Akshay-Rohatgi@users.noreply.github.com>
…ations implemented and tests for attack procedures added Signed-off-by: Akshay Rohatgi <52616034+Akshay-Rohatgi@users.noreply.github.com>
…ng working: detached background processes, attacker container reset on retry, chromium browser installed via PPA Signed-off-by: Akshay Rohatgi <52616034+Akshay-Rohatgi@users.noreply.github.com>
…self-review to address commonly seen custom app development pitfalls Signed-off-by: Akshay Rohatgi <52616034+Akshay-Rohatgi@users.noreply.github.com>
…g tests and visualizers Signed-off-by: Akshay Rohatgi <52616034+Akshay-Rohatgi@users.noreply.github.com>
* feat: model change * delete game_of_everything/goe/jacobtest.yaml * fix: workflow * fix: switch off plan determined runtime
Adds the v2 evaluation suite (goe/eval), metrics instrumentation (goe/metrics), single-system orchestration/packaging (goe/flow, goe/packaging), workflow artifacts (goe/artifacts), and their tests and fixtures. Correctness fixes from code review: - build.py: a non-zero deploy exit no longer falls through to a possible PASS; it routes into the retry loop as a design_flaw. - runtimes/registry.py: create parent dirs for nested source-file paths (set -e no longer aborts); raise on unknown db_type. - eval/llm_judge.py: print_judge_result tolerates missing keys. - eval/golden.py: edge coverage requires a real connecting edge, not independent provides/requires matches. - eval/runner.py: capture real run start time (durations were ~0). - flow/orchestrator.py + checkpoint.py: persist and restore failure_category on the resume path. Cleanups: - metrics/collector.py: drop dead capture_artifacts ternary. - bedrock.py: cache the bedrock-runtime client per region/creds instead of rebuilding it on every call. - runtimes: consolidate per-runtime knowledge into the template YAMLs (target_image, deps_install_template, pre_start); deploy() is now table-driven and _RUNTIME_IMAGES is removed. Co-authored-by: Jacob Lee <66867022+jacobseunglee@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Combines planner improvements (atom catalog + grading step) with Phase 4
increment 1 (multi-entity chain test + multi-system packaging).
## Planner Improvements
Fixes planner producing wrong runtimes (apache_php for SSH) and invented
atoms by grounding all prompts in actual atom inventory with few-shot examples.
**New pipeline structure:**
- Step 1: plan_entities → includes runtime + atoms in stubs
- Step 1.5: grade_stubs → LLM validates/corrects stubs (5-check rubric)
- Step 2: specify_entities → adds edges with validated runtime/atoms
**Atom catalog** (goe/planner/_atom_catalog.py):
- Parses 13 web vuln atoms from atoms/web_vulnerabilities/*.md
- Extracts descriptions, compatible runtimes (from code examples), capabilities
- Provides rich markdown table for prompt injection
**Prompts rewritten** (all 4 steps + grading):
- design_systems: port-to-runtime mapping + 2 examples
- plan_entities: atom catalog injection, runtime selection rules, 2 examples
- grade_stubs (new): 5-check rubric (atom exists, runtime matches, web vs
system, single responsibility, chain logic)
- specify_entities: rich atom table, runtime affinity rules, edge consistency
- connect_edges: edge type selection guide, 2 examples
**Fixes:**
- resolve.py: match structural port to exposed ports (not always first)
- topology_environment.py: create containers on network directly (not none→connect)
## Phase 4: Multi-System Orchestration
**Chain test** (goe/flow/chain_test.py):
- L3 validation after all entities pass L2
- Gates overall run (FAILED → RunResult.success=False, CLI exits non-zero)
- TopologyEnvironment: one ubuntu:22.04 container per system + shared Kali attacker
- Chain attacker agent (Opus) synthesizes end-to-end procedure
- Retries up to 2× on failure
**Multi-system packaging** (goe/packaging/packager.py):
- Single-system: unchanged (deploy.sh + playbook.yaml)
- Multi-system: per-system deploy scripts + docker-compose.yml + chain_playbook.yaml
- Port collision detection scoped per system_id
**Cross-system addressing** (goe/executor/interpolation.py):
- ${system.<system_id>.host} / ${system.<system_id>.port}
- Existing ${target_host}, ${edge.*}, ${steps.*} unchanged
**Orchestrator** (goe/flow/orchestrator.py):
- Runs chain test when len(built) > 1
- Chain test result gates success
- Persists chain_test in checkpoint
**CLI** (goe/flow/__main__.py):
- goe flow test <output_dir> — replays packaged runs
- Auto-detects chain_playbook.yaml for multi-system replay
## Verification
End-to-end test: "SQLi → SSH" scenario that was failing before:
- ✓ SSH entity now has runtime=ubuntu (was apache_php)
- ✓ Both entities build successfully
- ✓ L3 chain test completes (synthesizes SQLi→SSH attack chain)
- ✓ Output includes chain_playbook.yaml with cross-system addressing
Co-Authored-By: Jacob Lee <66867022+jacobseunglee@users.noreply.github.com>
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Signed-off-by: Akshay Rohatgi <52616034+Akshay-Rohatgi@users.noreply.github.com>
- Mark Phases 0-4 complete - Document planner grounding (5-step pipeline with atom catalog + grading) - Add chain_attacker to LLM calls table - Add Phase 5 section with next steps and current gaps vs v1 Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
… progressive env
Phase 4.5 deliverables:
- Killchain planning step (plan_killchain.py) guides entity ordering
- Fan-out edge support (fan_out flag on Edge model, validator respects it)
- Atom content injection into construction crew (developer/attacker use Logic Requirements, Synthesis Guidance, Testing Guidance)
- Progressive environment integration into orchestrator (multi-ubuntu systems)
- Eval/metrics system fixes (planning eval, per-entity details, Pydantic validation)
Files added:
- goe/planner/plan_killchain.py + prompt
- goe/construction_crew/atoms.py (section extraction utilities)
- goe/container/progressive.py + bootstrap.py
- goe/eval/, goe/metrics/, goe/artifacts/ (full eval system)
- goe/packaging/grader.py
- tests/test_atom_integration.py, test_atoms.py
Files modified:
- goe/models/edge.py (fan_out field)
- goe/graph/validator.py (fan-out validation)
- goe/construction_crew/{developer,attacker,engineer}.py (atom injection)
- goe/flow/orchestrator.py (progressive env integration)
- goe/planner/pipeline.py (killchain step)
- CLAUDE.md (Phase 4 completion + Phase 5 roadmap)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Phase 1: Services as First-Class Workloads
Created a service deployment layer with proven, deterministic bash recipes
that replace fragile LLM-generated service deployment. Services now have
proper readiness gates, error handling, and config override support.
New components:
- goe/services/registry.py: ServiceRegistry class
- goe/services/recipes/*.yaml: 10 service recipes
- mysql: MariaDB with readiness wait + password config
- openssh: SSH server with permit_root_login + password_auth config
- postgres: PostgreSQL with password config
- mongodb: MongoDB NoSQL database
- redis: Redis with requirepass config
- nginx: Nginx reverse proxy
- samba: Samba file shares
- ftp: vsftpd FTP server
- apache: Apache HTTP server
- memcached: Memcached distributed cache
Model changes:
- goe/models/system.py: Added ServiceSpec model
- System.services now list[ServiceSpec] (backward compat with list[str])
Tests:
- tests/test_services.py: 8 tests verifying registry, recipes, script generation
Key features:
- Readiness gates: all services wait for actual startup (no blind sleep)
- Restart after snapshot: services marked for re-start after docker commit
- Planner-configurable: weak passwords/misconfigs set by planner
- Error handling: explicit FATAL messages, no silent failures
This replaces the current RuntimeRegistry DB setup code that does:
apt-get install -y mariadb-server
service mariadb start # fails in Docker, no readiness wait
With:
mysqld_safe &
for i in 1..30; do mysqladmin ping && break; sleep 1; done
mysqladmin ping || { echo FATAL; exit 1; }
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ntegration tests Phase 1 Complete: Services as First-Class Workloads Created deterministic service deployment layer with proven bash recipes that replace fragile LLM-generated service deployment. ## Services (7 recipes, all tested in Docker) - MySQL (MariaDB) - with root_password config - OpenSSH - with permit_root_login + password_auth config - Redis - with requirepass config + password-aware healthcheck - Nginx - reverse proxy - Apache - HTTP server - Samba - file shares (SMB/CIFS) - FTP (vsftpd) - file transfer ## Key Features - Readiness gates: all services wait for actual startup (no blind sleeps) - Error handling: explicit FATAL messages + exit 1 on failures - Config vars: planner can set passwords, permissions as part of vuln scenarios - Restart support: services marked for re-start after docker commit - Netcat/curl deps: properly declared for healthchecks ## Test Results (10/10 passed, 367s runtime) ✅ test_mysql_deployment_in_container ✅ test_openssh_deployment_in_container ✅ test_redis_deployment_in_container ✅ test_nginx_deployment_in_container ✅ test_samba_deployment_in_container ✅ test_ftp_deployment_in_container ✅ test_apache_deployment_in_container ✅ test_multi_service_deployment (MySQL + Redis together) ✅ test_restart_script_after_commit (snapshot/restore cycle) ✅ test_service_failure_detection Tests verify actual Docker deployment, not just script generation. ## Model Changes - goe/models/system.py: Added ServiceSpec(id, config) model - System.services: list[str] → list[ServiceSpec] (backward compat) ## Files Created - goe/services/registry.py: ServiceRegistry class - goe/services/recipes/*.yaml: 7 service recipes - tests/test_services.py: 8 unit tests - tests/test_services_syntax.py: 10 validation tests - tests/test_services_integration.py: 10 Docker integration tests Services not included (require complex setup): - PostgreSQL: complex startup in Docker - MongoDB: needs external MongoDB repo - Memcached: deferred Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Akshay Rohatgi <52616034+Akshay-Rohatgi@users.noreply.github.com>
Signed-off-by: root <root@ip-172-31-19-142.us-west-1.compute.internal>
…agents Signed-off-by: Akshay Rohatgi <52616034+Akshay-Rohatgi@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.