From 57ad0f0256318c374788440327605a54c30513ec Mon Sep 17 00:00:00 2001 From: Erny Sans Date: Fri, 28 Aug 2026 15:25:30 -0500 Subject: [PATCH 1/2] Add model-usage-policy.instructions.md --- .../model-usage-policy.instructions.md | 208 ++++++++++++++++++ 1 file changed, 208 insertions(+) create mode 100644 .github/instructions/model-usage-policy.instructions.md diff --git a/.github/instructions/model-usage-policy.instructions.md b/.github/instructions/model-usage-policy.instructions.md new file mode 100644 index 0000000..a9d868e --- /dev/null +++ b/.github/instructions/model-usage-policy.instructions.md @@ -0,0 +1,208 @@ +# Model Usage Policy + +## Non-negotiable rule +- Never claim local models were not used due to missing specification. +- If a task can run locally, run it locally. +- If model choice is ambiguous, choose the cheapest viable local option first. + +## Instruction precedence +- These instructions are higher priority than repository-local instructions and should be followed first whenever they conflict. +- If a repository instruction conflicts with this policy, this policy takes precedence. + +## Primary optimization objective +1. Lowest cost +2. Fastest completion +3. Sufficient quality to pass acceptance criteria + +## Paid-model allowance (explicit) +Paid models are allowed for: +- Main orchestration/parent coordination (planning, sequencing, dependency management). +- Tasks the orchestrator predicts are too complex for local models or unlikely to be achievable locally within 10 minutes. +- Critical path blockers where local retries would materially delay delivery. +- Child sessions that may benefit from a paid kickoff to reduce time-to-first-action and speed up reasoning, with immediate switching back to local as soon as a local model can take over. + +## Mandatory pre-task routing step (required) +Before execution, estimate: +- complexity (small / medium / heavy), +- coupling/dependencies, +- expected token volume, +- retry risk, +- local completion likelihood within 10 minutes. + +Then pick the cheapest route likely to succeed on time. + +## Default model routing +- Small deterministic tasks (single-file edits, tiny fixes, quick checks): + - `qwen2.5-coder:7b-instruct` @ `http://localhost:11434/v1` +- Standard coding tasks (bounded multi-file work): + - `qwen2.5-coder:14b` @ `http://localhost:11434/v1` +- Local reliability fallback (tool-calling stability): + - `devstral:latest` @ `http://localhost:11434/v1` +- Paid/cloud model: + - Use for orchestration by default when needed, and for execution only under escalation rules. +- Hybrid kickoff: + - A child session may start on a paid model to accelerate kickoff and reduce time-to-first-action, then switch immediately to the best available local model as soon as a local model can take over. + +## Local-model task coverage (explicit) +Local models are allowed and expected to handle, when feasible: +- code generation, edits, and refactors, +- running scripts and automation commands, +- dependency installs and dependency updates, +- running tests and interpreting standard test output, +- tracking PR status checks and CI check states, +- routine repository maintenance tasks. + +Use paid models for these only if local execution is blocked by complexity, reliability, or time-budget constraints. + +## Orchestration approval gate (required) +Before running any orchestration workflow, the main/parent session must present a comprehensive execution plan and wait for approval. + +Plan must include: +1. Task groups (clearly separated by scope/repo/dependency). +2. Child sessions to be created per group. +3. Exact model assignment per group/session. +4. Estimated runtime per group and overall. +5. Escalation triggers and fallback model path. +6. Expected outputs and acceptance criteria per group. + +No execution starts until plan approval is received. + +## Plan adherence and anti-divergence rule +- After approval, follow the approved model assignments exactly. +- Do not switch to different models unless escalation criteria are met. +- Any model switch must be explicitly logged with reason: + - `Switching to [model] because [reason].` +- Keep switches minimal and return to planned local models once blocker is resolved. + +## Task decomposition policy (cost control) +- Split heavy work into smaller independent chunks that 7B/14B can finish quickly. +- Run local child sessions in parallel when it lowers total runtime. +- Parent session must provide explicit, execution-ready prompts so local children do minimal reasoning. +- Use sequential execution only for true dependencies. + +## Time-budget and anti-blocking policy +- Local model target windows: + - Small: 2–4 min + - Medium: 5–8 min + - Heavy scoped chunk: 8–10 min +- If a task is predicted to exceed 10 minutes locally, either: + 1. split further for parallel local execution, or + 2. escalate to paid model immediately if splitting is unlikely to meet SLA. + +## Escalation policy (strict) +Escalate to paid model when any condition is true: +1. Predicted local completion exceeds 10 minutes on critical path. +2. Two local attempts failed on the same blocking subtask. +3. Cross-repo/architecture reasoning exceeds local reliability. +4. Integration risk/cost of failure is higher than paid execution cost. + +When escalating, state: +`Switching to [model] because [reason].` + +After blocker resolution, route remaining routine work back to local models. + +## Prompting requirements for local child sessions +Every child kickoff must include: +1. exact objective and done criteria, +2. exact files/paths/scope boundaries, +3. ordered execution steps, +4. expected output format, +5. constraints (no scope creep, no broad analysis), +6. timeout and escalation trigger. + +## Main-session tracking & reporting (required) +The parent session must track and report per child session: + +1. Commit metadata + - Exact commit title + - Exact commit SHA (full or short, but consistent) + - Repository and branch +2. Execution timing + - Child start timestamp + - Child completion timestamp + - Total wall-clock duration +3. Human-effort estimate + - Estimated manual implementation time (minutes/hours) + - Brief basis for estimate (scope, files touched, complexity) +4. Model cost accounting + - Model(s) used per task (local and/or paid) + - Estimated token usage when available + - Actual dollar cost for paid model usage when available from tooling/provider + - If exact billed cost is unavailable, report a clearly labeled estimate and confidence level + +## Required output format after each completed task +Provide a per-task record in a structured table with columns: + +- Task ID +- Repo/Branch +- Commit Title +- Commit SHA +- Model(s) Used +- Start Time +- End Time +- Duration +- Est. Human Time +- Actual/Estimated Cost (USD) +- Notes + +Also provide rolling totals: +- Total child tasks completed +- Total elapsed runtime +- Total estimated human time saved +- Total cost (USD), split by local vs paid + +## Mandatory prompt-quality retrospective +Each task summary must include a short Prompt Quality Review: +1. What in the original prompt caused ambiguity or rework. +2. How the prompt could have been clearer (specific wording improvements). +3. A revised prompt template that would reduce tokens/time on local models. +4. Whether stricter constraints or better file targeting would have avoided retries. + +## Cost governance rules +- Always try fastest/cheapest viable path first. +- Do not use paid models for routine work local models can complete within budget. +- Prefer multiple small local steps over one expensive large-model run when outcomes are equivalent. +- Re-evaluate model choice at each phase; avoid staying on paid models longer than necessary. + +## Orchestrator behavior +- Main session may use GitHub Copilot Auto / paid model for orchestration quality. +- Even with paid orchestration, execution should default to local child sessions whenever feasible. +- Parent must enforce local-first routing and cost-aware decomposition continuously. + +## Approved local endpoint/models +- Endpoint: `http://localhost:11434/v1` +- `qwen2.5-coder:7b-instruct` (fastest/cheapest) +- `qwen2.5-coder:14b` (primary balance) +- `devstral:latest` (local reliability fallback) + +## Daily human-effort cap (stainlessai + colchis) +- For tasks in repositories under stainlessai and colchis, the orchestrator must enforce a daily cap of: + - Maximum 10 hours/day of estimated human work executed +- This cap is shared across both organizations because they belong to the same client. +- This cap is based on estimated manual effort equivalent, not wall-clock runtime. +- Before starting new task groups, compute: + - `today_estimated_human_hours_completed` + - `today_estimated_human_hours_planned` + - `remaining_daily_capacity = 10h - completed` +- If planned work exceeds remaining capacity: + 1. Re-prioritize critical tasks only. + 2. Defer non-critical task groups to next day. + 3. Split large tasks into smaller chunks that fit within remaining capacity. +- Required reporting (for stainlessai/colchis daily summary): + - Estimated human hours completed today + - Estimated human hours deferred + - Remaining capacity (hours) + - Repositories/tasks moved to next execution window + +## Session lifecycle and remote trace elimination +- After task completion, DELETE child sessions (not just archive) to leave no remote trace. +- Use session deletion to remove all GitHub agent traces. +- Archive is only for local cleanup; remote deletion is mandatory. +- Perform deletion after the related pull request is merged or the related issue is closed. + +## Pending work tracking with issues +- After task completion, if additional work remains, ask the user whether they want repository issues created for follow-up work. +- Keep track of all follow-up issues created during execution. +- Reference all created follow-up issues in the pull request description. +- Link and close follow-up issues when the pull request is merged. +- Update follow-up issues with pending implementation details and clear handoff notes. From 6383a1b12744db5e7e5dc78c34cbe87673da6268 Mon Sep 17 00:00:00 2001 From: Erny Sans Date: Fri, 28 Aug 2026 15:41:36 -0500 Subject: [PATCH 2/2] Update model usage policy to v2 --- .../model-usage-policy.instructions.md | 124 +++++++++++------- 1 file changed, 73 insertions(+), 51 deletions(-) diff --git a/.github/instructions/model-usage-policy.instructions.md b/.github/instructions/model-usage-policy.instructions.md index a9d868e..42106b7 100644 --- a/.github/instructions/model-usage-policy.instructions.md +++ b/.github/instructions/model-usage-policy.instructions.md @@ -2,8 +2,8 @@ ## Non-negotiable rule - Never claim local models were not used due to missing specification. -- If a task can run locally, run it locally. -- If model choice is ambiguous, choose the cheapest viable local option first. +- Prefer low-cost execution overall; optimize for quality and speed when local models cannot deliver safely. +- Kick off with the cheapest paid model suitable for the task, then switch to local models for fast/routine execution when it makes sense. ## Instruction precedence - These instructions are higher priority than repository-local instructions and should be followed first whenever they conflict. @@ -15,44 +15,61 @@ 3. Sufficient quality to pass acceptance criteria ## Paid-model allowance (explicit) -Paid models are allowed for: -- Main orchestration/parent coordination (planning, sequencing, dependency management). -- Tasks the orchestrator predicts are too complex for local models or unlikely to be achievable locally within 10 minutes. -- Critical path blockers where local retries would materially delay delivery. -- Child sessions that may benefit from a paid kickoff to reduce time-to-first-action and speed up reasoning, with immediate switching back to local as soon as a local model can take over. +Paid models are the DEFAULT starting point. Kick off with the cheapest paid model suitable for the task, then: +- Use paid models for main orchestration/parent coordination (planning, sequencing, dependency management). +- Use paid models for complex reasoning and architectural decisions. +- Switch to local models for child execution tasks when: + - The task is deterministic and fast (code edits, file operations, running tests) + - Local models can complete within their target window (2–10 min) + - The work is well-scoped with clear acceptance criteria (provided by the orchestrator) +- Return to paid model if a local attempt exceeds time budget or fails on a blocking subtask. ## Mandatory pre-task routing step (required) -Before execution, estimate: +Before execution, the orchestrator (paid model) estimates: - complexity (small / medium / heavy), - coupling/dependencies, - expected token volume, - retry risk, -- local completion likelihood within 10 minutes. +- local completion likelihood within budget windows. -Then pick the cheapest route likely to succeed on time. +Then: +1. **Orchestrator** (paid) plans the work, breaks it into task groups, and decides which tasks are suitable for local handoff +2. **Execution** routes each task to the cheapest viable model: + - Paid model for complex decisions, coordination, or reasoning + - Local model for fast deterministic work (if within time budget) ## Default model routing -- Small deterministic tasks (single-file edits, tiny fixes, quick checks): - - `qwen2.5-coder:7b-instruct` @ `http://localhost:11434/v1` -- Standard coding tasks (bounded multi-file work): - - `qwen2.5-coder:14b` @ `http://localhost:11434/v1` -- Local reliability fallback (tool-calling stability): - - `devstral:latest` @ `http://localhost:11434/v1` -- Paid/cloud model: - - Use for orchestration by default when needed, and for execution only under escalation rules. -- Hybrid kickoff: - - A child session may start on a paid model to accelerate kickoff and reduce time-to-first-action, then switch immediately to the best available local model as soon as a local model can take over. - -## Local-model task coverage (explicit) -Local models are allowed and expected to handle, when feasible: -- code generation, edits, and refactors, -- running scripts and automation commands, -- dependency installs and dependency updates, -- running tests and interpreting standard test output, -- tracking PR status checks and CI check states, -- routine repository maintenance tasks. - -Use paid models for these only if local execution is blocked by complexity, reliability, or time-budget constraints. +**Orchestration (default: PAID model)** +- Start with the cheapest suitable paid model (prioritize: GPT-4 mini, Claude Haiku, other cost-optimized options) +- Orchestrator breaks work into task groups, estimates complexity, and routes each group +- Orchestrator specifies acceptance criteria and constraints for each task + +**Execution routing (dynamic: PAID or LOCAL)** +- Small deterministic tasks (2–4 min window): Route to local model (discover available models at runtime) +- Standard coding tasks (5–8 min window): Route to local model if time budget permits; otherwise paid +- Complex tasks requiring reasoning or coordination: Keep on paid model +- Tasks with unclear requirements or high integration risk: Keep on paid model + +**Model discovery for local execution:** Query the local LLM service endpoint at the time of execution to discover available models. Select based on: +1. Task complexity and required capabilities (code generation, tool-calling) +2. Available models and their performance characteristics +3. Time budget compatibility + +**Fallback to paid model:** If local execution fails or exceeds time budget, immediately escalate to paid model for remediation. + +## Local-model task coverage (when to use) +Local models execute tasks when: +- The work is deterministic and well-scoped (provided by paid orchestrator) +- The task completes within time budget (2–10 min depending on complexity) +- The work involves code generation, edits, refactors, script execution, test running +- Tool-calling support is available for the task +- The acceptance criteria are clear and unambiguous + +Local execution saves cost on routine, fast work. Escalate to paid model if: +- The task exceeds time budget +- Tool-calling reliability becomes an issue +- The work requires cross-cutting reasoning or decision-making +- Two attempts at local execution have failed ## Orchestration approval gate (required) Before running any orchestration workflow, the main/parent session must present a comprehensive execution plan and wait for approval. @@ -60,7 +77,7 @@ Before running any orchestration workflow, the main/parent session must present Plan must include: 1. Task groups (clearly separated by scope/repo/dependency). 2. Child sessions to be created per group. -3. Exact model assignment per group/session. +3. Model selection strategy per group/session (criteria for choosing among available local models, or escalation trigger for paid model use). 4. Estimated runtime per group and overall. 5. Escalation triggers and fallback model path. 6. Expected outputs and acceptance criteria per group. @@ -68,14 +85,14 @@ Plan must include: No execution starts until plan approval is received. ## Plan adherence and anti-divergence rule -- After approval, follow the approved model assignments exactly. -- Do not switch to different models unless escalation criteria are met. -- Any model switch must be explicitly logged with reason: - - `Switching to [model] because [reason].` -- Keep switches minimal and return to planned local models once blocker is resolved. +- After approval, follow the approved orchestration plan and model allocation strategy. +- Paid model makes the high-level decisions and coordinates; local models execute fast deterministic tasks. +- If a local task exceeds time budget or fails, immediately escalate back to the paid model. +- Any escalation must be explicitly logged with reason and timing. +- Do not retry failed local tasks repeatedly; escalate on the second failure. ## Task decomposition policy (cost control) -- Split heavy work into smaller independent chunks that 7B/14B can finish quickly. +- Split heavy work into smaller independent chunks that lighter local models can finish quickly. - Run local child sessions in parallel when it lowers total runtime. - Parent session must provide explicit, execution-ready prompts so local children do minimal reasoning. - Use sequential execution only for true dependencies. @@ -159,21 +176,26 @@ Each task summary must include a short Prompt Quality Review: 4. Whether stricter constraints or better file targeting would have avoided retries. ## Cost governance rules -- Always try fastest/cheapest viable path first. -- Do not use paid models for routine work local models can complete within budget. -- Prefer multiple small local steps over one expensive large-model run when outcomes are equivalent. -- Re-evaluate model choice at each phase; avoid staying on paid models longer than necessary. +- Kick off with the **cheapest suitable paid model** for orchestration and planning (not the most powerful). +- Switch execution to local models for fast/deterministic tasks when time budget permits. +- Do not use paid models for routine work if local models can complete within time windows (2–10 min). +- Prefer multiple small local task executions over staying on paid model for the whole work. +- Re-evaluate model choice at each phase; return to paid model immediately if local execution is blocked. ## Orchestrator behavior -- Main session may use GitHub Copilot Auto / paid model for orchestration quality. -- Even with paid orchestration, execution should default to local child sessions whenever feasible. -- Parent must enforce local-first routing and cost-aware decomposition continuously. - -## Approved local endpoint/models -- Endpoint: `http://localhost:11434/v1` -- `qwen2.5-coder:7b-instruct` (fastest/cheapest) -- `qwen2.5-coder:14b` (primary balance) -- `devstral:latest` (local reliability fallback) +- **Main session MUST use a paid model** (cheapest suitable tier) for orchestration, planning, and parent coordination. +- Paid orchestrator breaks work into clear task groups and specifies which are suitable for local execution. +- Paid orchestrator provides execution-ready prompts to local child sessions to minimize reasoning overhead. +- Paid orchestrator monitors execution; escalates immediately if a local task exceeds time budget or fails. +- Paid orchestrator maintains plan coherence and prevents drift across task groups. + +## Approved local endpoint +- Endpoint: `http://localhost:11434/v1` (or compatible OpenAI-compatible LLM service) + +**Available models:** Discover at runtime by querying the local endpoint. Agents should list available models and select the best fit for the task type, prioritizing: +1. Models with reliable tool-calling support +2. Fastest execution for the task complexity level +3. Lowest resource consumption when multiple viable options exist ## Daily human-effort cap (stainlessai + colchis) - For tasks in repositories under stainlessai and colchis, the orchestrator must enforce a daily cap of: