Runaway FileWatch host-event loop freezes TUI and grows debug log to 13 GB

[tdihp]

Describe the bug

A long-running/resumed Copilot CLI session can enter a tight loop that emits:

[DEBUG] [rust:copilot_runtime::protocol::jsonrpc::engine] No connection accepted a host event {"kind":"FileWatch"}

Once the loop becomes continuous, the terminal UI stops responding while the
process consumes about two CPU cores and writes hundreds of KB/s to its debug
log.

In the affected process:

• The final 10,000 log records were all the identical discarded FileWatch
  event.
• Current CPU measured over five seconds was approximately 200%.
• RSS was approximately 1.1 GB.
• The process log had reached 13 GB and was growing at approximately 460 KB/s.
• The process remained alive; this was not an OS OOM kill or terminal flow
  control pause.
• The host had ample free memory, disk space, and inodes.
• USE_TGREP=false was already effective. Startup telemetry contained
  disabled_reason: "use_tgrep_false", and no tgrep child was running.

This appears related to IDE integration, but I cannot yet prove that opening
VS Code is the sole trigger. The affected session logged IDE lock-file watcher
startup and was running in a workspace open in VS Code. Disabling IDE
auto-connect prevented IDE discovery and the continuing event storm in a
controlled fresh-session test.

This differs from #3701: there was no repeated MCP server spawning or leaked
MCP process tree. The runaway activity was inside the Copilot process's
FileWatch/JSON-RPC event path.

Affected version

GitHub Copilot CLI 1.0.81-9

The no-IDE control test was also run after auto-update installed 1.0.81-11.

Steps to reproduce the behavior

The exact transition into the continuous loop is not yet deterministic, but
this is the observed setup:

1. Open a large multi-repository workspace in VS Code.
2. Start Copilot CLI from that workspace root with debug logging enabled.
3. Resume and continue using a long-lived session.
4. Leave IDE auto-connect enabled (the default).
5. The session eventually becomes unresponsive while its log continuously
   prints the discarded FileWatch event shown above.

Useful checks while the failure is active:

ps -p <pid> -o pid,pcpu,pmem,rss,nlwp,stat,wchan,cmd
tail -n 10000 ~/.copilot/logs/process-*.log |
  sed -E 's/^[0-9TZ:.-]+ \[[A-Z]+\] \[[^]]+\] //' |
  sort | uniq -c | sort -nr | head

The affected process showed:

10000 No connection accepted a host event {"kind":"FileWatch"}

Controlled comparison

I configured:

{
  "ide": {
    "autoConnect": false
  }
}

Then started a fresh CLI session from the same workspace while VS Code remained
open. In that session:

• There were no IDE lock watcher, IDE discovery, or ide_connected records.
• The process had no inotify watch on ~/.copilot/ide.
• Only five FileWatch records appeared during startup.
• Log growth then stopped, current CPU dropped to approximately 1%, and the TUI
  remained responsive.

For comparison, using the invalid dotted JSON key
"ide.autoConnect": false produced a settings warning, started the IDE lock
watcher, detected VS Code, and connected to the IDE. The nested object shown
above is required.

Expected behavior

• FileWatch events should be consumed, coalesced, debounced, or dropped without
  entering an unbounded CPU/logging loop.
• A disconnected/uninterested host-event consumer should not cause every
  filesystem event to be logged indefinitely.
• IDE lock-file activity should not starve terminal input or rendering.
• Debug logging should be rate-limited for repeated identical events.

Additional context

• OS: WSL2 Linux 6.6.87.2-microsoft-standard-WSL2
• Architecture: ARM64
• Terminal: VS Code integrated terminal
• Workspace root contained multiple repositories.
• Session indexing reported SESSION_INDEXING=true.
• tgrep was disabled with USE_TGREP=false.
• The affected process had two inotify descriptors resolving to the resumed
  session's own ~/.copilot/session-state/<session>/ and research/
  directories. The first FileWatch records also appeared before the log line
  that started the IDE lock-file watcher, so IDE auto-connect may be an
  amplifier or trigger rather than the underlying event source.
• Another session from the same environment previously accumulated a 2 GB log
  and approximately 4 GB RSS with large bursts of the same FileWatch message,
  but later became idle instead of remaining in the tight loop.

Workaround:

{
  "ide": {
    "autoConnect": false
  }
}

Restart Copilot CLI after applying the setting. Manual /ide connection
remains available.

[tdihp]

New evidence materially narrows this down: IDE auto-connect is not required.

After terminating the first runaway process, I resumed the same local session.
The replacement process immediately reproduced the failure even though its
telemetry consistently reports:

"config_ide_autoConnect": "false"

There are no Starting IDE lock file watcher, Connected to IDE, or
ide_connected records in the replacement process log. Its only inotify
descriptors resolve to the resumed session's own:

~/.copilot/session-state/<session>/
~/.copilot/session-state/<session>/research/

Observed timeline:

05:50:04.158 Session indexing debug: SESSION_INDEXING=true
05:50:06.344 first discarded FileWatch event

The storm then continued from the first minute onward:

05:51  116,443 events/min
05:52  125,341 events/min
05:53  106,727 events/min
...
06:15  136,025 events/min

Current replacement-process measurements:

CPU:             ~239%
log growth:      ~339 KB/s
RSS:             ~1.15 GB
dominant tail:   9,691 / 10,000 records are discarded FileWatch events

The watched session directory was not continuously changing during a
three-second sample. Its persistent footprint is also modest (105 files,
~56 MB), while another non-spinning session has 3,080 files and ~1.97 GB of
state. This does not look like a simple threshold on session file count.

A brand-new session with ide.autoConnect=false remains stable. Resuming this
specific existing session reproduces the storm immediately. The likely trigger
is therefore persisted/resumed session watcher or event state, rather than VS
Code lock-file activity itself. The IDE-disabled workaround prevents one input
path but does not recover an already affected session; avoiding resume and
starting a new session is currently the reliable workaround.

[tdihp]

Further tracing of the shipped app.js identifies the watcher ownership and
explains why only two watches can still exhaust the process.

The interactive TUI unconditionally mounts two React hooks for local sessions:

1. A plan-status hook watches dirname(plan.md), which is the session root.
2. A research-status hook watches <session>/research. If that directory does
   not exist, it temporarily watches the session root until research/
   appears.

These are exactly the two inotify descriptors observed in the runaway process:

~/.copilot/session-state/<session>/
~/.copilot/session-state/<session>/research/

They are non-recursive directory watches. The file count and number of watches
are therefore not the source of exhaustion.

The relevant bundled-code path is:

plan/research React hook
  -> wz(path, { persistent: false }, callback)
  -> supportWatchPathWeak(path)
  -> async `await handle.next()` pump
  -> native Rust FileWatch host-event dispatch

Watching the directory is reasonable because plan.md, research reports, or
the research/ directory may be created or atomically replaced. The
JavaScript callbacks filter for plan.md or research .md files. However,
that filtering occurs only after the native watcher/event layer processes the
raw directory event.

The runaway process had only these two watches, but generated approximately
2,000-3,000 rejected events per second:

No connection accepted a host event {"kind":"FileWatch"}

Each rejection is dispatched and debug-logged individually. There is no
visible coalescing, rate limit, backoff, or automatic shutdown of a watcher
whose events have no consumer. That is sufficient to consume multiple CPU
cores and grow the log by hundreds of KB/s despite having only two inotify
watches.

Subagents appear to be an amplifier/trigger rather than a continuing source.
They create bursts of task transitions, session-state writes, rewind
snapshots, research output, and native session/host-channel lifecycle churn.
The evidence is consistent with a teardown/restore race leaving a weak watcher
producer detached from its consuming connection. Once the session reaches that
state, resuming it recreates the loop immediately and it continues even while
the watched directories are not changing.

In short, this is not inotify watch exhaustion:

two normal non-recursive watches
  -> watcher consumer becomes detached
  -> unbounded rejected FileWatch dispatch/log loop
  -> CPU and log exhaustion

There is currently no documented setting to disable these plan/research
watchers. ide.autoConnect=false disables only the separate IDE lock-file
watcher.

[tdihp]

Correction after further investigation: the continuous FileWatch process
initially examined was not the exact terminal instance reported as frozen.

The correct process also exhibited a substantial rejected-FileWatch event
burst. However, its event ordering indicates that this burst may be a symptom
or amplifier after the TUI event consumer disconnects, rather than the initial
cause of the freeze:

subagent starts
  -> TUI/session-host event consumer stops receiving timeline events
  -> runtime and subagents continue independently
  -> FileWatch events have no consumer and are repeatedly rejected
  -> CPU, logging, and memory pressure increase
  -> FileWatch activity may subside while the TUI remains frozen

This resembles #4533, where runtime/subagent activity continues after the
terminal stops updating. Concurrent ownership of a resumed session was also
present and may be relevant, but that relationship is not yet established.

Please treat the FileWatch storm as a confirmed resource-amplification problem,
while causality for the initial TUI detachment remains under investigation.

[MilkyWay008]

That FileWatch host-event flood at 2-3k rejections/sec is the TUI React watcher hooks firing into a detached jsonrpc engine, and the inotify watches on the two session-state dirs are just the trigger surface, so disabling auto-connect wouldn't stop it. As a stopgap, avoid resuming sessions that already looped, and truncate the debug log before relaunching so the 13GB doesn't come back instantly. There's a related report about the same class of watcher loop, worth cross-linking in case maintainers can pool the traces. WSL2 is an unusual enough combo that the reproduce details you posted will carry the issue. oh also, btw, I built a portable Hermes OTG and I run it on USB as my rescue agent; super handy doing quick diagnose & fix, check it, imo it'll help you. https://github.com/MilkyWay008/Hermes-OTG

[peacenix]

The discriminator is plan.md in the session root, combined with a regression after 1.0.81-1.

Same session, same machine, plan.md present throughout:

build	run	rejected FileWatch events
1.0.81-1	Aug 18-25, 7 days	0
1.0.81-9	Aug 26	758,629 / 4,920,665
1.0.81-14	Aug 27	6,018,006
1.0.81	Aug 27	16,634,832

Two other local sessions with no plan.md -- one with a larger 92 MB events.jsonl and 96 subagent starts -- stay flat on -9 and on GA. Both conditions are needed: the file, and a build after -1.

This fits your hook analysis. The plan-status hook watches dirname(plan.md), which is the session root, and events.jsonl is appended there on every event. First agent activity starts the loop and it then self-sustains -- one process held ~2,350 rejections/sec for 18 h overnight at a constant rate with the session idle. Your three-second sample was right: once started it does not need ongoing directory changes.

Moving plan.md out of the session root fixed it in place, no restart: 277% -> flat at idle. An affected session does not have to be abandoned.

[cwedgwood]

Confirming this can be fatal, not only a hang.

Same conditions (GA 1.0.81, plan.md in the session root). The process aborted roughly ten minutes after start:

node::OOMErrorHandler
v8::internal::V8::FatalProcessOutOfMemory
v8::internal::Heap::CollectGarbage
v8::internal::MinorGCJob::Task::RunInternal

V8 reached its own heap limit. This was not an OS OOM kill — the cgroup had headroom and recorded no kill.

[cwedgwood]

Still present in 1.0.82-0.

Same conditions (plan.md in the session root). Rejected FileWatch events, measured over a short session:

build	rate
1.0.81	~530/s
1.0.82-0	~540/s

Unchanged.

[tdihp]

changing log to default doesn't seem to help. This pretty much renders plan mode unusable :(.