feat(DM01-6251): decouple blocking K8s API calls from scheduler main loop

[liuwei08]

Summary

• Move GET /api/v1/nodes and GET ibpipelineinvocations from the main scheduler loop into a daemon background thread that refreshes every 10s
• Main loop reads K8s state from a lock-protected in-memory cache — no more blocking HTTP calls on the critical scheduling path
• Increase main loop sleep(1) → sleep(3) to reduce DB and K8s API pressure

Root Cause

Under K8s API load, the two blocking HTTP calls (10s timeout each) caused the nominal 1s scheduler loop to actually run every 10–20s. With a backlog of queued jobs this multiplied into queue-wait times of hours.

Evidence: generator/di-embedded-tests queue_wait = 10,211s (~2.8h). Normal expected wait: < 5s.

How the fix works

Before:

main loop (every ~20s when K8s is slow):
  HTTP GET /api/v1/nodes          ← blocks up to 10s
  HTTP GET /ibpipelineinvocations ← blocks up to 10s
  schedule()   ← jobs finally get scheduled
  sleep(1)

After:

background thread (every 10s, independent):
  HTTP GET /api/v1/nodes          → writes to _k8s_nodes cache
  HTTP GET /ibpipelineinvocations → writes to _k8s_pipelines cache

main loop (every 3s, stable):
  read _k8s_nodes cache    ← microseconds
  read _k8s_pipelines cache ← microseconds
  schedule()               ← runs 20x/min instead of ~3x/min
  sleep(3)

Risk

• K8s state is max 10s stale — node capacity and orphan detection delay by at most 10s, acceptable
• abort/timeout response latency increases by at most 3s, acceptable
• Scheduling correctness unchanged — schedule(), handle_aborts(), handle_timeouts() logic untouched
• Background thread exceptions are caught and logged, main loop unaffected

Test plan

• Deploy to test instance and verify scheduler starts without error
• Submit a batch of jobs and confirm queue_wait drops to seconds
• Verify orphaned job cleanup still works
• Monitor scheduler logs for background thread exceptions