Auto unload on timeout

[brycehenson]

adds unload after a period of inactivity.

      - MODEL_AUTO_UNLOAD_ENABLED=true
      - MODEL_AUTO_UNLOAD_TIMEOUT_SECONDS=30

request for feeback

Would you prefer a different flag name ?

TODO

• test on gpu
• test on cpu
• test on rocm

[remsky]

Nice work, careful timer handling. Two things:

1. generate_from_phonemes in tts_service.py uses the backend directly, so the idle timer could unload the model mid-use. Can we move the begin/end pair into one context manager e.g

   # model_manager.py
   @asynccontextmanager
   async def hold(self):
       await self._begin_request()
       try:
           yield
       finally:
           await self._end_request()

   then both generate() and generate_from_phonemes wrap their work in async with manager.hold():. One path for the accounting, less code overall.

2. On the flag: I think we can drop MODEL_AUTO_UNLOAD_ENABLED and let MODEL_AUTO_UNLOAD_TIMEOUT_SECONDS=0 (default) mean off. _auto_unload_enabled() already treats timeout <= 0 that way if I follow.

[brycehenson]

Thanks, good points. Implemented 1 and 2.

validation with unit tests:

• 276 passed, 17 deselected, 7 warnings in 3.51s

Validation with integrations tests:

• docker compose -f docker/docker-compose.test.yml up --build --abort-on-container-exit --exit-code-from test-client
• Result: 17 passed in 44.78s

ps: Thank you for building and maintaining such a great tool !

[brycehenson]

@remsky merged in master and fixed a small bug in the test_generate_from_phonemes_uses_default_voice_code model_manager.hold mock. Anything else you'd like changed before this is ready to merge?

[remsky]

@brycehenson Nope, looks good thanks! Merged in now