We have KV Caching in a bunch of models now, including Flux2 and QwenImage 2.1.
While it helps make inference faster, it also increases memory consumption. For example, in QwenImage 2.1, with KV Caching, peak memory goes from 32.3G -> 36.0G.
To make this trade-off better for our users, I think we could start thinking about KV cache offloading.
The offloading could have different flavors:
- Offload to and onload from RAM (sync, will compromise speed)
- Offload to and onload from RAM but async (like group offload, should provide the best trade-off)
- Same options but with secondary storage
Need to think about a nice abstraction for this. @asomoza do you think this could be helpful? @DN6 let me know your thoughts too.
We have KV Caching in a bunch of models now, including Flux2 and QwenImage 2.1.
While it helps make inference faster, it also increases memory consumption. For example, in QwenImage 2.1, with KV Caching, peak memory goes from 32.3G -> 36.0G.
To make this trade-off better for our users, I think we could start thinking about KV cache offloading.
The offloading could have different flavors:
Need to think about a nice abstraction for this. @asomoza do you think this could be helpful? @DN6 let me know your thoughts too.