How many can optillm handle?

[creativenets2]

I am using 4 optimizations (moa, leap, bon, self_concistency) using |.

I tried using 5 requests sometimes i got 500:

optillm-openrouter | 2025-10-27 09:25:50,172 - INFO - Generated 1 initial completions using n parameter. Tokens used: 502
optillm-openrouter | 2025-10-27 09:26:04,917 - INFO - 127.0.0.1 - - [27/Oct/2025 09:26:04] "GET /health HTTP/1.1" 200 -
optillm-openrouter | 2025-10-27 09:26:34,985 - INFO - 127.0.0.1 - - [27/Oct/2025 09:26:34] "GET /health HTTP/1.1" 200 -
optillm-openrouter | 2025-10-27 09:26:39,811 - ERROR - Error processing request: Expecting value: line 327 column 1 (char 1793)
optillm-openrouter | 2025-10-27 09:26:39,811 - INFO - 172.1.7.5 - - [27/Oct/2025 09:26:39] "POST /v1/chat/completions HTTP/1.1" 500 -
optillm-openrouter | 2025-10-27 09:26:40,832 - INFO - Received request to /v1/chat/completions

was there any limit on how many requests the optillm can handle?

Can it run 5-10 requests simultaneously using 3-5 optimizations on each? If so, where do you think the 500 is coming from?

[codelion]

THe number of requests shouldn't be an issue, I have used it for synthetic data generation with several parallel requests and servers as in - https://x.com/asankhaya/status/1972233930117144943 Are you using all optimizations with the same model and which model is it?

[creativenets2]

i used it on kimi k2, mini max m2 and glm 4.6, although this routes to open router. I dont think open router is rate limiting since the error would be different.

[creativenets2]

some errors are like:
optillm-openrouter | 2025-10-27 18:07:25,910 - ERROR - Error processing request: 'NoneType' object is not subscriptable
optillm-openrouter | 2025-10-27 18:07:25,910 - INFO - 172.1.7.75 - - [27/Oct/2025 18:07:25] "POST /v1/chat/completions HTTP/1.1" 500 -

while most is like the one i pasted above

[codelion]

For some of the approaches like bon and moa we require the underlying api endpoint to be able to support n responses. Can you check if you are able to use them separately with openrouter? It may be returning only 1 response while the code expects 3 and tries to access it and errors.

[creativenets2]

yes its random. so for example i am processing 20 files using the 4 optimization (max 2 concurrent jobs), 5-10 will have this random 500 error. But if i use only one or two i dont have these 500 errors. so is it because of the complexity?

[codelion]

I will need to trace through and debug if you can share the exact config or some small test case I can take a look.

[creativenets2]

In regards to config.... I did not change anything actually, so whats there on the default when you clone the repo and just call the optimization is what i did.

Are you looking for a different config?