Skip to content

feat(llm_client): thread-pooled vLLM batch with max_concurrency; add …#80

Open
funaileo wants to merge 1 commit into
meta-llama:mainfrom
funaileo:feat/vllm-batch-threadpool
Open

feat(llm_client): thread-pooled vLLM batch with max_concurrency; add …#80
funaileo wants to merge 1 commit into
meta-llama:mainfrom
funaileo:feat/vllm-batch-threadpool