vLLM openai docker - tool config
#21
by
johner420
- opened
Hi, complete vLLM noob. I have the docker running, but i want to use tool calling. I need to set a parser, any idea what i sent it to? Also, any other recommended args? I've not set anything other than:
--model='Nanbeige/Nanbeige4.1-3B' --gpu-memory-utilization 0.85 --max_model_len=8192 --max_num_seqs=2 --enforce-eager --enable-auto-tool-choice --tool-call-parser
I figured it out, it was 'hermes' if others are trying. It's a great little model!