Tool Calling

Tool calling (also known as function calling) enables models to interact with external tools and APIs. Furiosa-LLM supports tool calling for models trained with this capability.

Tool Calling Parsers

The system converts model outputs into the OpenAI response format through a designated parser implementation. Tool calling parsers are model-dependent, as different models use different formats for tool calls.

Currently, Furiosa-LLM supports the following tool calling parsers:

• hermes: For models using the Hermes tool calling format (e.g., EXAONE-4.0, Qwen3 series). Note that Qwen3-Coder is an exception within the Qwen3 series; use qwen3_xml for it.

• llama3_json: For Llama series models (e.g., Llama 3.1, Llama 3.2)

• openai: For models using the OpenAI tool calling format (e.g., gpt-oss-20b, gpt-oss-120b)

• gemma4: For Gemma 4 models

• qwen3_xml (alias qwen3_coder): For Qwen3-Coder models, whose chat template asks for an XML call format (<tool_call><function=...><parameter=...>) instead of the Hermes JSON.

For automatic tool calling, start the server with --enable-auto-tool-choice and select the appropriate parser using --tool-call-parser.

Tool Choice Options

The tool_choice parameter controls how the model selects tools to call. Furiosa-LLM supports the following options:

• auto (default when tools are supplied): The model decides whether to call a tool or respond directly based on the conversation context.

• none (default when no tools are supplied): Disables tool calling for the request.

• required: Forces the model to call at least one tool. The model cannot respond without making a tool call.

• {"type": "function", "function": {"name": "<function_name>"}}: Forces the model to call a specific named function.

These options apply to both streaming and non-streaming Chat Completions requests for models that support tool calling.

Supply a non-empty tools list for required or a named function choice. For these choices, the server constrains generation to the tool’s JSON schema and returns the result as tool_calls.

With tool_choice="none", tool definitions are omitted from the prompt and tool calls are suppressed in the response.

For more details on the tool calling specification, refer to the OpenAI Chat API documentation.

Online Example with Named Function Calling

Online Example with Named Function Calling

import json
import os
from openai import OpenAI

base_url = os.getenv("OPENAI_BASE_URL", "http://localhost:8000/v1")
api_key = os.getenv("OPENAI_API_KEY", "EMPTY")
client = OpenAI(base_url=base_url, api_key=api_key)

def get_weather(location: str, unit: str):
    return f"Getting the weather for {location} in {unit}..."

def get_time(timezone: str):
    return f"Getting the current time in {timezone}..."

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather in a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string", "description": "City and state, e.g., 'San Francisco, CA'"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
                },
                "required": ["location", "unit"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "get_time",
            "description": "Get the current time in a given timezone",
            "parameters": {
                "type": "object",
                "properties": {
                    "timezone": {"type": "string", "description": "Timezone, e.g., 'America/Los_Angeles'"}
                },
                "required": ["timezone"]
            }
        }
    }
]

# The user asks for both weather and time, but ``tool_choice`` forces the
# model to call ``get_weather`` only. The response is constrained to
# ``get_weather``'s schema, so the model cannot smuggle a ``timezone``
# argument (which belongs to ``get_time``) into the call.
response = client.chat.completions.create(
    model=client.models.list().data[0].id,
    messages=[{"role": "user", "content": "What's the weather and time in San Francisco?"}],
    tools=tools,
    tool_choice={"type": "function", "function": {"name": "get_weather"}},  # Force specific function
    temperature=0.0,
)

tool_call = response.choices[0].message.tool_calls[0].function
print(f"Function called: {tool_call.name}")
print(f"Arguments: {tool_call.arguments}")
assert tool_call.name == "get_weather", f"Expected 'get_weather' but got '{tool_call.name}'"
print(f"Result: {get_weather(**json.loads(tool_call.arguments))}")