Vision-Language Models

Vision-Language Models#

Furiosa-LLM serves Vision-Language (VL) models that accept text and images through the OpenAI-compatible Chat Completions API and the Responses API. The first supported architecture is Qwen3-VL (e.g., Qwen/Qwen3-VL-32B-Instruct); see Supported Models for the authoritative list.

Image inputs may be passed as a remote https:// URL, a base64 data: URL, or a local file:// path. Preprocessing (resizing, patching, prompt-token expansion) is handled by the Hugging Face AutoProcessor that ships with the model.

Note

Video input is not currently supported. Support is planned for Furiosa SDK 2026.5.0.

Quick Start#

Launch a server with a vision-language model that includes a compatible FXB:

furiosa-llm serve furiosa-ai/Qwen3-VL-32B-Instruct

Send a chat-completion request with an image and a text prompt:

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
    "model": "Qwen/Qwen3-VL-32B-Instruct",
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://developer.furiosa.ai/latest/en/_images/Furiosa_RNGD_image.png"
                    }
                },
                {"type": "text", "text": "Describe this image in one short sentence."}
            ]
        }
    ],
    "max_tokens": 128
    }' \
    | python -m json.tool

(see the Python Example section below for a Python client example).

Image Input Formats#

The image_url.url field in a chat-completion message accepts three URL schemes:

Remote URL (http:// / https://) — the server fetches the image with httpx.

{
  "type": "image_url",
  "image_url": {"url": "https://example.com/photo.png"}
}

Base64 data URL — useful for inline payloads:

{
  "type": "image_url",
  "image_url": {"url": "data:image/png;base64,iVBORw0KGgoAAAANSUh..."}
}

Local file (file://) — disabled by default; requires the --allowed-local-media-path server flag (see below).
```
{
  "type": "image_url",
  "image_url": {"url": "file:///srv/media/photo.png"}
}
```

A request may contain multiple image_url parts interleaved with text parts. The model’s chat template controls how image placeholders are rendered into the prompt:

OpenAI-format chat templates (the common case) iterate the content array themselves, so image placeholders land at their original positions automatically — no extra server flag needed.
String-format chat templates flatten content into a single text string. By default, placeholders for any images are prepended to the prompt; pass --interleave-mm-strings to keep them at their original positions instead.

Server Options#

The following furiosa-llm serve flags configure multimodal behaviour. Requests that violate any of them are rejected with HTTP 400 before reaching the NPU.

Flag	Default	Description
`--image-limit-per-prompt N`	unlimited	Maximum number of images allowed per request. Exceeding the limit returns HTTP 400.
`--video-limit-per-prompt N`	unlimited	Maximum number of videos allowed per request. Video input is not currently supported (see the note above), so this flag has no effect yet.
`--allowed-local-media-path PATH`	disabled	Allow `file://` URLs whose resolved path is under `PATH`. Required to serve local images; other `file://` URLs are rejected.
`--allowed-media-domains D [D …]`	all allowed	Whitelist of remote domains for SSRF protection. When set, only images from listed domains are fetched; others return HTTP 400.
`--interleave-mm-strings`	off	See Image Input Formats above. No-op for the common OpenAI-format chat templates.
`--mm-processor-cache-gb GB`	`4.0`	Size of the multimodal preprocessing cache in GiB. Set to `0` to disable preprocessing caching; encoded embeddings can still be reused. See Caching Multimodal Inputs with UUID below.
`--media-io-kwargs JSON`	`{}`	Default image-fetch options, keyed by modality. See Multimodal Options for timeouts and request overrides.
`--mm-processor-kwargs JSON`	`{}`	Processor defaults with optional `images_kwargs`. See Multimodal Options for model-specific controls and request overrides.

Example with multiple allowed domains (nargs="+"):

furiosa-llm serve Qwen/Qwen3-VL-32B-Instruct \
    --fxb /path/to/qwen3-vl-32b.fxb \
    --allowed-media-domains developer.furiosa.ai cdn.example.com

Warning

Local file access (file://) is disabled unless --allowed-local-media-path is set. Treat it as a security boundary — only point it at directories whose contents you are willing to expose to chat-completion clients.

Tip

For production deployments fetching from the public internet, restrict --allowed-media-domains to the smallest set you need to limit SSRF and bandwidth-amplification risk.

Multimodal Options#

media_io_kwargs uses an image object. mm_processor_kwargs accepts processor options and optional images_kwargs. videos_kwargs and audio_kwargs are currently rejected because only image processing is supported. Set server defaults with the CLI:

furiosa-llm serve furiosa-ai/Qwen3-VL-32B-Instruct \
    --media-io-kwargs '{"image": {"timeout": 10}}' \
    --mm-processor-kwargs '{"min_pixels": 1024}'

Chat Completions and Responses requests can override these defaults with media_io_kwargs and mm_processor_kwargs in the JSON request body. With the OpenAI Python client, pass these fields through extra_body:

import os
from openai import OpenAI

client = OpenAI(
    base_url=os.getenv("OPENAI_BASE_URL", "http://localhost:8000/v1"),
    api_key=os.getenv("OPENAI_API_KEY", "EMPTY"),
)
response = client.chat.completions.create(
    model=client.models.list().data[0].id,
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {
                "url": "https://developer.furiosa.ai/latest/en/_images/Furiosa_RNGD_image.png"
            }},
            {"type": "text", "text": "Describe this image."},
        ],
    }],
    max_completion_tokens=32,
    extra_body={
        "media_io_kwargs": {"image": {"timeout": 5}},
        "mm_processor_kwargs": {"min_pixels": 4096},
    },
)
print(response.choices[0].message.content)

Overrides apply only to that request and do not change server defaults. Omitting either dictionary, setting it to null, or passing {} retains defaults. The two dictionaries have different merge rules, described below.

Image-fetch options#

Request option keys override defaults within each modality. Passing {"image": {}} retains default image-fetch options.

The following fetch option is supported:

Option	Validation	Default	Behavior
`media_io_kwargs.image.timeout`	Finite positive number of seconds; strings, booleans, and `null` are rejected.	30 seconds when omitted from both server defaults and the request.	Controls HTTP fetching, not image processing or generation. Has no effect on local files, inline data URLs, or references without a URL.

The server’s local-path and domain restrictions still apply and cannot be overridden through this dictionary.

Processor options#

Request keys replace matching top-level processor defaults, including entire nested objects. For example, an override of size replaces the entire default size object rather than merging its dimensions.

Image-specific options can be supplied in images_kwargs:

{"mm_processor_kwargs": {"min_pixels": 1024, "images_kwargs": {"max_pixels": 200704}}}

This uses both controls. A request’s images_kwargs replaces the entire default images_kwargs object; {"images_kwargs": {}} clears that group and keeps the remaining top-level options. An option must not appear both at the top level and inside images_kwargs, including after defaults and request options are combined. This applies to all option names, even when their values are equal.

Registered image processor options#

The catalog lists recommended runtime controls, rather than every upstream processor argument. The common options below apply to Qwen-VL, EXAONE 4.5, and Gemma 4. EXAONE uses the Qwen image processor with additional window-aligned resizing. Defaults come from the loaded processor configuration. The table shows the Transformers 5.8 class fallbacks; checkpoint configuration can override them, including normalization values. Omitted options use those loaded defaults. An explicit null is forwarded as such and is not generally equivalent to omission.

Option	Accepted values	Default	Behavior
`do_convert_rgb`	Boolean or `null`	`true`	Convert inputs to RGB. EXAONE performs this before window-aligned resizing.
`resample`	Integer `0` through `5`, or `null`	`3` (bicubic)	Pillow filter: nearest (0), Lanczos (1), bilinear (2), bicubic (3), box (4), or Hamming (5). The selected backend determines supported filters.
`do_rescale`	Boolean or `null`	`true`	Multiply pixel values by `rescale_factor`.
`rescale_factor`	Finite number or `null`	`1 / 255`	Pixel rescaling multiplier, used when `do_rescale` is enabled.
`do_normalize`	Boolean or `null`	Qwen: `true`. Gemma: `false`.	Normalize pixels with `image_mean` and `image_std`.
`image_mean`	Number, sequence of numbers, or `null`	Qwen: `[0.48145466, 0.4578275, 0.40821073]`. Gemma: `[0, 0, 0]`.	Scalar or per-channel mean; used when normalization is enabled.
`image_std`	Number, sequence of numbers, or `null`	Qwen: `[0.26862954, 0.26130258, 0.27577711]`. Gemma: `[1, 1, 1]`.	Scalar or per-channel standard deviation; used when normalization is enabled.
`input_data_format`	String or `null`	`null` (infer from input)	Input channel placement: `channels_first` or `channels_last`. EXAONE forwards channel-first data after its initial conversion.
`device`	Device string or `null`; offline also accepts `torch.device`	`null` (input device)	Device for image processing, such as `cpu`. PIL images start on CPU.

Model-specific controls are listed below. Omitted sizing controls retain Furiosa’s automatic sizing. EXAONE always performs window-aligned resizing.

Option	Model family	Accepted values	Default and behavior
`do_resize`	Qwen-VL / Gemma 4	Boolean or `null`	`true`. Enable image resizing.
`disable_grouping`	Qwen-VL / EXAONE 4.5	Boolean or `null`	`null` (automatic). Disable grouping equal-sized images for processing.
`min_pixels`	Qwen-VL / EXAONE 4.5	Positive integer	With FXB buckets, `(processor.patch_size * processor.merge_size) ** 2`. Minimum resize pixel count; must not exceed `max_pixels`.
`max_pixels`	Qwen-VL / EXAONE 4.5	Positive integer	Selected automatically from the image and FXB vision buckets. Maximum resize pixel count; EXAONE also enforces window alignment.
`max_soft_tokens`	Gemma 4	Integer: `70`, `140`, `280`, `560`, or `1024`	Selected automatically for the image and compiled capacity, starting from the loaded processor limit (HF fallback: `280`).

Without FXB vision buckets, sizing falls back to the loaded processor’s settings. images and return_tensors are supplied by the runtime and are not user processor options.

Strings, booleans, floats, and null are rejected for min_pixels, max_pixels, and max_soft_tokens. Defaults, request overrides, and their merged result are validated. Cross-field bounds are checked on the combined options. Unregistered options continue to pass through; the loaded processor determines whether they are supported. These options also participate in cache identity.

Processing remains subject to the loaded FXB’s compiled vision capacity. A valid option value alone does not guarantee that an image fits. The converter continues to use the loaded processor’s patch and merge geometry for capacity checks and output conversion; geometry overrides must agree with it.

Preprocessing cache hits skip image processing. Their keys include the supplied processor kwargs after server defaults and request overrides are merged and images_kwargs is flattened. Changing a supplied option such as do_rescale selects a separate cache entry. Supply the image again when requesting an option set that has not been cached; UUIDs still identify the original image.

Processor configuration defaults and automatically selected sizing values are not added to the preprocessing key. Omitting an option and explicitly supplying its default can therefore select different entries. Reuse still applies the existing min_pixels / max_pixels checks for Qwen-VL and EXAONE 4.5, or the max_soft_tokens check for Gemma 4. Entries outside the requested bounds are dropped from the hits and reprocessed. These checks do not guarantee that an automatically sized cached result matches a fresh processing run.

Validation errors#

Invalid request options return HTTP 400 or 422; invalid offline options raise ValueError. Invalid CLI option structures or fetch options are argument errors, while model-specific processor validation occurs when loading the model. These checks also apply to requests without images.

If native vision encoding or embedding reuse returns a token count that disagrees with the processed image layout, the request is aborted before text generation. Chat Completions and Responses return HTTP 500 InternalServerError as JSON, including for streaming requests, before sending any SSE events. Offline callers receive MultimodalTokenCountError (a RuntimeError); LLMEngine delivers it through step(), and AsyncLLMEngine raises it while iterating generate().

See Tokenizer API for tokenization-specific behavior and Multi-Turn Conversations for option handling with stored conversations.

Caching Multimodal Inputs with UUID#

Images are automatically hashed by content, and their processed results are cached even when no uuid is supplied. Repeated images can reuse cached results and skip pre-processing, subject to the processor-bound checks described in Multimodal Options.

Cached results contain image preprocessing output, independent of the prompt. You can reuse an image with different text or at a different position in a later prompt. Each request still validates its image placeholders and expanded context length; reusing an image does not reuse a previous response.

After a preprocessing cache miss, the server preprocesses the supplied image again before checking for reusable encoded embeddings. Automatic resizing can change with the other images in a request. Encoded embeddings are reused only when the media identity, supplied processor options, processed pixels, and layout match; this applies with or without a UUID.

You can optionally tag each image with a stable uuid to reference its cached result on follow-up requests without resending the image. UUIDs provide an additional lookup mechanism; omitting a UUID does not opt out of caching.

Add uuid next to image_url on the content part:

{
  "type": "image_url",
  "image_url": {"url": "https://example.com/receipt.png"},
  "uuid": "receipt-2026-05-01-abc"
}

A follow-up may then omit the image bytes ("image_url": null) and reference the upload by uuid alone:

{
  "type": "image_url",
  "image_url": null,
  "uuid": "receipt-2026-05-01-abc"
}

If no reusable entry exists for that UUID and the effective processor options (for example, an unknown UUID, eviction, changed options, or an entry outside the requested bounds), the server re-runs pre-processing when image_url is also supplied; otherwise it returns HTTP 400 unknown multimodal uuid, and the client should retry with the image attached. A UUID-only reference also skips fetching; supplying a URL still fetches and decodes the image before checking the processing cache. Offline callers can use multi_modal_uuids for the same reuse behavior (see Offline API below).

Cache capacity is controlled by --mm-processor-cache-gb (default 4.0, 0 disables preprocessing caching). The cap counts the pixel buffers in preprocessing entries, rather than total server memory. The least-recently-used entries are evicted when the cache is full. Encoded embeddings have a separate lifetime and can still be reused after preprocessing eviction or when preprocessing caching is disabled. UUID-only requests require a resident preprocessing entry; a retained encoded embedding alone cannot satisfy them. The cache and UUID mappings are local to the process and reset on restart.

Python Example#

Using the OpenAI Python client against a Furiosa-LLM server:

"""
OpenAI-Compatible Server - Vision-Language Chat Completion Example

Sends a chat-completion request containing an image and a text prompt
to a running Furiosa-LLM server.

The image URL below is hosted on the Furiosa developer documentation
site and is intended to be stable. For other image-input forms (base64
``data:`` URLs and local ``file://`` paths), see the Vision-Language
Models documentation page.
"""

import os

from openai import OpenAI

base_url = os.getenv("OPENAI_BASE_URL", "http://localhost:8000/v1")
api_key = os.getenv("OPENAI_API_KEY", "EMPTY")
client = OpenAI(base_url=base_url, api_key=api_key)

IMAGE_URL = "https://developer.furiosa.ai/latest/en/_images/Furiosa_RNGD_image.png"

response = client.chat.completions.create(
    model=client.models.list().data[0].id,
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "image_url", "image_url": {"url": IMAGE_URL}},
                {"type": "text", "text": "Describe this image in one short sentence."},
            ],
        }
    ],
    max_completion_tokens=32,
    temperature=0.0,
)

print(response.choices[0].message.content)

Offline API#

The same image inputs are accepted by the offline LLM / LLMEngine APIs via the multi_modal_data field on the prompt dict. Pass PIL images keyed by modality:

from PIL import Image
from furiosa_llm import LLM, SamplingParams

image = Image.open("photo.png")
prompts = [
    {
        "prompt": "<|vision_start|><|image_pad|><|vision_end|>Describe this image.",
        "multi_modal_data": {"image": [image]},
        "mm_processor_kwargs": {"min_pixels": 4096},
    }
]

with LLM(
    "Qwen/Qwen3-VL-32B-Instruct",
    fxb="/path/to/qwen3-vl-32b.fxb",
    mm_processor_kwargs={"min_pixels": 1024},
) as llm:
    outputs = llm.generate(prompts, SamplingParams(max_tokens=64))
    print(outputs[0].outputs[0].text)

LLM(mm_processor_kwargs=...) sets processor defaults. Each prompt’s mm_processor_kwargs overrides those defaults using the same processor merge rules as the server; the example uses min_pixels=4096 for this prompt. Offline callers supply decoded images, so media_io_kwargs is not an offline LLM option.

Content-based caching and the UUID references described in Caching Multimodal Inputs with UUID above are also available through the offline API. Add multi_modal_uuids alongside multi_modal_data, in the same order; pass None for items that do not need a UUID reference. Those items still participate in content-based caching.

Each modality’s UUID list must have exactly as many entries as its image list. A bare UUID string is also accepted for a single image. Repeating a UUID keeps each occurrence in the prompt: provide a corresponding image placeholder for every occurrence, even when they reuse the same cached result.

For example:

prompts = [
    {
        "prompt": "<|vision_start|><|image_pad|><|vision_end|>Describe this image.",
        "multi_modal_data": {"image": [image]},
        "multi_modal_uuids": {"image": ["receipt-2026-05-01-abc"]},
    }
]

A follow-up prompt may then omit the image and reference it by UUID alone — pass None in the multi_modal_data slot:

followup = [
    {
        "prompt": "<|vision_start|><|image_pad|><|vision_end|>Translate to Korean.",
        "multi_modal_data": {"image": [None]},
        "multi_modal_uuids": {"image": ["receipt-2026-05-01-abc"]},
    }
]

See API Reference for the full API surface.