Mistral NeMo

Mistral NeMo#

Mistral NeMo is a 12B instruction-tuned, auto-regressive dense transformer developed jointly by Mistral AI and NVIDIA for assistant-style text generation.

Furiosa-LLM runs Mistral NeMo in FP8 using RedHatAI’s quantized release (static FP8 weights with dynamic FP8 activation quantization; the KV cache stays in 16-bit precision). FuriosaAI publishes a pre-compiled build under the furiosa-ai organization on the Hugging Face Hub, shipping a Furiosa Executable Bundle (FXB) for running it on FuriosaAI RNGD with Furiosa-LLM. The same weights also run on other frameworks (such as vLLM, SGLang, and Transformers); for usage with those, see the upstream model card linked below.

Available Models#

Model

Quantization

RNGD cards

Notes

furiosa-ai/Mistral-Nemo-Instruct-2407-FP8

FP8

2

12B instruction-tuned

  • Architecture: Mistral NeMo (dense), MistralForCausalLM

  • Input / Output: Text / Text

  • Quantization: The linear (Transformer block) weights are quantized to FP8 (static), and activations use dynamic FP8 quantization at runtime (per-token, without offline calibration). The KV cache stays in 16-bit precision.

Usage#

To run this model with Furiosa-LLM, follow the example commands below after installing Furiosa-LLM and its prerequisites.

Launch the server#

The simplest way to serve the model is:

# Launch the server, listening on port 8000 by default
furiosa-llm serve furiosa-ai/Mistral-Nemo-Instruct-2407-FP8

When the server is ready, you will see:

INFO:     Started server process [27507]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

Basic Usage#

The server exposes an OpenAI-compatible API. You can send a request with curl:

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
    "model": "furiosa-ai/Mistral-Nemo-Instruct-2407-FP8",
    "messages": [{"role": "user", "content": "What is the capital of France?"}]
    }' \
    | python -m json.tool

Advanced Usage#

Mistral-Nemo-Instruct-2407-FP8 is a non-reasoning chat model and does not require a reasoning parser or model-specific chat-template flags. For general serving options, see the Furiosa-LLM Server guide.

Learn more#