LLMEngine class

LLMEngine class#

Overview#

The LLMEngine provides an interface for text generation, supporting configuration through command-line arguments.

Example Usage#

import argparse
from typing import List, Tuple

from furiosa_llm import EngineArgs, LLMEngine, RequestOutput, SamplingParams


def create_test_prompts() -> List[Tuple[str, SamplingParams]]:
    """Create a list of test prompts with their sampling parameters."""
    return [
        ("A robot may not injure a human being",
        SamplingParams(temperature=0.0)),
        ("To be or not to be,",
        SamplingParams(temperature=0.8, top_k=5)),
        ("What is the meaning of life?",
        SamplingParams(n=1,
                        temperature=0.8,
                        top_p=0.95)),
    ]


def process_requests(engine: LLMEngine,
                    test_prompts: List[Tuple[str, SamplingParams]]):
    """Continuously process a list of prompts and handle the outputs."""
    request_id = 0

    while test_prompts or engine.has_unfinished_requests():
        if test_prompts:
            prompt, sampling_params = test_prompts.pop(0)
            engine.add_request(str(request_id), prompt, sampling_params)
            request_id += 1

        request_outputs: List[RequestOutput] = engine.step()

        for request_output in request_outputs:
            if request_output.finished:
                print(request_output)


def initialize_engine(args: argparse.Namespace) -> LLMEngine:
    """Initialize the LLMEngine from the command line arguments."""
    engine_args = EngineArgs.from_cli_args(args)
    return LLMEngine.from_engine_args(engine_args)


def main(args: argparse.Namespace):
    """Main function that sets up and runs the prompt processing."""
    engine = initialize_engine(args)
    test_prompts = create_test_prompts()
    process_requests(engine, test_prompts)


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser = EngineArgs.add_cli_args(parser)
    args = parser.parse_args()
    main(args)

The script can be executed with various arguments defined in EngineArgs, as shown in the following example:

python engine_cli.py --model /path/to/model --devices npu:0

For a comprehensive list of available arguments for EngineArgs, please refer to the section below.

Arguments supported by LLMEngine#

usage: engine_cli.py [-h] --model MODEL [--fxb FXB] [--revision REVISION]
                     [--tokenizer TOKENIZER] [--tokenizer-mode TOKENIZER_MODE]
                     [--devices DEVICES]
                     [--pipeline-parallel-size PIPELINE_PARALLEL_SIZE]
                     [--data-parallel-size DATA_PARALLEL_SIZE]
                     [--npu-queue-limit NPU_QUEUE_LIMIT]
                     [--max-processing-samples MAX_PROCESSING_SAMPLES]
                     [--experimental-enable-npu-sampling]

options:
  -h, --help            show this help message and exit
  --model MODEL         The Hugging Face model ID or local model directory. Currently
                        only one model is supported per server.
  --fxb FXB             Optional path to a compatible FXB. When omitted, a bundle is
                        discovered from the model directory or local FXB cache.
  --revision REVISION   The specific model revision on Hugging Face Hub if the model
                        is given as a Hugging Face model id. It can be a branch name,
                        a tag name, or a commit id. Its default value is main.
                        However, if a given model belongs to the furiosa-ai
                        organization, the model will use the release model tag by
                        default.
  --tokenizer TOKENIZER
                        The name or path of a HuggingFace Transformers tokenizer.
  --tokenizer-mode TOKENIZER_MODE
                        The tokenizer mode. "auto" will use the fast tokenizer if
                        available, and "slow" will always use the slow tokenizer.
  --devices DEVICES     The devices to run the model. It can be a single device or a
                        comma-separated list of devices. Each device can be either
                        "npu:X" or "npu:X:Y", where X is a device index and Y is a NPU
                        core range notation (e.g. "npu:0" for whole npu 0, "npu:0:0"
                        for core 0 of NPU 0, and "npu:0:0-3" for fused core 0-3 of npu
                        0). If not given, all available unoccupied devices will be
                        used.
  --pipeline-parallel-size PIPELINE_PARALLEL_SIZE
                        The size of the pipeline parallelism group. If not given, it
                        will use the value from the loaded FXB.
  --data-parallel-size DATA_PARALLEL_SIZE
                        The size of the data parallelism group. If not given, it will
                        be inferred from total available PEs and other parallelism
                        degrees.
  --npu-queue-limit NPU_QUEUE_LIMIT
                        The NPU queue limit of the scheduler config.
  --max-processing-samples MAX_PROCESSING_SAMPLES
                        The maximum processing samples. Used as an hint for the
                        scheduler.
  --experimental-enable-npu-sampling
                        [EXPERIMENTAL] Draw the next token with the artifact's on-chip
                        sampling kernel instead of on the host. Requires an artifact
                        compiled with sampling kernels; engine init fails if the
                        artifact carries no sampling kernel. (default: False)

API Reference#

class furiosa_llm.LLMEngine(native_engine: NativeEngineLike, tokenizer: PythonBackend | TokenizersBackend, task_type: Literal['generate', 'embed', 'classify', 'score', 'token_embed', 'token_classify', 'plugin'], max_model_len: int, llm: LLM | None = None, model_path: str | None = None, trust_remote_code: bool = False)[source]#

LLMEngine receives requests and generates texts. Implements the API interface compatible with vLLM’s LLMEngine, but this class is based on furiosa-runtime and FuriosaAI NPU.

The request scheduling approach of this engine is different from that of vLLM’s . While vLLM provides fine-grained control over decoding via the step method, this engine immediately begins text generation in the background as soon as a request is submitted via add_request(), continuing asynchronously until completion. The generated results are placed in a queue that clients can retrieve by calling step().

The Furiosa native engine handles scheduling and batching internally, allowing clients to retrieve results via step() calls without needing to manage the decoding schedule.

abort_request(request_id: str | Iterable[str])[source]#

Aborts request(s) with the given ID.

add_request(request_id: str, prompt: str | TextPrompt | TokensPrompt, params: SamplingParams | PoolingParams) → None[source]#

Adds a new request to the engine. The decoding iteration starts immediately after adding the request. Errors raised after submission are delivered by step().

Parameters:

  • request_id – The unique id of the request.

  • prompt – The prompt to the LLM.

  • params – The sampling | pooling parameters of the request.

classmethod from_engine_args(args: EngineArgs) → LLMEngine[source]#

Creates an LLMEngine from EngineArgs.

has_unfinished_requests() → bool[source]#

Returns True if there are unfinished requests.

step() → list[RequestOutput | PoolingRequestOutput][source]#

Returns newly generated results of one decoding iteration from the queue.

Raises the original exception of a failed request and retires that request. Results preceding an error are returned first; later events remain queued. A native vision token-count mismatch raises MultimodalTokenCountError; the multimodal request has already been aborted before this is delivered.