LLM class

class furiosa_llm.LLM(model: str, task_type: str | None = None, llm_config: LLMConfig | None = None, qformat_path: PathLike | None = None, qparam_path: PathLike | None = None, quant_ckpt_file_path: PathLike | None = None, hf_overrides: Dict[str, Any] = {}, bucket_config: BucketConfig | None = None, speculative_model: str | LLM | None = None, speculative_model_llm_config: LLMConfig | None = None, speculative_model_qformat_path: PathLike | None = None, speculative_model_qparam_path: PathLike | None = None, speculative_model_quant_ckpt_file_path: PathLike | None = None, speculative_model_config: Dict[str, Any] = {}, speculative_model_bucket_config: BucketConfig | None = None, speculative_model_paged_attention_num_blocks: int | None = None, num_speculative_tokens: int | None = None, max_seq_len_to_capture: int = 2048, max_prompt_len: int | None = None, tensor_parallel_size: int = 4, pipeline_parallel_size: int = 1, data_parallel_size: int | None = None, tokenizer: PreTrainedTokenizer | PreTrainedTokenizerFast | None = None, tokenizer_mode: Literal['auto', 'slow'] = 'auto', trust_remote_code: bool = False, seed: int | None = None, devices: str | Sequence[Device] | None = None, param_file_path: PathLike | None = None, param_saved_format: Literal['safetensors', 'pt'] = 'safetensors', do_decompositions_for_model_rewrite: bool = False, comp_supertask_kind: Literal['edf', 'dfg', 'fx'] | None = None, cache_dir: PathLike | None = PosixPath('/root/.cache/furiosa/llm'), backend: LLMBackend | None = None, use_blockwise_compile: bool = True, num_blocks_per_supertask: int = 1, num_blocks_per_pp_stage: Sequence[int] | None = None, embed_all_constants_into_graph: bool = False, paged_attention_num_blocks: int | None = None, paged_attention_block_size: int = 1, kv_cache_sharing_across_beams_config: KvCacheSharingAcrossBeamsConfig | None = None, scheduler_config: SchedulerConfig = SchedulerConfig(npu_queue_limit=2, max_processing_samples=65536, spare_blocks_ratio=0.2, is_offline=False, prefill_chunk_size=None), packing_type: Literal['IDENTITY'] = 'IDENTITY', compiler_config_overrides: Mapping | None = None, use_random_weight: bool = False, num_pipeline_builder_workers: int = 1, num_compile_workers: int = 1, skip_engine: bool = False, artifacts_export_path: str | PathLike | None = None, *, _cleanup: bool = True, _pipelines_with_metadata: Sequence[PipelineWithMetadata] | None = None, _custom_buckets: Sequence[Bucket | BucketWithOutputLogitsSize] = [], _optimize_logit_shape: bool = False, _model_metadata: ModelMetadata | None = None, _unpadded_vocab_size: int | None = None, **kwargs)

Bases: object

An LLM for generating texts from given prompts and sampling parameters.

Parameters:

• model – The name of the pretrained model. This corresponds to pretrained_model_name_or_path in HuggingFace Transformers.

• task_type – The type of the task. This corresponds to task in HuggingFace Transformers. See https://huggingface.co/docs/transformers/main/en/quicktour#pipeline for more details.

• llm_config – The configuration for the LLM. This includes quantization and optimization configurations.

• qformat_path – The path to the quantization format file.

• qparam_path – The path to the quantization parameter file.

• quant_ckpt_file_path – The path to the quantized parameters checkpoint file.

• hf_overrides – Additional HuggingFace Transformers model configuration. This is a dictionary that includes the configuration for the model.

• bucket_config – Config for bucket generating policy. If not given, the model will use single one batch, max_seq_len_to_capture attention size bucket per each phase.

• speculative_model – Specualtive model for speculative decoding.

• speculative_model_llm_config – The configuration for the specualtive model. This includes quantization and optimization configurations.

• speculative_model_qformat_path – The path to the quantization format file for the specualtive model.

• speculative_model_qparam_path – The path to the quantization parameter file for the specualtive model.

• speculative_model_quant_ckpt_file_path – The path to the quantized parameters checkpoint file for the specualtive model.

• speculative_model_config – Additional HuggingFace Transformers model configuration for the specualtive model. This is a dictionary that includes the configuration for the model.

• speculative_model_bucket_config – Config for bucket generating policy. If not given, the model will use single one batch, max_seq_len_to_capture attention size bucket per each phase.

• speculative_model_paged_attention_num_blocks – The maximum number of blocks that each k/v storage per layer can store for the specualtive model. This argument must be given if the specualtive model uses paged attention.

• num_speculative_tokens – The number of tokens that specualtive model will generate speculatively during each iteration of the decoding process

• max_seq_len_to_capture – Maximum sequence length covered by LLM engine. Sequence with larger context than this will not be covered. The default is 2048.

• max_prompt_len – Maximum prompt sequence length covered by LLM engine. Prompt larger than this cannot be handled. If not given, will be obtained from bucket and other configs.

• tensor_parallel_size – The number of PEs for each tensor parallelism group. The default is 4.

• pipeline_parallel_size – The number of pipeline stages for pipeline parallelism. The default is 1, which means no pipeline parallelism.

• data_parallel_size – The size of the data parallelism group. If not given, it will be inferred from total avaialble PEs and other parallelism degrees.

• trust_remote_code – Trust remote code when downloading the model and tokenizer from HuggingFace.

• tokenizer – The name or path of a HuggingFace Transformers tokenizer.

• tokenizer_mode – The tokenizer mode. “auto” will use the fast tokenizer if available, and “slow” will always use the slow tokenizer.

• seed – The seed to initialize the random number generator for sampling.

• devices – The devices to run the model. It can be a single device or a list of devices. Each device can be either “npu:X” or “npu:X:*” where X is a specific device index. If not given, available devices will be used.

• param_file_path – The path to the parameter file to use for pipeline generation. If not specified, the parameters will be saved in a temporary file which will be deleted when LLM is destroyed.

• param_saved_format – The format of the parameter file. Only possible value is “safetensors” now. The default is “safetensors”.

• do_decompositions_for_model_rewrite – Whether to decompose some ops to describe various parallelism strategies with mppp config. When the value is True, mppp config that matches with the decomposed FX graph should be given.

• comp_supertask_kind – The format that pipeline’s supertasks will be represented as. Possible values are “fx”,”dfg”, and “edf”, and the default is “edf”.

• cache_dir – The cache directory for all generated files for this LLM instance. When its value is None, caching is disabled. The default is “$HOME/.cache/furiosa/llm”.

• backend – The backend implementation to run forward() of a model for the LLM. If not specified, the backend will be chosen based on the device kind.

• use_blockwise_compile – If True, each task will be compiled in the unit of transformer block, and compilation result for transformer block is generated once and reused. The default is True.

• num_blocks_per_supertask – The number of transformer blocks that will be merged into one supertask. This option is valid only when use_blockwise_compile=True. The default is 1.

• num_blocks_per_pp_stage – The number of transformers blocks per each pipeline parallelism stage. If not given, transformer blocks will be distributed equally.

• embed_all_constants_into_graph – Whether to embed constant tensors into graph or make them as input of the graph and save them as separate files. The default is False.

• paged_attention_num_blocks – The maximum number of blocks that each k/v storage per layer can store. This argument must be given if model uses paged attention.

• paged_attention_block_size – The maximum number of tokens that can be stored in a single paged attention block. This argument must be given if model uses paged attention.

• kv_cache_sharing_across_beams_config – Configuration for sharing kv cache across beams. This argument must be given if and only if the model is optimized to share kv cache across beams. If this argument is given, decode phase buckets with batch size of batch_size * kv_cache_sharing_across_beams_config.beam_width will be created.

• scheduler_config – Configuration for the scheduler, allowing to maximum number of tasks which can be queued to HW, maximum number of samples that can be processed by the scheduler, and ratio of spare blocks that are reserved by scheduler.

• packing_type – Packing algorithm. Possible values are “IDENTITY” only for now

• compiler_config_overrides – Overrides for the compiler config. This is a dictionary that includes the configuration for the compiler.

• use_random_weight – If True, the model will be initialized with random weights.

• num_pipeline_builder_workers – number of workers used for building pipelines (except for compilation). The default is 1 (no parallelism). Setting this value larger than 1 reduces pipeline building time, especially for large models, but requires much more memory.

• num_compile_workers – number of workers used for compilation. The default is 1 (no parallelism).

• skip_engine – If True, the native runtime engine will not be initialized. This is useful when you need the pipelines for other purposes than running them with the engine.

• artifacts_export_path – The path to export the artifacts. With artifacts, you can create LLM without quantizing or compiling the model again.

classmethod from_artifacts(path: str | PathLike, bucket_config: BucketConfig | None = None, speculative_model: LLM | None = None, num_speculative_tokens: int | None = None, speculative_model_bucket_config: BucketConfig | None = None, use_speculative_decoding_if_possible: bool = True, data_parallel_size: int | None = None, pipeline_parallel_size: int | None = None, num_blocks_per_pp_stage: Sequence[int] | None = None, tokenizer: PreTrainedTokenizer | PreTrainedTokenizerFast | None = None, tokenizer_mode: Literal['auto', 'slow'] = 'auto', seed: int | None = None, devices: str | Sequence[Device] | None = None, cache_dir: PathLike = PosixPath('/root/.cache/furiosa/llm'), backend: LLMBackend | None = None, paged_attention_num_blocks: int | None = None, scheduler_config: SchedulerConfig | None = None, speculative_model_paged_attention_num_blocks: int | None = None, packing_type: Literal['IDENTITY'] = 'IDENTITY', skip_engine: bool = False, *, _cleanup=True, **kwargs) → LLM

Instantiate LLM from saved artifacts without quantization and compilation.

Parameters:

• path – A path to artifacts to load.

• bucket_config – Config for bucket generating policy. If not given, all buckets in the artifacts will be used.

• speculative_model – Draft model for speculative decoding.

• num_speculative_tokens – The number of tokens that specualtive model will generate speculatively during each iteration of the decoding process.

• speculative_model_bucket_config – Bucket generating config for speculative_model. If not given, all specualtive model buckets (if exists) in the artifacts will be used.

• use_speculative_decoding_if_possible – If True, speculative decoding will be used if possible (speculative_model is given or there’s artifacts for specualtive model in the artifacts.). Otherwise, speculative decoding will not be used.

• data_parallel_size – The size of the data parallelism group. If not given, it will be inferred from total avaialble PEs and other parallelism degrees.

• pipeline_parallel_size – The size of the pipeline parallelism. The argument is valid only for artifacts that use blockwise compilation.

• num_blocks_per_pp_stage – The number of transformer blocks per each pipeline parallelism stage. The argument is valid only for artifacts that use blockwise compilation. If only pipeline_parallel_size is provided, transformer blocks will be distributed equally.

• tokenizer – The name or path of a HuggingFace Transformers tokenizer.

• tokenizer_mode – The tokenizer mode. “auto” will use the fast tokenizer if available, and “slow” will always use the slow tokenizer.

• seed – The seed to initialize the random number generator for sampling.

• devices – The devices to run the model. It can be a single device or a list of devices. Each device can be either “npu:X” or “npu:X:*” where X is a specific device index. If not given, devices saved in the artifacts will be used.

• cache_dir – The cache directory for all generated files for this LLM instance. When its value is None, caching is disabled. The default is “$HOME/.cache/furiosa/llm”.

• backend – The backend implementation to run forward() of a model for the LLM. The default is LLMBackend.TORCH_PIPELINE_RUNNER.

• paged_attention_num_blocks – The maximum number of blocks that each k/v storage per layer can store. This argument must be given if model uses paged attention.

• scheduler_config – Configuration for the scheduler, allowing to maximum number of tasks which can be queued to HW, maximum number of samples that can be processed by the scheduler, and ratio of spare blocks that are reserved by scheduler. If this is not given, scheduler config saved in the artifacts will be used.

• speculative_model_paged_attention_num_blocks – The maximum number of blocks that each k/v storage per layer can store for the specualtive model. This argument must be given if the specualtive model uses paged attention.

• packing_type – Packing algorithm. Possible values are “IDENTITY” only for now

• skip_engine – If True, the native runtime engine will not be initialized. This is useful when you need the pipelines for other purposes than running them with the engine.

generate(prompts: str | List[str], sampling_params: SamplingParams = SamplingParams(n=1, best_of=1, repetition_penalty=1.0, temperature=1.0, top_p=1.0, top_k=-1, min_p=0.0, use_beam_search=False, length_penalty=1.0, early_stopping=False, max_tokens=16, min_tokens=0, output_kind=RequestOutputKind.CUMULATIVE), prompt_token_ids: BatchEncoding | None = None, tokenizer_kwargs: Dict[str, Any] | None = None) → RequestOutput | List[RequestOutput]

Generate texts from given prompts and sampling parameters.

Parameters:

• prompts – The prompts to generate texts.

• sampling_params – The sampling parameters for generating texts.

• prompt_token_ids – Pre-tokenized prompt input as a BatchEncoding object. If not provided, the prompt will be tokenized internally using the tokenizer.

• tokenizer_kwargs – Additional keyword arguments passed to the tokenizer’s encode method, such as {“use_special_tokens”: True}.

Returns:

A list of RequestOutput objects containing the generated completions in the same order as the input prompts.

classmethod load_artifact(path: str | PathLike, *, devices: str | Sequence[Device] | None = None, data_parallel_size: int | None = None, pipeline_parallel_size: int | None = None, num_blocks_per_pp_stage: Sequence[int] | None = None, bucket_config: BucketConfig | None = None, scheduler_config: SchedulerConfig | None = None, paged_attention_num_blocks: int | None = None, tokenizer: PreTrainedTokenizer | PreTrainedTokenizerFast | None = None, tokenizer_mode: Literal['auto', 'slow'] = 'auto', seed: int | None = None, cache_dir: PathLike = PosixPath('/root/.cache/furiosa/llm'), skip_engine: bool = False, **kwargs) → LLM

Instantiate LLM from saved artifacts without quantization and compilation.

Parameters:

• path – A path to artifacts to load.

• devices – The devices to run the model. It can be a single device or a list of devices. Each device can be either “npu:X” or “npu:X:*” where X is a specific device index. If not given, all available devices will be used.

• data_parallel_size – The size of the data parallelism group. If not given, it will be inferred from total avaialble PEs and other parallelism degrees.

• pipeline_parallel_size – The size of the pipeline parallelism. The argument is valid only for artifacts that use blockwise compilation.

• num_blocks_per_pp_stage – The number of transformer blocks per each pipeline parallelism stage. The argument is valid only for artifacts that use blockwise compilation. If only pipeline_parallel_size is provided, transformer blocks will be distributed equally.

• bucket_config – Config for bucket generating policy. If not given, all buckets in the artifacts will be used.

• scheduler_config – Configuration for the scheduler, allowing to maximum number of tasks which can be queued to HW, maximum number of samples that can be processed by the scheduler, and ratio of spare blocks that are reserved by scheduler. If this is not given, scheduler config saved in the artifacts will be used.

• paged_attention_num_blocks – The maximum number of blocks that each k/v storage per layer can store. This argument must be given if model uses paged attention.

• tokenizer – The name or path of a HuggingFace Transformers tokenizer.

• tokenizer_mode – The tokenizer mode. “auto” will use the fast tokenizer if available, and “slow” will always use the slow tokenizer.

• seed – The seed to initialize the random number generator for sampling.

• cache_dir – The cache directory for all generated files for this LLM instance. When its value is None, caching is disabled. The default is “$HOME/.cache/furiosa/llm”.

• skip_engine – If True, the native runtime engine will not be initialized. This is useful when you need the pipelines for other purposes than running them with the engine.

classmethod load_artifacts(path: str | PathLike, **kwargs) → LLM

Instantiate LLM from saved artifacts without quantization and compilation.

Please note that this method is being deprecated. Use load_artifact instead.

Parameters:

• path – A path to artifacts to load.

• devices – The devices to run the model. It can be a single device or a list of devices. Each device can be either “npu:X” or “npu:X:*” where X is a specific device index. If not given, all available devices will be used.

• data_parallel_size – The size of the data parallelism group. If not given, it will be inferred from total avaialble PEs and other parallelism degrees.

• pipeline_parallel_size – The size of the pipeline parallelism. The argument is valid only for artifacts that use blockwise compilation.

• num_blocks_per_pp_stage – The number of transformer blocks per each pipeline parallelism stage. The argument is valid only for artifacts that use blockwise compilation. If only pipeline_parallel_size is provided, transformer blocks will be distributed equally.

• bucket_config – Config for bucket generating policy. If not given, all buckets in the artifacts will be used.

• scheduler_config – Configuration for the scheduler, allowing to maximum number of tasks which can be queued to HW, maximum number of samples that can be processed by the scheduler, and ratio of spare blocks that are reserved by scheduler. If this is not given, scheduler config saved in the artifacts will be used.

• paged_attention_num_blocks – The maximum number of blocks that each k/v storage per layer can store. This argument must be given if model uses paged attention.

• tokenizer – The name or path of a HuggingFace Transformers tokenizer.

• tokenizer_mode – The tokenizer mode. “auto” will use the fast tokenizer if available, and “slow” will always use the slow tokenizer.

• seed – The seed to initialize the random number generator for sampling.

• cache_dir – The cache directory for all generated files for this LLM instance. When its value is None, caching is disabled. The default is “$HOME/.cache/furiosa/llm”.

• skip_engine – If True, the native runtime engine will not be initialized. This is useful when you need the pipelines for other purposes than running them with the engine.

async stream_generate(prompt: str, sampling_params: SamplingParams = SamplingParams(n=1, best_of=1, repetition_penalty=1.0, temperature=1.0, top_p=1.0, top_k=-1, min_p=0.0, use_beam_search=False, length_penalty=1.0, early_stopping=False, max_tokens=16, min_tokens=0, output_kind=RequestOutputKind.CUMULATIVE), prompt_token_ids: BatchEncoding | None = None, tokenizer_kwargs: Dict[str, Any] | None = None, is_demo: bool = False) → AsyncGenerator[str, None]

Generate texts from given prompt and sampling parameters.

Parameters:

• prompt – The prompt to generate texts. Note that unlike generate, this API supports only a single prompt.

• sampling_params – The sampling parameters for generating texts.

• prompt_token_ids – Pre-tokenized prompt input as a BatchEncoding object. If not provided, the prompt will be tokenized internally using the tokenizer.

• tokenizer_kwargs – Additional keyword arguments passed to the tokenizer’s encode method, such as {“use_special_tokens”: True}.

Returns:

A stream of generated output tokens.