SamplingParams class

SamplingParams class#

class furiosa_llm.SamplingParams(*, n: int = 1, repetition_penalty: float = 1.0, temperature: float = 1.0, top_p: float = 1.0, top_k: int = -1, min_p: float = 0.0, stop: str | list[str] | None = None, stop_token_ids: list[int] | None = None, include_stop_str_in_output: bool = False, ignore_eos: bool = False, max_tokens: int | None = 16, min_tokens: int = 0, skip_special_tokens: bool = True, logprobs: int | None = None, prompt_logprobs: int | None = None, output_kind: RequestOutputKind = RequestOutputKind.CUMULATIVE, structured_outputs: StructuredOutputsParams | None = None, reasoning_structured_outputs: StructuredOutputsParams | None = None, extra_args: dict | None = None)[source]#

Bases: object

Sampling parameters for text generation.

Parameters:

  • n – Number of output sequences to return for the given prompt.

  • repetition_penalty – Float that penalizes new tokens based on whether they appear in the prompt and the generated text so far. Values > 1 encourage the model to use new tokens, while values < 1 encourage the model to repeat tokens.

  • temperature – Float that controls the randomness of the sampling. Lower values make the model more deterministic, while higher values make the model more random. Zero means greedy sampling.

  • top_p – Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to 1 to consider all tokens.

  • top_k – Integer that controls the number of top tokens to consider. Set to -1 to consider all tokens.

  • min_p – Float that represents the minimum probability for a token to be considered, relative to the probability of the most likely token. Must be in [0, 1]. Set to 0 to disable this.

  • stop – List or single non-empty string(s) that stop generation when the first string is matched. By default, the matched stop string is removed from the returned text. When streaming, Furiosa-LLM may hold back a trailing window up to the longest configured stop string to prevent a partial match from being emitted.

  • stop_token_ids – Token IDs that stop the generation when they are generated. The returned output will contain the stop tokens unless the stop tokens are special tokens.

  • include_stop_str_in_output – Whether to retain the matched stop string in the returned text. Defaults to False. When True, text after the match within the token that completed it is removed. Streaming does not need to hold back a trailing stop-string window in this mode.

  • ignore_eos – Whether to ignore the EOS token and continue generating tokens after the EOS token is generated.

  • max_tokens – Maximum number of tokens to generate per output sequence. If the value is None, it is capped to the maximum sequence length.

  • min_tokens – Minimum number of tokens to generate per output sequence before EOS or stop_token_ids can be generated

  • skip_special_tokens – Whether to skip special tokens in the output.

  • logprobs – Number of log probabilities to return per output token. When set to None, no probability is returned. If set to a non-None value, the result includes the log probabilities of the specified number of most likely tokens, as well as the chosen tokens. Note that the implementation follows the OpenAI API: The API will always return the log probability of the sampled token, so there may be up to logprobs+1 elements in the response.

  • prompt_logprobs – Number of log probabilities to return per prompt token. When set to None (default), no prompt logprobs are returned. When set to a non-negative integer, returns the top-k log probabilities for each prompt token position, plus the actual token’s logprob. Set to -1 to return log probabilities for all vocabulary tokens. Warning: Using -1 can cause significant memory and network overhead as it returns logprobs for the entire vocabulary (e.g., ~150K tokens for Qwen models) at each prompt position.

  • structured_outputs – Parameter for specifying structured output requirements.

  • reasoning_structured_outputs – Parameter that matches the reasoning part of model output. This will be used to strip the reasoning tokens before applying structured output requirements on the answer content.

  • extra_args – Arbitrary additional arguments carried alongside the standard sampling parameters. All extra (undeclared) keys of an API request are collected here. Specific consumers extract the keys they care about; e.g. kv_transfer_params is extracted from this dict.

  • kv_transfer_params – Access point of the remote KV connector, extracted from extra_args["kv_transfer_params"]. There can be one of two ZMQ endpoints ("host:port"): "remote_decoder" and "remote_prefiller". If set, the worker connects to the given endpoint to enable PD-disaggregated prefill or decoding. "id" is the shared identifier of the P-D pair: it must be the same value on both the prefiller and the decoder, and the connector uses it to match the two sides.

Default Sampling Parameters from generation_config.json#

If a model artifact contains a generation_config.json file, Furiosa-LLM uses the values in that file as the effective defaults for the fields listed below. The file is copied verbatim from the source Hugging Face model during artifact build — Furiosa-LLM does not customize it. If the file is absent, the plain SamplingParams() defaults apply.

The following generation_config.json keys are honored, and are mapped to SamplingParams fields as shown:

generation_config.json

SamplingParams

repetition_penalty

repetition_penalty

temperature

temperature

top_k

top_k

top_p

top_p

min_p

min_p

max_new_tokens

max_tokens

stop_strings

stop

Offline API (LLM.generate / LLM.chat / LLM.stream_generate)#

The offline API uses a binary decision based on whether the caller passes sampling_params:

  • sampling_params=None (the default) — the model’s generation_config.json defaults are applied. See furiosa_llm.LLM.get_default_sampling_params().

  • sampling_params=SamplingParams(...) — the user’s object is used as-is. No per-field merge with the model’s generation_config.json is performed.

OpenAI-compatible server#

The Chat Completions, Completions, and Responses endpoints resolve each sampling-related field in three tiers:

  1. The value specified in the request body, if set.

  2. The value from the model’s generation_config.json, if present.

  3. The API default shown in the endpoint’s parameter table (see Chat API (/v1/chat/completions), Completions API (/v1/completions), and API Reference).

Examples#

This section provides examples of how to use the token generation methods available in the SDK.

2. Random Sampling with top_p / top_k Parameters#

SamplingParams(min_tokens=10, max_tokens=100, top_p=0.3, top_k=100)

This method uses random sampling techniques for token generation, allowing for diverse outputs.

  • Parameters:

    • min_tokens: Minimum number of tokens to generate.

    • max_tokens: Maximum number of tokens to generate.

    • top_p: Cumulative probability for nucleus sampling.

    • top_k: Number of highest probability tokens to consider.

  • Behavior:

    • Each generation may yield different results, even with the same input text and parameters, enhancing variability.

    • Generation may terminate before reaching max_tokens if an End Of Sequence (EOS) token is generated.

    • The EOS token will not be generated before reaching the specified min_tokens.

3. Stop Strings#

Pass a list of non-empty strings to stop to end generation when the first string is matched. By default, the matched string is removed from the returned text. A stop-string match sets finish_reason to "stop" and records the matched string in stop_reason:

params = SamplingParams(stop=["<END>"])
output = llm.generate("Write a short answer.", params)[0].outputs[0]
# If the model generates "Answer<END>", output.text is "Answer".
# output.stop_reason is "<END>".

Set include_stop_str_in_output=True to retain the matched string. Text after the match within the completing token is still removed:

params = SamplingParams(
    stop=["<END>"],
    include_stop_str_in_output=True,
)
output = llm.generate("Write a short answer.", params)[0].outputs[0]
# If the model generates "Answer<END>", output.text is "Answer<END>".

For streamed output with the default exclusion behavior, Furiosa-LLM may delay emitting a trailing window up to the longest configured stop string so a partial match does not leak into the output. Retained-stop mode does not need that delay.