SamplingParams class

SamplingParams class#

class furiosa_llm.SamplingParams(*, n: int = 1, repetition_penalty: float = 1.0, temperature: float = 1.0, top_p: float = 1.0, top_k: int = -1, min_p: float = 0.0, stop: str | list[str] | None = None, stop_token_ids: list[int] | None = None, include_stop_str_in_output: bool = False, ignore_eos: bool = False, max_tokens: int | None = 16, min_tokens: int = 0, skip_special_tokens: bool = True, logprobs: int | None = None, prompt_logprobs: int | None = None, output_kind: RequestOutputKind = RequestOutputKind.CUMULATIVE, structured_outputs: StructuredOutputsParams | None = None, reasoning_structured_outputs: StructuredOutputsParams | None = None, repetition_detection: RepetitionDetectionParams | None = None, extra_args: dict | None = None)[source]#

Bases: object

Sampling parameters for text generation.

Parameters:

  • n – Number of output sequences to return for the given prompt.

  • repetition_penalty – Float that penalizes new tokens based on whether they appear in the prompt and the generated text so far. Values > 1 encourage the model to use new tokens, while values < 1 encourage the model to repeat tokens.

  • temperature – Float that controls the randomness of the sampling. Lower values make the model more deterministic, while higher values make the model more random. Zero means greedy sampling.

  • top_p – Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to 1 to consider all tokens.

  • top_k – Integer that controls the number of top tokens to consider. Set to -1 to consider all tokens.

  • min_p – Float that represents the minimum probability for a token to be considered, relative to the probability of the most likely token. Must be in [0, 1]. Set to 0 to disable this.

  • stop – List or single non-empty string(s) that stop generation when the first string is matched. By default, the matched stop string is removed from the returned text. When streaming, Furiosa-LLM may hold back a trailing window up to the longest configured stop string to prevent a partial match from being emitted.

  • stop_token_ids – Token IDs that stop the generation when they are generated. The returned output will contain the stop tokens unless the stop tokens are special tokens.

  • include_stop_str_in_output – Whether to retain the matched stop string in the returned text. Defaults to False. When True, text after the match within the token that completed it is removed. Streaming does not need to hold back a trailing stop-string window in this mode.

  • ignore_eos – Whether to ignore the EOS token and continue generating tokens after the EOS token is generated.

  • max_tokens – Maximum number of tokens to generate per output sequence. If the value is None, it is capped to the maximum sequence length.

  • min_tokens – Minimum number of tokens to generate per output sequence before EOS or stop_token_ids can be generated

  • skip_special_tokens – Whether to skip special tokens in the output.

  • logprobs – Number of log probabilities to return per output token. When set to None, no probability is returned. If set to a non-None value, the result includes the log probabilities of the specified number of most likely tokens, as well as the chosen tokens. Note that the implementation follows the OpenAI API: The API will always return the log probability of the sampled token, so there may be up to logprobs+1 elements in the response.

  • prompt_logprobs – Number of log probabilities to return per prompt token. When set to None (default), no prompt logprobs are returned. When set to a non-negative integer, returns the top-k log probabilities for each prompt token position, plus the actual token’s logprob. Set to -1 to return log probabilities for all vocabulary tokens. Warning: Using -1 can cause significant memory and network overhead as it returns logprobs for the entire vocabulary (e.g., ~150K tokens for Qwen models) at each prompt position.

  • structured_outputs – Parameter for specifying structured output requirements.

  • reasoning_structured_outputs – Parameter that matches the reasoning part of model output. This will be used to strip the reasoning tokens before applying structured output requirements on the answer content.

  • repetition_detection – Stops generation early with finish_reason “repetition” once the output ends in an N-gram pattern repeated at least min_count times. Not checked before min_tokens tokens are generated.

  • extra_args – Arbitrary additional arguments carried alongside the standard sampling parameters. All extra (undeclared) keys of an API request are collected here. Specific consumers extract the keys they care about; e.g. kv_transfer_params is extracted from this dict.

  • kv_transfer_params – Access point of the remote KV connector, extracted from extra_args["kv_transfer_params"]. There can be one of two ZMQ endpoints ("host:port"): "remote_decoder" and "remote_prefiller". If set, the worker connects to the given endpoint to enable PD-disaggregated prefill or decoding. "id" is the shared identifier of the P-D pair: it must be the same value on both the prefiller and the decoder, and the connector uses it to match the two sides.

class furiosa_llm.RepetitionDetectionParams(max_pattern_size: int = 0, min_pattern_size: int = 0, min_count: int = 0)[source]#

Parameters for detecting repetitive N-gram patterns in output tokens.

These parameters also function as request JSON body parameters (in “repetition_detection” field) in OpenAI-compatible interfaces.

max_pattern_size: int = 0#

Maximum size of N-gram pattern to detect for sequence repetition. Set to 0 to disable. Must be used together with min_count.

min_count: int = 0#

Minimum number of times an N-gram pattern must repeat to trigger detection. Must be >= 2. Must be used together with max_pattern_size.

min_pattern_size: int = 0#

Minimum N-gram pattern size to check for sequence repetition. If set to 0, it defaults to 1. Must be <= max_pattern_size.

Default Sampling Parameters from generation_config.json#

If the model repository or local directory contains a generation_config.json file, Furiosa-LLM uses its values as the effective defaults for the fields listed below. These values are read directly from the source model configuration and are not customized. If the file is absent, the plain SamplingParams() defaults apply.

The following generation_config.json keys are honored, and are mapped to SamplingParams fields as shown:

generation_config.json

SamplingParams

repetition_penalty

repetition_penalty

temperature

temperature

top_k

top_k

top_p

top_p

min_p

min_p

max_new_tokens

max_tokens

stop_strings

stop

Offline API (LLM.generate / LLM.chat / LLM.stream_generate)#

The offline API uses a binary decision based on whether the caller passes sampling_params:

  • sampling_params=None (the default) — the model’s generation_config.json defaults are applied. See furiosa_llm.LLM.get_default_sampling_params().

  • sampling_params=SamplingParams(...) — the user’s object is used as-is. No per-field merge with the model’s generation_config.json is performed.

OpenAI-compatible server#

The Chat Completions, Completions, and Responses endpoints resolve each sampling-related field in three tiers:

  1. The value specified in the request body, if set.

  2. The value from the model’s generation_config.json, if present.

  3. The API default shown in the endpoint’s parameter table (see Chat API (/v1/chat/completions), Completions API (/v1/completions), and API Reference).

Examples#

This section provides examples of how to use the token generation methods available in the SDK.

2. Random Sampling with top_p / top_k Parameters#

SamplingParams(min_tokens=10, max_tokens=100, top_p=0.3, top_k=100)

This method uses random sampling techniques for token generation, allowing for diverse outputs.

  • Parameters:

    • min_tokens: Minimum number of tokens to generate.

    • max_tokens: Maximum number of tokens to generate.

    • top_p: Cumulative probability for nucleus sampling.

    • top_k: Number of highest probability tokens to consider.

  • Behavior:

    • Each generation may yield different results, even with the same input text and parameters, enhancing variability.

    • Generation may terminate before reaching max_tokens if an End Of Sequence (EOS) token is generated.

    • The EOS token will not be generated before reaching the specified min_tokens.

3. Stop Strings#

Pass a list of non-empty strings to stop to end generation when the first string is matched. By default, the matched string is removed from the returned text. A stop-string match sets finish_reason to "stop" and records the matched string in stop_reason:

params = SamplingParams(stop=["<END>"])
output = llm.generate("Write a short answer.", params)[0].outputs[0]
# If the model generates "Answer<END>", output.text is "Answer".
# output.stop_reason is "<END>".

Set include_stop_str_in_output=True to retain the matched string. Text after the match within the completing token is still removed:

params = SamplingParams(
    stop=["<END>"],
    include_stop_str_in_output=True,
)
output = llm.generate("Write a short answer.", params)[0].outputs[0]
# If the model generates "Answer<END>", output.text is "Answer<END>".

For streamed output with the default exclusion behavior, Furiosa-LLM may delay emitting a trailing window up to the longest configured stop string so a partial match does not leak into the output. Retained-stop mode does not need that delay.

4. Repetition Detection#

In some cases, it is known that LLMs may unwantedly fall into repeating the same word or phrases without making progress. The repetition_detection parameter can help detect those cases and stop generation at that point.

This feature detects when a pattern (N-gram, in terms of token) of certain length repeats a certain number of times. Then the generation stops with finish_reason == "repetition".

  • min_pattern_size and max_pattern_size bound the length of patterns to trigger repetition detection inclusively.

    • Both number should be nonnegative, and max_pattern_size should be greater than or equal to min_pattern_size.

    • If min_pattern_size is set to 0, it is effectively identical to a value of 1.

    • If max_pattern_size is set to 0, repetition detection is disabled, and the output will never stop with finish_reason == "repetition".

  • min_count should be >= 2, and denotes the number of times the pattern should repeat to trigger repetition.

This feature can be used as follows.

params = SamplingParams(
    max_tokens=1024,
    repetition_detection=RepetitionDetectionParams(
        max_pattern_size=256,
        min_pattern_size=8,
        min_count=5,
    ),
)
output = llm.chat([{
    "role": "user",
    "content": (
        "Write a very short story, but end it with 100 "
        "repeats of 'And they lived happily ever after.'"
    )
}], sampling_params=params)[0].outputs[0]
# The original output without repetition detection could repeat it forever,
# but the model will now stop after saying "And they ... after." five times.
# Also, output.finish_reason should equal to "repetition", and
# output.stop_reason should equal to "repetition_detected".

Note that repetition detection will incur additional computation with O(max_pattern_size - min_pattern_size) time per token. A large max_pattern_size can therefore affect generation performance negatively.