autoregressive_mixin

autoregressive_mixin

Reusable helpers for cached autoregressive decoders.

CachedAutoregressiveGenerationState dataclass

CachedAutoregressiveGenerationState(step_index, sequence_length, past_key_values, next_inputs, attention_mask=None, cache_position=None, position_ids=None, completed_sequence_mask=None)

State carried by cached autoregressive generation loops.

Parameters:

Name	Type	Description	Default
step_index	int	Number of generated steps already decoded.	required
sequence_length	int	Prefix plus generated sequence length currently in cache.	required
past_key_values	PastKeyValues	Cached transformer key/value state.	required
next_inputs	Tensor	Decoder-specific next-step input tensor. Concrete decoders decide whether it contains token IDs, embeddings, or continuous values.	required
attention_mask	Tensor | None	Optional cache-aware attention mask.	None
cache_position	Tensor | None	Optional cache position tensor for HuggingFace models.	None
position_ids	Tensor | None	Optional position IDs for next_inputs with matching shape up to the last dimension, e.g. (B, 1) for one token.	None
completed_sequence_mask	Tensor | None	Optional boolean mask with shape (B,) where True marks samples that already met their stop condition.	None

AutoregressiveDecoderMixin

Cached autoregressive generation loop.