Introduction

FuriosaAI’s Tensor Contraction Processor (TCP) is a massively parallel AI accelerator targeting inference workloads. Unlike high-level frameworks like PyTorch and XLA, which abstract away memory layouts and hardware scheduling, TCP exposes direct programmer control without requiring the byte-level reasoning of low-level kernel APIs.

TCP’s Virtual Instruction Set Architecture (Virtual ISA, or vISA) is the programming interface that exposes this control. It lets programmers reason in tensors while directly managing memory allocation and tensor unit scheduling. This manual introduces that interface, targeting two audiences: programmers writing vISA directly and compiler developers generating it. Both audiences assume basic Rust familiarity. See the language manual if needed.

Warning

Alpha Test Build: Experimental Software

This software is an early, experimental, and incomplete build intended strictly for technical evaluation and internal testing.

Before using this software for any production work, critical tasks, or for important data, you must consult with Furiosa engineers.

Your feedback is vital to our development. Please provide it.

Installation

Install three pieces:

1. Rust toolchain (pinned): the Furiosa optimizer is a rustc driver, ABI-locked to a specific nightly.

   rustup toolchain install nightly-2026-05-01
   

   The same channel is pinned in rust-toolchain.toml; cargo activates it automatically when you cd into a project that includes that file.

2. cargo-furiosa-opt: the cargo subcommand that injects the right --cfg backend="..." and pre-compiles kernels for NPU.

   cargo +nightly-2026-05-01 install cargo-binstall
   cargo +nightly-2026-05-01 binstall cargo-furiosa-opt
   

3. Furiosa SDK + physical NPU (only for --backend npu): the NPU backend dispatches to real hardware via the SDK’s kernel driver and PE runtime (furiosa-driver-rngd, furiosa-smi, etc.; see the SDK documentation).

   The emulation and typecheck backends do not require the SDK. They run host-side with no NPU dependency, so a customer who only intends to develop or evaluate kernels does not need to install the SDK at all.

Your First Program

Use cargo-generate to scaffold a fresh project from the base-template starter, which ships with the five worked examples covered in the Quick Start chapter:

cargo install cargo-generate
cargo generate furiosa-ai/furiosa-opt base-template
cd base-template

Layout

base-template/
├── Cargo.toml                               # `[package.metadata.furiosa-opt]` marks it a kernel package
├── README.md
├── rust-toolchain.toml
└── src/
    ├── lib.rs                                # `pub mod kernel;`
    ├── kernel/                               # every #[device] function lives here
    │   ├── mod.rs                            # `pub mod {constant_add,...}_kernel;`
    │   ├── constant_add_kernel.rs            # `#[device] fn constant_add_kernel(...)`
    │   ├── elementwise_mul_kernel.rs
    │   ├── dot_product_kernel.rs
    │   ├── gemv_kernel.rs
    │   └── gemm_kernel.rs
    ├── constant_add.rs                       # host binary that `launch()`es its kernel
    ├── elementwise_mul.rs
    ├── dot_product.rs
    ├── gemv.rs
    └── gemm.rs

Keep these layout rules intact:

• A package opts into kernel compilation by declaring [package.metadata.furiosa-opt] in its Cargo.toml.
• Host programs should live as direct src/*.rs files and are registered with explicit [[bin]] path = "src/<name>.rs" entries in Cargo.toml.
• Do not move host programs into src/bin/, examples/, or tests/; the rustc plugin scans cargo targets rooted at src/ and skips those other locations.

All five kernel examples live under src/kernel/ and are re-exported through src/kernel/mod.rs and src/lib.rs. Each binary’s main() only calls launch(kernel, ...). The value comparison against a host-side reference lives in a #[cfg(test)] mod tests block inside the same file.

Run a worked example

# Host-side emulation (default; no NPU hardware required).
cargo furiosa-opt run --release --bin gemm

# Mapping/shape verification only — kernel body runs against phantom (empty) tensors.
cargo furiosa-opt --backend typecheck run --release --bin gemm

# Real NPU dispatch (requires the SDK and a physical NPU; see Installation step 3).
cargo furiosa-opt --backend npu run --release --bin gemm

Verify against the reference

# Full numeric comparison on emulated values.
cargo furiosa-opt test --release --bin gemm

# Under typecheck the comparison loop trivially passes: `actual` is the
# phantom-empty Vec, so the per-element assertion has zero iterations.
cargo furiosa-opt --backend typecheck test --release --bin gemm

Add a Kernel

1. Drop src/kernel/<name>_kernel.rs with a #[device(...)] pub fn <name>_kernel(...).
2. Append pub mod <name>_kernel; to src/kernel/mod.rs.
3. Add src/<name>.rs as the host program that calls launch(<name>_kernel, ...).
4. Register a matching [[bin]] entry in Cargo.toml with path = "src/<name>.rs".
5. Run your kernel with cargo furiosa-opt run --release --bin <name>.

Development Tools

The Furiosa IR Optimizer provides utilities for developing, testing, and optimizing vISA programs on TCP devices. It complements the Furiosa SDK’s compiler by giving developers fine-grained control over program behavior, whether the programmer writes vISA by hand or a compiler generates it.

Backends

A vISA program is a Rust program that uses the furiosa-opt-std API. cargo furiosa-opt selects which backend evaluates the kernel by setting --cfg backend="...":

# Default: emulation backend, no NPU hardware needed.
cargo furiosa-opt run --release

# Fast mapping/shape verification (kernel body runs with phantom tensors).
cargo furiosa-opt --backend typecheck run --release

# Real NPU dispatch (requires the SDK and a physical NPU).
cargo furiosa-opt --backend npu run --release

cargo check (under any backend) only runs the type checker; it does not execute kernel function bodies and therefore cannot reach mapping assertions like Collect output packet must be exactly 32 bytes. Use --backend typecheck run for that.

cargo furiosa-opt forwards every cargo flag verbatim, so cargo run, cargo test, cargo check, and cargo build all have direct equivalents:

cargo furiosa-opt build              # cargo build with emulation backend
cargo furiosa-opt --backend npu test # cargo test on real NPU

See the cargo furiosa-opt appendix for the complete command reference.

Language Server

furiosa-rust-analyzer-proxy is a proxy for rust-analyzer that provides standard Rust IDE features with enhanced support for mapping expressions. It keeps the usual rust-analyzer experience while simplifying verbose types like Stride<Symbol<A>, 8> into readable mapping expressions like m![A / 8].

For installation and configuration, see the Language Server appendix.

Schedule Viewer

The Schedule Viewer visualizes the execution timeline to help identify performance bottlenecks. Use furiosa-opt to export a schedule JSON file, then open it with furiosa-schedule-viewer.

For installation and usage, see the Schedule Viewer appendix.

Book Organization

Each chapter builds on the previous: mapping and moving tensors establish the data model, computing tensors covers the pipeline engines, and scheduling and kernel examples show how to compose them into real programs.

• Quick Start: How vISA programming works, introduced through worked examples covering element-wise operations and tensor contractions.
• Mapping Tensors: How logical tensors map to physical memory: axis layout, stride, padding, and tiling.
• Moving Tensors: How data moves between memory tiers (HBM, DM) and the Tensor Unit via Fetch, Commit, and DMA engines.
• Computing Tensors: How the Tensor Unit pipeline (Switch, Collect, Contraction, Vector, Cast, Transpose) transforms data each cycle.
• Scheduling: How operations are ordered and executed concurrently across contexts.
• Kernel Examples: End-to-end examples showing how mapping, movement, computation, and scheduling combine into real kernels.

License

This documentation and the entire furiosa-opt repository are licensed under the Apache License Version 2.0.

Quick Start

This chapter explains TCP through five running examples, each introducing one new hardware concept. The first two examples cover element-wise operations. The remaining three cover tensor contractions (dot product, GEMV, and GEMM).

Mathematical Background

TCP is a tensor-native processor built to accelerate tensor contraction.

Tensor

A tensor is a mapping from a tensor index to a value, where the tensor’s shape defines the valid indices.

A shape is an unordered set of named axes. The shapes \(\{\texttt{N} = 4, \texttt{C} = 3\}\) and \(\{\texttt{C} = 3, \texttt{N} = 4\}\) identify the same tensor: axis names carry the meaning, not the position. A tensor index is formed by specifying an index value for each axis. For shape \(\{\texttt{N} = 4, \texttt{C} = 3\}\), the valid indices are \(\{\texttt{N}: 0, \texttt{C}: 0\}\), \(\{\texttt{N}: 0, \texttt{C}: 1\}\), \(\{\texttt{N}: 0, \texttt{C}: 2\}\), \(\{\texttt{N}: 1, \texttt{C}: 0\}\), and so on.

Once an axis ordering is chosen, a tensor behaves like a familiar multi-dimensional array, similar to NumPy’s ndarray:

• 0D Tensor (Scalar): a single number like \(5.2\)
• 1D Tensor (Vector): a sequence like \([1, 2, 3]\) with one axis
• 2D Tensor (Matrix): a \(2 \times 4\) grid with two axes
• 4D Tensor: a batch of RGB images with shape \(\{\texttt{N} = 4, \texttt{C} = 3, \texttt{H} = 256, \texttt{W} = 512\}\)

Tensor Contraction

A tensor contraction generalizes matrix multiplication to arbitrary tensors: two input tensors are multiplied element-wise and summed along their shared (contracted) axes. Every contraction decomposes into three steps: Broadcast, Multiply, Reduce. Einsum notation expresses contractions compactly: list each input tensor by its axis labels, output axes follow the → arrow, and any input axis absent from the output is contracted.

The following table shows three contractions, with their einsum notation and Broadcast-Multiply-Reduce decomposition:

Tensor Contraction Processor

Hardware Hierarchy

A TCP device consists of four nested hardware levels:

Tensor Unit

The Tensor Unit is a fixed pipeline: Fetch → Switch → Collect → Contraction → Vector → Cast → Transpose → Commit. Most stages operate independently within each slice. The Switch Engine is the exception, connecting slices to distribute data across the slice array.

See Computing Tensors for .contract_outer(), .contract_packet(), .contract_time(), .contract_lane(), .cast(), .switch(), .vector_fxp(), and each engine in the pipeline.

Memory Tiers

See Moving Tensors for .to_dm(), .to_hbm(), .fetch(), .commit(), and the complete memory tier model.

Tensor Mapping

TCP’s Virtual ISA exposes the hardware hierarchy through its type system. Each tensor type encodes the element type and how each logical axis distributes across the hardware hierarchy.

For example, DmTensor<bf16, m![1], m![1 # 2], m![A / 8 # 256], m![A % 8]> (with axes![A = 2048]) represents a bf16 vector with the axis A on one chip (m![1]), one of two clusters (m![1 # 2]), distributed across 256 slices (m![A / 8 # 256]) with 8 elements per slice (m![A % 8]). Each element of A therefore maps to a well-defined position within exactly one slice.

Three operators in m![] build this distribution:

• / splits by stride: A / 8 gives 2048 / 8 = 256 slice indices.
• % gives the inner count: A % 8 gives the 8 in-slice indices.
• # pads to the hardware unit count: # 256 pads to 256 slices, with any excess slots holding arbitrary values.

TCP also introduces two parameters for tensors flowing through the Tensor Unit pipeline: Time indexes pipeline iterations; Packet indexes elements within each iteration.

See Mapping Tensors for axes![], m![], HbmTensor, DmTensor, and the full mapping expression reference.

Execution Contexts

Every device kernel has two execution contexts running concurrently on separate hardware resources: ctx.main and ctx.sub. main runs the primary computation. sub runs a concurrent pipeline, typically used to prefetch operands into TRF or VRF while main computes. If main needs operands that sub is still fetching, main automatically waits for sub’s execution to ensure synchronization.

Both contexts share the same flat on-chip SRAM. DM addresses are optional: omit them (as in .to_dm() and .commit()) to let placement be assigned automatically, or pin a specific address with the _at variants (.to_dm_at(addr), .commit_at(addr)) when a kernel needs explicit, non-overlapping control.

In type signatures, the const-generic Tu identifies which context a tensor flows through ({ Tu::Main } or { Tu::Sub }).

See Scheduling for ctx.main, ctx.sub, launch(), and how operations are scheduled and run in parallel across contexts.

Kernel Examples

Constant Addition

The first kernel takes a vector of integers and adds the constant 1 to each element. It uses one chip, one of two clusters, and all 256 slices in that cluster, with one 8-element group per slice. Adding 1 to each element uses the Vector Engine’s fixed-point operation vector_fxp(FxpBinaryOp::AddFxp, 1).

flowchart TB
    HOST[Host] <-->|PCIe DMA| HBM[(HBM)]
    HBM <-->|Tensor DMA| DM[(DM)]

    subgraph TU[Tensor Unit]
        direction TB
        FE[Fetch] --> CO[Collect] --> VE["Vector (AddFxp +1)"] --> CM[Commit]
    end

    DM -->|stream| FE
    CM -->|stream| DM

to_dm moves data from HBM to DM, splitting the flat tensor across 256 slices. The begin → fetch → collect → vector_init → vector_intra_slice_tag → vector_fxp → vector_final → commit chain processes each slice in one pass. TagMode::Zero configures the pipeline to execute on every cycle.

Kernel (src/kernel/constant_add_kernel.rs):

use furiosa_opt_std::prelude::*;

axes![A = 2048];

pub type Chip = m![1];
pub type Cluster = m![1 # 2];
pub type Slice = m![A / 8 # 256];

#[device(chip = 1)]
pub fn constant_add_kernel(ctx: &mut Context, input: &HbmTensor<i32, Chip, m![A]>) -> HbmTensor<i32, Chip, m![A]> {
    // HBM → DM: split 2048 elements across 256 slices (8 elements per slice)
    let dm = input.to_dm::<Cluster, Slice, m![A % 8]>(&mut ctx.tdma);

    let result = ctx
        .main
        .begin(dm.view())
        // Fetch: stream 8-element packets from DM into the pipeline
        .fetch::<m![1], m![A % 8]>()
        // Collect: normalize the stream into 32-byte flits (8 × i32)
        .collect::<m![1], m![A % 8]>()
        // Vector Engine: enter pipeline and arm unconditionally
        .vector_init()
        .vector_intra_slice_tag(TagMode::Zero)
        // Add the scalar constant 1 to every element
        .vector_fxp(FxpBinaryOp::AddFxp, 1)
        // Exit VE and commit: trim the packet to the commit width, then write
        // results back to DM
        .vector_final()
        .commit_trim::<m![A % 8]>()
        .commit::<m![A % 8]>();

    // DM → HBM
    result.to_hbm(&mut ctx.tdma)
}

Host program (src/constant_add.rs):

use furiosa_opt_std::prelude::*;
use {{ crate_name }}::kernel::constant_add_kernel::{A, constant_add_kernel};
use rand::SeedableRng;
use rand::rngs::SmallRng;

#[tokio::main]
async fn main() {
    let mut ctx = Context::acquire();
    let mut rng = SmallRng::seed_from_u64(42);
    let input = HostTensor::<i32, m![A]>::rand(&mut rng);
    let in_hbm = input.to_hbm(&mut ctx.pdma).await;
    let _out_hbm = launch(constant_add_kernel, (&mut ctx, &in_hbm)).await;
    println!("Constant Add: kernel ran");
}

#[cfg(test)]
mod tests {
    use super::*;

    #[tokio::test]
    async fn matches_reference() {
        let mut ctx = Context::acquire();

        let mut rng = SmallRng::seed_from_u64(42);
        let input = HostTensor::<i32, m![A]>::rand(&mut rng);
        let in_hbm = input.to_hbm(&mut ctx.pdma).await;

        // Reference: out[i] = in[i] + 1.
        let in_buf: Vec<i32> = input.into_vec();
        let expected: Vec<i32> = in_buf.iter().map(|&x| x.wrapping_add(1)).collect();

        let out_hbm = launch(constant_add_kernel, (&mut ctx, &in_hbm)).await;

        // Under the typecheck backend `actual` is empty (phantom tensors), so
        // the loop trivially runs zero iterations and the assertion is skipped.
        let actual: Vec<i32> = out_hbm.to_host::<m![A]>(&mut ctx.pdma).await.into_vec();
        for (i, (&e, &a)) in expected.iter().zip(&actual).enumerate() {
            assert_eq!(e, a, "constant_add mismatch at i={i}: expected {e}, actual {a}");
        }
    }
}

Elementwise Multiplication

The second kernel multiplies two same-shape vectors element-wise. One operand flows through the pipeline. The other is stored in the VRF (Vector Register File), a per-slice register file that the Vector Engine reads every cycle.

flowchart TB
    LHS_HBM[(lhs: HBM)] -->|Tensor DMA| LHS_DM[(lhs: DM)]
    RHS_HBM[(rhs: HBM)] -->|Tensor DMA| RHS_DM[(rhs: DM)]

    subgraph sub[sub context]
        direction LR
        sFE[Fetch] --> sCO[Collect]
    end

    subgraph main[main context]
        direction LR
        mFE[Fetch] --> mCO[Collect] --> VE["Vector (MulInt)"] --> CM[Commit]
    end

    RHS_DM --> sFE
    LHS_DM --> mFE
    sCO --> VRF[(VRF)]
    VRF --> VE
    CM --> OUT_DM[(result: DM)]
    OUT_DM -->|Tensor DMA| OUT_HBM[(HBM)]

This example introduces the sub context, which preloads one operand into the VRF while the main context streams. The sub context loads rhs_dm into the VRF through the Fetch → Collect → .to_vrf() pipeline. rhs_dm occupies a DM region disjoint from lhs_dm so the two do not overlap. The main context then streams lhs_dm and multiplies each element by its VRF counterpart using MulInt. The hardware runs both contexts concurrently where possible.

Kernel (src/kernel/elementwise_mul_kernel.rs):

use furiosa_opt_std::prelude::*;

axes![A = 2048];

pub type Chip = m![1];
pub type Cluster = m![1 # 2];
pub type Slice = m![A / 8 # 256];

#[device(chip = 1)]
pub fn elementwise_mul_kernel(
    ctx: &mut Context,
    lhs: &HbmTensor<i32, Chip, m![A]>,
    rhs: &HbmTensor<i32, Chip, m![A]>,
) -> HbmTensor<i32, Chip, m![A]> {
    // Move both operands from HBM to DM (DM placement is assigned automatically).
    let lhs_dm = lhs.to_dm::<Cluster, Slice, m![A % 8]>(&mut ctx.tdma);
    let rhs_dm = rhs.to_dm::<Cluster, Slice, m![A % 8]>(&mut ctx.tdma);

    // Sub context: load rhs into VRF (runs concurrently with the main context below).
    // VRF holds a per-slice operand that the Vector Engine reads every cycle.
    let rhs_vrf: VrfTensor<i32, Chip, Cluster, Slice, m![A % 8]> = ctx
        .sub
        .begin(rhs_dm.view())
        .fetch::<m![1], m![A % 8]>()
        .collect::<m![A % 8 / 8], m![A % 8 % 8]>()
        .to_vrf();

    // Main context: multiply every lhs element by its rhs counterpart from VRF
    let result = ctx
        .main
        .begin(lhs_dm.view())
        .fetch::<m![1], m![A % 8]>()
        .collect::<m![1], m![A % 8]>()
        .vector_init()
        .vector_intra_slice_tag(TagMode::Zero)
        // Each slice multiplies its 8 lhs elements by the matching 8 rhs elements in VRF
        .vector_fxp(FxpBinaryOp::MulInt, &rhs_vrf)
        .vector_final()
        .commit_trim::<m![A % 8]>()
        .commit::<m![A % 8]>();

    result.to_hbm(&mut ctx.tdma)
}

Host program (src/elementwise_mul.rs):

use furiosa_opt_std::prelude::*;
use {{ crate_name }}::kernel::elementwise_mul_kernel::{A, elementwise_mul_kernel};
use rand::SeedableRng;
use rand::rngs::SmallRng;

#[tokio::main]
async fn main() {
    let mut ctx = Context::acquire();
    let mut rng = SmallRng::seed_from_u64(42);
    let lhs = HostTensor::<i32, m![A]>::rand(&mut rng);
    let rhs = HostTensor::<i32, m![A]>::rand(&mut rng);
    let lhs_hbm = lhs.to_hbm(&mut ctx.pdma).await;
    let rhs_hbm = rhs.to_hbm(&mut ctx.pdma).await;
    let _out_hbm = launch(elementwise_mul_kernel, (&mut ctx, &lhs_hbm, &rhs_hbm)).await;
    println!("Elementwise Mul: kernel ran");
}

#[cfg(test)]
mod tests {
    use super::*;

    #[tokio::test]
    async fn matches_reference() {
        let mut ctx = Context::acquire();

        let mut rng = SmallRng::seed_from_u64(42);
        let lhs = HostTensor::<i32, m![A]>::rand(&mut rng);
        let rhs = HostTensor::<i32, m![A]>::rand(&mut rng);

        let lhs_hbm = lhs.to_hbm(&mut ctx.pdma).await;
        let rhs_hbm = rhs.to_hbm(&mut ctx.pdma).await;

        // Reference: out[i] = lhs[i] * rhs[i].
        let lhs_buf: Vec<i32> = lhs.into_vec();
        let rhs_buf: Vec<i32> = rhs.into_vec();
        let expected: Vec<i32> = lhs_buf.iter().zip(&rhs_buf).map(|(&a, &b)| a.wrapping_mul(b)).collect();

        let out_hbm = launch(elementwise_mul_kernel, (&mut ctx, &lhs_hbm, &rhs_hbm)).await;

        let actual: Vec<i32> = out_hbm.to_host::<m![A]>(&mut ctx.pdma).await.into_vec();
        for (i, (&e, &a)) in expected.iter().zip(&actual).enumerate() {
            assert_eq!(e, a, "elementwise_mul mismatch at i={i}: expected {e}, actual {a}");
        }
    }
}

Dot Product

The dot product \(I, I \rightarrow 1\) reduces both operands along the same axis with no broadcast step. As in the previous example, one operand flows through the pipeline. The other is held stationary in the TRF (Tensor Register File), a per-slice register file that the Contraction Engine reads each cycle. The sub context loads rhs into the TRF via Fetch → Collect → .to_trf(). TrfAddress::Full dedicates the entire TRF to this tensor. .contract_outer() invokes the Contraction Engine’s Stream Adapter and the TRF Sequencer. The Stream Adapter pairs adjacent 32-byte flits into the Outer stage’s 64-byte packet; the TRF Sequencer reads the stationary RHS. Both feed the elementwise multiplier per lane. .contract_packet() reduce-adds those products spatially via the hardware reduction tree. .contract_time::<m![1]>() then accumulates temporally, producing a scalar per slice. .contract_lane() folds the 8 lanes into the output (trivial fold here at Lane = m![1]). .cast() converts the f32 accumulator output back to bf16.

Kernel (src/kernel/dot_product_kernel.rs):

use furiosa_opt_std::prelude::*;

axes![A = 2048];

pub type Chip = m![1];
pub type Cluster = m![1 # 2];
pub type Slice = m![1 # 256]; // 1 active slice; m![A / 8 # 256] would distribute across all 256
pub type Time = m![1]; // No temporal iteration
pub type Lane = m![1]; // No lane parallelism

#[device(chip = 1)]
pub fn dot_product_kernel(
    ctx: &mut Context,
    lhs: &HbmTensor<bf16, Chip, m![A]>,
    rhs: &HbmTensor<bf16, Chip, m![A]>,
) -> HbmTensor<bf16, Chip, m![1]> {
    // HBM → DM
    let lhs: DmTensor<bf16, Chip, Cluster, Slice, m![A]> = lhs.to_dm(&mut ctx.tdma);
    let rhs: DmTensor<bf16, Chip, Cluster, Slice, m![A]> = rhs.to_dm(&mut ctx.tdma);

    // Sub context: load rhs into TRF (TrfAddress::Full dedicates the entire TRF to this tensor)
    let rhs: TrfTensor<bf16, Chip, Cluster, Slice, Lane, m![A]> = ctx
        .sub
        .begin(rhs.view())
        .fetch::<Time, m![A]>()
        .collect::<m![{ Time }, A / 16], m![A % 16]>()
        .to_trf();

    // Main context: stream lhs through the Contraction Engine, reduce along A
    let result: DmTensor<bf16, Chip, Cluster, Slice, m![1 # 8]> = ctx
        .main
        .begin(lhs.view())
        .fetch::<Time, m![A]>()
        .collect::<m![A / 16], m![A % 16]>()
        // Pair consecutive 32-byte flits into 64-byte packets, halving time steps (A/16 → A/32)
        .contract_outer::<m![A / 32], m![A % 32], _, _, _>(&rhs)
        .contract_packet::<m![1]>()
        .contract_time::<m![1]>()
        .contract_lane::<m![1], m![1 # 8]>(LaneMode::Interleaved)
        .cast::<bf16, m![1 # 16]>() // cast f32 accumulator output back to bf16
        .commit_trim::<m![1 # 8]>()
        .commit();

    // DM → HBM
    result.to_hbm(&mut ctx.tdma)
}

Host program (src/dot_product.rs):

use furiosa_opt_std::prelude::*;
use {{ crate_name }}::kernel::dot_product_kernel::{A, dot_product_kernel};
use rand::SeedableRng;
use rand::rngs::SmallRng;

#[tokio::main]
async fn main() {
    let mut ctx = Context::acquire();
    let mut rng = SmallRng::seed_from_u64(42);
    let lhs = HostTensor::<bf16, m![A]>::rand(&mut rng);
    let rhs = HostTensor::<bf16, m![A]>::rand(&mut rng);
    let lhs_hbm = lhs.to_hbm(&mut ctx.pdma).await;
    let rhs_hbm = rhs.to_hbm(&mut ctx.pdma).await;
    let _out_hbm = launch(dot_product_kernel, (&mut ctx, &lhs_hbm, &rhs_hbm)).await;
    println!("Dot Product: kernel ran");
}

#[cfg(test)]
mod tests {
    use super::*;

    #[tokio::test]
    async fn matches_reference() {
        let mut ctx = Context::acquire();

        let mut rng = SmallRng::seed_from_u64(42);
        let lhs = HostTensor::<bf16, m![A]>::rand(&mut rng);
        let rhs = HostTensor::<bf16, m![A]>::rand(&mut rng);

        let lhs_hbm = lhs.to_hbm(&mut ctx.pdma).await;
        let rhs_hbm = rhs.to_hbm(&mut ctx.pdma).await;

        // Reference: sum_i lhs[i] * rhs[i] in f32, then round to bf16.
        let lhs_buf: Vec<bf16> = lhs.into_vec();
        let rhs_buf: Vec<bf16> = rhs.into_vec();
        let expected_f32: f32 = lhs_buf
            .iter()
            .zip(&rhs_buf)
            .map(|(&a, &b)| f32::from(a) * f32::from(b))
            .sum();
        let expected = bf16::from_f32(expected_f32);

        let out_hbm = launch(dot_product_kernel, (&mut ctx, &lhs_hbm, &rhs_hbm)).await;

        let actual_buf: Vec<bf16> = out_hbm.to_host::<m![1]>(&mut ctx.pdma).await.into_vec();
        if let Some(&actual) = actual_buf.first() {
            let diff = (f32::from(actual) - f32::from(expected)).abs();
            let tol = (0.02 * f32::from(expected).abs()).max(0.5);
            assert!(
                diff <= tol,
                "dot_product mismatch: expected {expected:?}, actual {actual:?}, diff {diff} > tol {tol}"
            );
        }
    }
}

GEMV

GEMV \(IJ, J \rightarrow I\) distributes the output dimension I across slices: each slice computes one row \(y_i = \sum_j A_{ij} x_j\). Unlike the dot product (where all slices reduce along the same axis and no redistribution is needed), here each slice needs the full vector to contract against its row, so data must be broadcast across slices before the contraction.

Kernel (src/kernel/gemv_kernel.rs):

use furiosa_opt_std::prelude::*;

axes![I = 256, J = 2048];

pub type Chip = m![1];
pub type Cluster = m![1 # 2];
pub type Slice = m![I]; // Distribute output dimension across slices
pub type Time = m![J / 32]; // Temporal iterations for reduction dimension
pub type Packet = m![J % 32]; // Packet size for reduction dimension
pub type Lane = m![1];

#[device(chip = 1)]
pub fn gemv_kernel(
    ctx: &mut Context,
    matrix: &HbmTensor<bf16, Chip, m![I, J]>,
    vector: &HbmTensor<bf16, Chip, m![J]>,
) -> HbmTensor<bf16, Chip, m![I]> {
    // Move data from HBM to DM
    let matrix: DmTensor<bf16, Chip, Cluster, Slice, m![J]> = matrix.to_dm(&mut ctx.tdma);
    let vector: DmTensor<bf16, Chip, Cluster, Slice, m![J]> = vector.to_dm(&mut ctx.tdma);

    // Load vector into TRF
    let vector_trf: TrfTensor<bf16, Chip, Cluster, Slice, Lane, m![J]> = ctx
        .sub
        .begin(vector.view())
        .fetch::<m![1], m![J]>()
        // Collect Engine: split into 32-byte flits.
        .collect::<m![J / 16], m![J % 16]>()
        .to_trf();

    // Compute GEMV: matrix × vector
    // Key difference: `I` maps to slice (preserved), `J` gets reduced
    let result: DmTensor<bf16, Chip, Cluster, Slice, m![1 # 4]> = ctx
        .main
        .begin(matrix.view())
        .fetch::<m![J / 16], m![J % 16]>()
        .collect::<m![J / 16], m![J % 16]>()
        .contract_outer::<Time, Packet, _, _, _>(&vector_trf)
        .contract_packet::<m![1]>()
        .contract_time::<m![1]>()
        .contract_lane::<m![1], m![1 # 8]>(LaneMode::Interleaved)
        .cast::<bf16, m![1 # 16]>()
        .commit_trim::<m![1 # 4]>()
        .commit();

    // Transfer result to HBM
    result.to_hbm(&mut ctx.tdma)
}

Host program (src/gemv.rs):

use furiosa_opt_std::prelude::*;
use {{ crate_name }}::kernel::gemv_kernel::{I, J, gemv_kernel};
use rand::SeedableRng;
use rand::rngs::SmallRng;

#[tokio::main]
async fn main() {
    let mut ctx = Context::acquire();
    let mut rng = SmallRng::seed_from_u64(42);
    let matrix = HostTensor::<bf16, m![I, J]>::rand(&mut rng);
    let vector = HostTensor::<bf16, m![J]>::rand(&mut rng);
    let matrix_hbm = matrix.to_hbm(&mut ctx.pdma).await;
    let vector_hbm = vector.to_hbm(&mut ctx.pdma).await;
    let _out_hbm = launch(gemv_kernel, (&mut ctx, &matrix_hbm, &vector_hbm)).await;
    println!("GEMV: kernel ran");
}

#[cfg(test)]
mod tests {
    use super::*;

    #[tokio::test]
    async fn matches_reference() {
        let mut ctx = Context::acquire();

        let mut rng = SmallRng::seed_from_u64(42);
        let matrix = HostTensor::<bf16, m![I, J]>::rand(&mut rng);
        let vector = HostTensor::<bf16, m![J]>::rand(&mut rng);

        let matrix_hbm = matrix.to_hbm(&mut ctx.pdma).await;
        let vector_hbm = vector.to_hbm(&mut ctx.pdma).await;

        // Reference: y[i] = sum_j matrix[i, j] * vector[j] in f32, rounded to bf16.
        let mat_buf: Vec<bf16> = matrix.into_vec();
        let vec_buf: Vec<bf16> = vector.into_vec();
        let expected: Vec<bf16> = mat_buf
            .chunks(J::SIZE)
            .map(|row| {
                let acc: f32 = row
                    .iter()
                    .zip(&vec_buf)
                    .map(|(&a, &b)| f32::from(a) * f32::from(b))
                    .sum();
                bf16::from_f32(acc)
            })
            .collect();

        let out_hbm = launch(gemv_kernel, (&mut ctx, &matrix_hbm, &vector_hbm)).await;

        let actual: Vec<bf16> = out_hbm.to_host::<m![I]>(&mut ctx.pdma).await.into_vec();
        for (i, (&e, &a)) in expected.iter().zip(&actual).enumerate() {
            let diff = (f32::from(a) - f32::from(e)).abs();
            let tol = (0.02 * f32::from(e).abs()).max(0.5);
            assert!(
                diff <= tol,
                "gemv mismatch at i={i}: expected {e:?}, actual {a:?}, diff {diff} > tol {tol}"
            );
        }
    }
}

GEMM

GEMM \(IK, JK \rightarrow IJ\) adds a second output dimension: both \(I\) and \(J\) appear in the output \(C_{ij} = \sum_k A_{ik} B_{jk}\). Each matrix broadcasts along its missing output dimension: \(A\) broadcasts across \(J\) and \(B\) broadcasts across \(I\).

The new concept is type Slice = m![I / 32, J / 32], which jointly maps both output dimensions to Slice so each slice computes a 16 × 16 tile of the output matrix. The Switch Engine moves each tile of B to the matching slice, so each slice sees only its portion of J. .contract_packet::<m![1]>() reduces along K spatially. .contract_time::<m![I]>() accumulates over time (preserving I), and .contract_lane::<m![I], m![J # 8]>(LaneMode::Interleaved) folds Lane into the output packet, preserving both I and J in the output.

Kernel (src/kernel/gemm_kernel.rs):

use furiosa_opt_std::prelude::*;

axes![I = 512, J = 512, K = 64];

pub type Chip = m![1];
pub type Cluster = m![1 # 2];
// Distribute output dimensions `I` and `J` across slices
pub type Slice = m![I / 32, J / 32]; // Each slice handles a 16 × 16 output tile
pub type Lane = m![J % 8];

#[device(chip = 1)]
pub fn gemm_kernel(
    ctx: &mut Context,
    a: &HbmTensor<bf16, Chip, m![I, K]>,
    b: &HbmTensor<bf16, Chip, m![J, K]>,
) -> HbmTensor<bf16, Chip, m![I, J]> {
    // Move data from HBM to DM
    let a: DmTensor<bf16, Chip, Cluster, Slice, m![I % 32, K]> = a.to_dm(&mut ctx.tdma);
    let b: DmTensor<bf16, Chip, Cluster, Slice, m![J % 32, K]> = b.to_dm(&mut ctx.tdma);

    // Load matrix B into TRF
    // Switch Engine distributes B across 256 slices
    // Each slice gets the full `K` dimension but only its (16 × 16) output tile
    // See: Switch Engine topologies for details on distribution
    let b_trf: TrfTensor<bf16, Chip, Cluster, Slice, Lane, m![J / 8 % 4, K]> = ctx
        .sub
        .begin(b.view())
        .fetch::<m![J % 8, J / 8 % 4], m![K]>()
        .collect::<m![J % 8, J / 8 % 4, K / 16], m![K % 16]>()
        .to_trf();

    // Compute GEMM: A × B
    // Switch Engine ensures matching (`I / 32`, `J / 32`) slice distribution
    // Contraction reduces along `K`, preserves `I` and `J`
    let result: DmTensor<bf16, Chip, Cluster, Slice, m![I % 32, J % 32]> = ctx
        .main
        .begin(a.view())
        .fetch::<m![I % 32, J / 8 % 4], m![K]>()
        .collect::<m![I % 32, J / 8 % 4, K / 16], m![K % 16]>()
        .contract_outer::<m![I % 32, J / 8 % 4, K / 32], m![K % 32], _, _, _>(&b_trf)
        .contract_packet::<m![1]>()
        .contract_time::<m![I % 32, J / 8 % 4]>()
        .contract_lane::<m![I % 32, J / 8 % 4], m![J % 8]>(LaneMode::Interleaved)
        .cast::<bf16, m![J % 8 # 16]>()
        .commit_trim::<m![J % 8]>()
        .commit();

    // Transfer result to HBM
    result.to_hbm(&mut ctx.tdma)
}

Host program (src/gemm.rs):

use furiosa_opt_std::prelude::*;
use {{ crate_name }}::kernel::gemm_kernel::{I, J, K, gemm_kernel};
use rand::SeedableRng;
use rand::rngs::SmallRng;

#[tokio::main]
async fn main() {
    let mut ctx = Context::acquire();
    let mut rng = SmallRng::seed_from_u64(42);
    let a = HostTensor::<bf16, m![I, K]>::rand(&mut rng);
    let b = HostTensor::<bf16, m![J, K]>::rand(&mut rng);
    let a_hbm = a.to_hbm(&mut ctx.pdma).await;
    let b_hbm = b.to_hbm(&mut ctx.pdma).await;
    let _out_hbm = launch(gemm_kernel, (&mut ctx, &a_hbm, &b_hbm)).await;
    println!("GEMM: kernel ran");
}

#[cfg(test)]
mod tests {
    use super::*;

    #[tokio::test]
    async fn matches_reference() {
        let mut ctx = Context::acquire();

        let mut rng = SmallRng::seed_from_u64(42);
        let a = HostTensor::<bf16, m![I, K]>::rand(&mut rng);
        let b = HostTensor::<bf16, m![J, K]>::rand(&mut rng);

        let a_hbm = a.to_hbm(&mut ctx.pdma).await;
        let b_hbm = b.to_hbm(&mut ctx.pdma).await;

        // Reference: C[i, j] = sum_k A[i, k] * B[j, k] in f32, rounded to bf16.
        let a_buf: Vec<bf16> = a.into_vec();
        let b_buf: Vec<bf16> = b.into_vec();
        let expected: Vec<bf16> = a_buf
            .chunks(K::SIZE)
            .flat_map(|a_row| {
                b_buf.chunks(K::SIZE).map(move |b_row| {
                    let acc: f32 = a_row
                        .iter()
                        .zip(b_row)
                        .map(|(&a, &b)| f32::from(a) * f32::from(b))
                        .sum();
                    bf16::from_f32(acc)
                })
            })
            .collect();

        let out_hbm = launch(gemm_kernel, (&mut ctx, &a_hbm, &b_hbm)).await;

        let actual: Vec<bf16> = out_hbm.to_host::<m![I, J]>(&mut ctx.pdma).await.into_vec();
        for (idx, (&e, &av)) in expected.iter().zip(&actual).enumerate() {
            let diff = (f32::from(av) - f32::from(e)).abs();
            let tol = (0.05 * f32::from(e).abs()).max(1.0);
            assert!(diff <= tol, "gemm mismatch at idx={idx}: expected {e:?}, actual {av:?}");
        }
    }
}

The examples above process tensors that fit in a single hardware pass. Real workloads often require partitioning into tiles. Two complementary strategies exist for workloads exceeding the 512 KB/slice DM capacity: temporal partitioning processes tiles sequentially over time, and spatial partitioning distributes tiles across parallel hardware units.

See Kernel Examples for end-to-end kernels that apply these strategies.

Mapping Tensors

Tensors have no intrinsic order of elements. A mapping assigns each tensor index to a buffer position, defining that order.

This chapter shows how layout choices affect memory access and explains TCP’s declarative approach to tensor layout with mapping expressions.

Layout and Performance

The mapping determines access efficiency: hardware reads memory in contiguous blocks, so elements stored far apart require more transfers. This choice affects programs too: programmers choose which axis is major (outermost, changes slowest) and which is minor (innermost, changes fastest, stored contiguously). Layout cannot change after allocation without copying and transposing data, so the choice at allocation time constrains all subsequent operations.

Consider a tensor with axes H (height, 6 rows) and W (width, 8 columns). The same tensor admits different mappings, each with different performance characteristics.

• H Major, W Minor: A scan along W is contiguous; a scan along H accesses one element per cache line.

  H=0								H=1								...
  a	b	c	d	e	f	g	h	i	j	k	l	m	n	o	p	...

• W Major, H Minor: A scan along H is contiguous; a scan along W accesses one element per cache line.

  W=0						W=1						W=2						...
  a	i	·	·	·	·	b	j	·	·	·	·	c	k	·	·	·	·	...

• 2×2 Tiles: Either choice sacrifices locality along one axis: H-major stores H-adjacent elements far apart, and W-major stores W-adjacent elements far apart. To avoid this trade-off, tiling groups nearby H and W indices into 2D tiles, achieving good locality along both axes, at the cost of a non-trivial address formula.

  t(0,0)				t(0,1)				t(0,2)				...
  a	b	i	j	c	d	k	l	e	f	m	n	...

  Decompositions also determine hardware execution structure. The outer dimension can become a hardware time loop, and the inner dimension becomes a parallel lane. TCP names these hardware dimensions Time (the sequential loop counter) and Packet (the parallel data lane width), used throughout this book. Spatial and Temporal Dimensions explains how decompositions map to them.

The Declarative Approach

Choosing optimal layout combinations manually is complex: the programmer must account for hardware geometry, alignment constraints, and execution patterns simultaneously. The compiler derives physical placement, alignment, and hardware scheduling.

Virtual ISA lets the programmer and compiler declare a mapping in terms of logical axes. For the H×W tensor from Layout and Performance, the leftmost axis is major and the rightmost is minor:

• H-major: m![H, W]
• W-major: m![W, H]
• 2×2 Tiles: m![H / 2, W / 2, H % 2, W % 2], where the first two dimensions are tile indices and the last two are positions within the tile.

Declarative mappings offer two benefits:

• Expressiveness: Layout is stated in terms of logical axes (m![H, W]), not raw strides or offsets.
• Correctness: Mapping expressions are normalized to canonical form and verified symbolically, turning layout properties into compile-time invariants.

Mapping expressions describe a tensor at every stage of its life: the same tensor can be stored in HBM, loaded into DM with a different layout, and streamed through the pipeline as packets. Each stage holds the same mathematical values in a different physical representation. Tensor Semantics formalizes this perspective and shows how it makes data movement composable with computation in the same pipeline.

Mapping Expressions

A mapping expression is a Rust type that encodes a mapping. This page defines its constructors and equivalence rules.

Axis Sizes

The axes! macro declares axis identifiers and their sizes. The following declaration applies throughout this section:

#![allow(unused)]
fn main() {
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 8, B = 512];
}

Mapping Interface

A mapping expression like m![H, W] is a Rust type that assigns each tensor index to a buffer position. Every mapping expression implements the M trait, which provides the buffer size and a function from buffer positions to tensor indices:

#![allow(unused)]
fn main() {
// Inside `furiosa_opt_std::prelude`...
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
use std::fmt::Debug;
pub trait M: Debug + Clone {
    /// The computed size for the given shape.
    const SIZE: usize;

    /// Converts the mapping expression type into a value.
    fn to_value() -> Mapping;

    /// Converts a buffer index to a tensor index, returning `None` if out-of-bounds.
    fn map(i: usize) -> Option<Index>;
}

/// Tensor index: a map from axis identifiers to coordinate values.
pub struct Index { /* ... */ }

/// Constructs tensor indices.
/// `i![A: 2, B: 3]` creates an `Index` with A = 2 and B = 3.
macro_rules! i {
    () => {};
    /* ... */
}
}

Usage Example: Host Tensor

The simplest concrete type built on the M trait is HostTensor<D, E>: a host memory buffer of element type D whose layout is fully determined by mapping E. E determines both the buffer size (E::SIZE) and the correspondence from buffer positions to tensor indices (E::map). HostTensor<bf16, m![A, B]> contains 4,096 elements of bf16 data. A HostTensor<D, E> holds a tensor \(T\) when:

• for every buffer index i and tensor index ti where E::map(i) = Some(ti),
• the i-th element stores the value of tensor \(T\) at index ti.

Device tensors such as HbmTensor and DmTensor have more complex layouts spanning multiple mapping expressions; see Spatial and Temporal Dimensions for details.

Constructors

Mapping expressions, including the layout E in HostTensor<D, E>, are built by composing small constructors, each of which transforms or combines simpler mappings. These expressions use arithmetic-like operators (/, %, and # for padding) to concisely define the mapping between tensor and linear buffer indices.

Symbol

A symbol is a single uppercase letter whose size comes from the shape declaration. The mapping m![A] maps 8 buffer indices linearly to tensor indices along the axis:

#![allow(unused)]
fn main() {
extern crate furiosa_opt_std;
extern crate furiosa_mapping;
use furiosa_opt_std::prelude::*;

axes![A = 8];

type E = m![A]; // Symbol<Ident::A, 8>

fn test_symbol() {
    assert_eq!(E::map(0), Some(i![A: 0]));
    assert_eq!(E::map(1), Some(i![A: 1]));
    assert_eq!(E::map(2), Some(i![A: 2]));
    for i in 0..E::SIZE {
        assert_eq!(E::map(i), Some(i![A: i]));
    }
    assert_eq!(E::map(E::SIZE), None);
}

test_symbol();
}

impl<S: AxisName> M for Symbol<S> {
    const SIZE: usize = S::SIZE;

    fn to_value() -> Mapping {
        Mapping::Symbol {
            symbol: S::NAME,
            size: S::SIZE,
        }
    }

    fn map(i: usize) -> Option<Index> {
        (i < S::SIZE).then(|| {
            let mut index = Index::new();
            index.add_term(
                Term {
                    inner: Atom::Symbol {
                        symbol: S::NAME,
                        size: S::SIZE,
                    },
                    stride: 1,
                    modulo: S::SIZE,
                },
                i,
            );
            index
        })
    }
}

Note

For every symbol A, the zeroth index i![A: 0] is equivalent to the empty tensor index i![].

Pair

The pair mapping m![A, B] stores a 2D tensor with shape \(\{A=8, B=512\}\) as a buffer of 4,096 elements. The mapping Pair<L, R> maps the Cartesian product of two spaces into a linear buffer where L is the major dimension and R is the minor dimension. The size is L::SIZE * R::SIZE, and the mapping uses floor division and modulo to decompose indices. m![A, B, C, D] expands to Pair<A, Pair<B, Pair<C, D>>> and is right-associative.

#![allow(unused)]
fn main() {
extern crate furiosa_opt_std;
extern crate furiosa_mapping;
use furiosa_opt_std::prelude::*;

axes![A = 8, B = 512];

type E = m![A, B]; // Pair<m![A], m![B]>

fn test_pair() {
    // First 512 elements hold A=0, next 512 hold A=1
    assert_eq!(E::map(0),   Some(i![A: 0, B: 0]));
    assert_eq!(E::map(511), Some(i![A: 0, B: 511]));
    assert_eq!(E::map(512), Some(i![A: 1, B: 0]));
    assert_eq!(E::map(519), Some(i![A: 1, B: 7])); // 519 == 512 * 1 + 7
    for i in 0..E::SIZE {
        assert_eq!(E::map(i), Some(i![A: i / <m![B]>::SIZE, B: i % <m![B]>::SIZE]));
    }
    assert_eq!(E::map(E::SIZE), None);
}

test_pair();
}

impl<L, R> M for Pair<L, R>
where
    L: M,
    R: M,
{
    const SIZE: usize = L::SIZE * R::SIZE;

    fn to_value() -> Mapping {
        Mapping::Pair {
            left: RBox::new(L::to_value()),
            right: RBox::new(R::to_value()),
        }
    }

    fn map(i: usize) -> Option<Index> {
        let mut l = L::map(i / R::SIZE)?;
        let r = R::map(i % R::SIZE)?;
        l.add(r);
        Some(l)
    }
}

Identity

The identity mapping m![1] creates a single-element buffer that maps buffer index 0 to the empty tensor index i![]. It serves as the identity element for Pair: m![1, A] and m![A, 1] are both equivalent to m![A].

#![allow(unused)]
fn main() {
extern crate furiosa_opt_std;
extern crate furiosa_mapping;
use furiosa_opt_std::prelude::*;

type E = m![1]; // Identity

fn test_identity() {
    assert_eq!(E::map(0), Some(i![]));
    assert_eq!(E::map(1), None);
}

test_identity();
}

/// The identity mapping (size-1 broadcast), the unit written `m![1]`.
pub type Identity = Broadcast<1>;

Padding

Padding aligns data to hardware requirements by adding unused buffer space. For example, the DMA engine requires rows to start on 64-byte boundaries. With axes![C = 13, D = 61], m![C, D] creates misaligned rows since 61 is not divisible by 64. m![C, D # 64] fixes this by aligning each row to 64-byte boundaries, using 3 extra elements per row.

#![allow(unused)]
fn main() {
extern crate furiosa_opt_std;
extern crate furiosa_mapping;
use furiosa_opt_std::prelude::*;

axes![C = 13, D = 61];

type E = m![C, D # 64]; // Pair<m![C], Padding<m![D], 64>>

fn test_padding() {
    assert_eq!(E::map(0),  Some(i![C: 0, D: 0]));
    assert_eq!(E::map(60), Some(i![C: 0, D: 60]));
    assert_eq!(E::map(61), None); // padding
    assert_eq!(E::map(62), None); // padding
    assert_eq!(E::map(63), None); // padding
    assert_eq!(E::map(64), Some(i![C: 1, D: 0]));
}

test_padding();
}

impl<L, const SIZE: usize, const KIND: PaddingKind> M for Padding<L, SIZE, KIND>
where
    L: M,
{
    const SIZE: usize = SIZE;

    fn to_value() -> Mapping {
        Mapping::Padding {
            inner: RBox::new(L::to_value()),
            padding: SIZE,
            kind: KIND,
        }
    }

    fn map(i: usize) -> Option<Index> {
        L::map(i)
    }
}

The padded slots’ content is part of the type, not just their count. Three kinds are tracked.

• m![A # m] (or m![A #{*} m]) is top padding to size m. Slots are accessible but hold arbitrary values. Raw DM tensors carry this. # is the shorthand; #{*} spells the kind out explicitly.
• m![A #{0} m] is zero-filled padding to size m. Slots are accessible and known to hold zero. The Fetch Adapter’s masking stage produces this from # m.
• m![A #{!} m] is bottom padding to size m. Slots are inaccessible and reads/writes are undefined behavior. This models addresses the compiler must avoid.

# defaults to top kind. The Rust type level mirrors this via a const generic of PaddingKind on Padding<L, SIZE, KIND>. Padding<L, N> is KIND = PaddingKind::Top, Padding<L, N, { PaddingKind::Zero }> is the zero-filled variant, and Padding<L, N, { PaddingKind::Bottom }> is inaccessible.

Resize

Resize constrains a mapping to a smaller logical size by truncating indices beyond the new size, discarding elements outside that range. Unlike padding, which expands the buffer, Resize shrinks the logical view. The mapping m![D = 2] takes only the first 2 elements of axis D, producing indices D = 0 and D = 1.

#![allow(unused)]
fn main() {
extern crate furiosa_opt_std;
extern crate furiosa_mapping;
use furiosa_opt_std::prelude::*;

axes![C = 2, D = 3];
type E = m![C, D = 2]; // Pair<m![C], Resize<m![D], 2>>

fn test_resize() {
    assert_eq!(E::map(0), Some(i![C: 0, D: 0]));
    assert_eq!(E::map(1), Some(i![C: 0, D: 1]));
    assert_eq!(E::map(2), Some(i![C: 1, D: 0]));
    assert_eq!(E::map(3), Some(i![C: 1, D: 1]));
    assert_eq!(E::map(4), None);
}

test_resize();
}

impl<L, const SIZE: usize> M for Resize<L, SIZE>
where
    L: M,
{
    const SIZE: usize = SIZE;

    fn to_value() -> Mapping {
        Mapping::Resize {
            inner: RBox::new(L::to_value()),
            resize: SIZE,
        }
    }

    fn map(i: usize) -> Option<Index> {
        if i < SIZE { L::map(i) } else { None }
    }
}

Tiling

Tiling is implemented through indexed views, pure metadata transformations without data copies. The .tile() method extracts a tile by resizing one dimension to the tile size and offsetting into the buffer.

#![allow(unused)]
fn main() {
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;

axes![A = 8, B = 512];

let tensor = unsafe { HbmTensor::<bf16, m![1], m![A, B]>::from_addr(0) };
let view = tensor.view(); // HbmTensorView::<'_, bf16, m![1], m![A, B]>
let tile01 = view.tile::<m![B], 2, m![A, B = 2 # 512]>(0); // HbmTensorView::<'_, bf16, m![1], m![A, B = 2 # 512]>
let tile23 = view.tile::<m![B], 2, m![A, B = 2 # 512]>(2); // HbmTensorView::<'_, bf16, m![1], m![A, B = 2 # 512]>
}

The .tile() method takes three type parameters and one value parameter.

• The tile dimension m![B] specifies which dimension to divide along.
• The tile size 2 specifies the number of elements per tile.
• The tile mapping m![A, B = 2 # 512] defines the resulting view’s mapping. The mapping B = 2 # 512 signifies that dimension B has a logical size of 2 within the view but exists within a physical footprint of 512. Without # 512, the stride between tiles would be 2 instead of 512, causing the view to read from wrong buffer positions.
• The starting index specifies which tile to extract. Passing 0 captures the range 0..2 for tile01, while passing 2 captures the range 2..4 for tile23.

Stride and Modulo

Stride (/) and modulo (%) decompose a single dimension into two: the outer (block index) and the inner (position within block). Consider the 512-element axis B divided into 8 blocks of 64 elements each. The mapping m![B / 64, B % 64] creates an 8 × 64 grid where the first dimension selects which block and the second dimension selects the position within that block:

#![allow(unused)]
fn main() {
extern crate furiosa_opt_std;
extern crate furiosa_mapping;
use furiosa_opt_std::prelude::*;
axes![A = 8, B = 512];
type D1 = m![B / 64]; // stride with size 8
type D2 = m![B % 64]; // modulo with size 64

type E = m![B / 64, B % 64]; // equivalent to `m![B]`

fn test_stride_modulo() {
    assert_eq!(E::map(130), Some(i![B / 64: 2, B % 64: 2])); // block 2, position 2: B = 64*2 + 2 = 130
    assert_eq!(E::map(130), <m![B]>::map(130));               // same result as flat m![B]

    for i in 0..8 {
        assert_eq!(D1::map(i), Some(i![B / 64: i]));
    }
    assert_eq!(D1::map(8), None);

    for j in 0..64 {
        assert_eq!(D2::map(j), Some(i![B % 64: j]));
    }
    assert_eq!(D2::map(64), None);

    for i in 0..8 {
        for j in 0..64 {
            assert_eq!(
                E::map(64 * i + j),
                <m![B]>::map(64 * i + j),
            );
        }
    }
    assert_eq!(E::map(512), None);
}

test_stride_modulo();
}

impl<L, const SIZE: usize> M for Stride<L, SIZE>
where
    L: M,
{
    const SIZE: usize = {
        assert!(L::SIZE % SIZE == 0, "Stride size must divide the original size");
        L::SIZE / SIZE
    };

    fn to_value() -> Mapping {
        Mapping::Stride {
            inner: RBox::new(L::to_value()),
            stride: SIZE,
        }
    }

    fn map(i: usize) -> Option<Index> {
        if i < Self::SIZE { L::map(i * SIZE) } else { None }
    }
}

impl<L, const SIZE: usize> M for Modulo<L, SIZE>
where
    L: M,
{
    const SIZE: usize = {
        assert!(L::SIZE % SIZE == 0, "Modulo size must divide the original size");
        SIZE
    };

    fn to_value() -> Mapping {
        Mapping::Modulo {
            inner: RBox::new(L::to_value()),
            modulo: SIZE,
        }
    }

    fn map(i: usize) -> Option<Index> {
        if i < Self::SIZE { L::map(i % L::SIZE) } else { None }
    }
}

Stride and modulo mappings can be visualized in tabular form. Consider the mapping m![B / 4, B % 4] with B::SIZE = 16. The following table shows how buffer indices are arranged: each row corresponds to a specific index of B / 4 (the stride axis), and each column corresponds to an index of B % 4 (the modulo axis):

Modulo differs from resize in how it handles buffer size:

• Resize shrinks the buffer by truncating indices beyond the new size.
• Modulo preserves the original buffer size while partitioning it into equal-sized blocks.

These operations can be nested for complex decompositions. The following example splits B into three dimensions where the buffer’s bit layout differs from that of the tensor index.

#![allow(unused)]
fn main() {
extern crate furiosa_opt_std;
extern crate furiosa_mapping;
use furiosa_opt_std::prelude::*;
axes![A = 8, B = 512];
// B's bits: 6 - 8,  0 - 4,          5
// Values:   0 - 7, 0 - 31,      0 - 1
type E = m![B / 64, B % 32, B / 32 % 2];

fn test_nested_stride() {
    assert_eq!(E::map(67), Some(i![B: 97])); // 67 = 64*1 + 2*1 + 1 (i=1,j=1,k=1) → B = 64*1 + 1 + 32*1 = 97
    // Verify B=97 round-trips: 97/64=1, 97%32=1, (97/32)%2=1
    assert_eq!(97 / 64, 1);
    assert_eq!(97 % 32, 1);
    assert_eq!((97 / 32) % 2, 1);

    // buffer index: 64 * i + 2 * j + k (i = block, j = position within block, k = sub-block)
    // tensor index B: 64 * i + j + 32 * k (rearranges bit positions)
    for i in 0..8 {
        for j in 0..32 {
            for k in 0..2 {
                assert_eq!(
                    E::map(64 * i + 2 * j + k),
                    Some(i![B: 64 * i + j + 32 * k]),
                );
            }
        }
    }
    assert_eq!(E::map(512), None);
}

test_nested_stride();
}

This kind of bit rearrangement maps naturally to hardware memory layouts where address bits are reordered for bank interleaving or cache efficiency. In binary, this rearranges bit positions: buffer 001_00001_1 becomes B = 001_1_00001. The buffer groups bits as [8:6]_[5:1]_[0] while B groups them as [8:6]_[5]_[4:0].

Tiling can operate on blocks rather than individual elements. The following example tiles by block using m![B / 32] and creates overlapping tiles:

#![allow(unused)]
fn main() {
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 8, B = 512];
let tensor = unsafe { HbmTensor::<bf16, m![1], m![A, B]>::from_addr(0) };
for i in 0..15 {
    let tile = tensor.view().tile::<m![B / 32], 2, m![A, B / 32 = 2 # 16, B % 32]>(i);
}
}

With B = 512, the dimension B / 32 has 16 blocks numbered 0-15. Each tile takes 2 consecutive blocks starting at index i. Tile 0 covers blocks {0, 1}, tile 1 covers blocks {1, 2}, and so on through tile 14 covering blocks {14, 15}. These tiles overlap because consecutive tiles share one block.

The tile mapping B / 32 = 2 resizes the block dimension to 2 since each tile contains exactly 2 blocks. When tiling with a single block, B / 32 = 1 simplifies to the identity m![1] since the dimension has only one value.

Escape

For complex mappings, define type aliases and reference them using { ... }. With separate mappings L = m![A] and R = m![B], combining them as m![{ L }, { R }] produces the same result as m![A, B]:

#![allow(unused)]
fn main() {
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 8, B = 512];
type L = m![A];
type R = m![B];
type E = m![{ L }, { R }]; // equivalent to `m![A, B]`

fn test_escape() {
    for i in 0..E::SIZE {
        assert_eq!(E::map(i), <m![A, B]>::map(i));
    }
}

test_escape();
}

This escape syntax breaks down complex mappings into named, reusable components.

Advanced Constructors

Skewed axis

A skewed axis creates a diagonal access pattern across two dimensions. Skewed axes introduce derived axis labels defined by arithmetic differences between existing axes; for example, B' = B - A defines a new axis B' whose coordinate at any point equals B minus A. Algorithms that process data along diagonals use this pattern, such as certain wavefront computations.

The expression m![A, B' = 4] with B' = B - A creates a mapping where each row is shifted relative to the previous one. The = operator specifies the logical size after skewing. The result wraps around using modular arithmetic.

For example, with axes![A = 4, B = 4] and B' = B - A:

When A = 1 and B' = 3, the original B coordinate wraps to 0 via modular arithmetic since B = (B' + A) % 4 = (3 + 1) % 4 = 0.

Indirect sequencing

Sliding (linear combination)

Note

Linear combination expressions $(e1:n1, ..., ed:nd) combine multiple dimensions with specified strides. Formal definition: size_S($(e1:n1, ..., ed:nd)) = 1 + sum_k((size_S(ek) - 1) * nk). The mapping S, $(e1:n1, ..., ed:nd) |- si ~ ti holds if there exist si1...sid, ti1...tid such that for all k: S, ek |- sik ~ tik, si = sum_k(sik * nk), and ti = sum_k(tik * nk).

Linear combinations can encode outer sum: e1 * e2 is equivalent to $(e1 : size_S(e2), e2 : 1). However, outer sum is preferred because it’s more resilient to axis reordering. Changing e1 * e2 to e2 * e1 doesn’t require manual stride updates.

Sliding operations access overlapping data blocks, essential for convolutional neural networks. Consider a buffer of 9 elements representing a tensor with shape \(\{N=5, F=3\}\) where each row is a 3-element slice that slides one element at a time. The tensor element at \((N, F)\) maps to buffer index \(N + 2F\):

$$ \begin{array}{c|ccc} & F=0 & F=1 & F=2 \\ \hline N=0 & 0 & 2 & 4 \\ N=1 & 1 & 3 & 5 \\ N=2 & 2 & 4 & 6 \\ N=3 & 3 & 5 & 7 \\ N=4 & 4 & 6 & 8 \\ \end{array} $$

Note

In this sliding pattern, a single space index can map to multiple tensor indices. For example, space index 4 maps to {4_N}, {2_N, 1_F}, and {2_F} simultaneously. This illustrates the non-one-to-one nature of (S, e).maps(si, ti).

This can be expressed using a linear combination expression where the N axis has stride 1 and the F axis has stride 2, yielding a total size of 1 + (5-1)*1 + (3-1)*2 = 9.

Equivalent Mapping

Different constructor combinations can produce the same mapping. Specifically, mappings E1 and E2 are equivalent when:

• E1::SIZE == E2::SIZE, and
• For every i, E1::map(i) == E2::map(i).

The equivalence relation is reflexive, symmetric, and transitive. The following identities capture common equivalences:

• Identity of pairs: for every E, E is equivalent both to m![{ E }, 1] and m![1, { E }].
• Stride-modulo decomposition: for every E whose size E::SIZE is divisible by n, E and m![{ E } / n, { E } % n] are equivalent.
• Pair projection: for every A and B, m![[{ A }, { B }] / B::SIZE] is equivalent to m![A] and m![[{ A }, { B }] % B::SIZE] is equivalent to m![B].
• Associativity of pairs: for every E1, E2, E3, m![{ E1 }, { E2 }, { E3 }], m![[{ E1 }, { E2 }], { E3 }], and m![{ E1 }, [{ E2 }, { E3 }]] are equivalent.
• Idempotent operations: for every E, E is equivalent to m![{ E } / 1], to m![{ E } # E::SIZE], and to m![{ E } = E::SIZE].
• Modulo by 1: for every E, m![E % 1] is equivalent to the identity mapping m![1].

Spatial and Temporal Dimensions

HostTensor<D, E> uses a single mapping to fully capture its layout. Device tensors split their layout across multiple dedicated dimensions:

• Spatial dimensions: Chip, Cluster, and Slice distribute data across the hardware hierarchy. In stream tensors, Packet additionally sizes parallel delivery within each temporal iteration.
• Temporal dimension: Time sequences the delivery iterations in stream tensors.

Spatial Dimensions

Each spatial level in the hardware hierarchy gets its own type parameter in the tensor type, enabling spatial parallelism. All units at each level are assumed to share the same mapping.

#![allow(unused)]
fn main() {
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
use std::marker::PhantomData;
// Assumed throughout this page.
axes![A = 8, B = 512];

// HBM tensors
struct HbmTensor<D: Scalar, Chip: M, Element: M> {
    /* ... */
    _marker: PhantomData<(D, Chip, Element)>,
}

// SRAM tensors
// DM (Data Memory), TRF (Tensor Register File), and VRF (Vector Register File)
struct DmTensor<D: Scalar, Chip: M, Cluster: M, Slice: M, Element: M> {
    /* ... */
    _marker: PhantomData<(D, Chip, Cluster, Slice, Element)>,
}
struct TrfTensor<D: Scalar, Chip: M, Cluster: M, Slice: M, Lane: M, Element: M> {
    /* ... */
    _marker: PhantomData<(D, Chip, Cluster, Slice, Lane, Element)>,
}
struct VrfTensor<D: Scalar, Chip: M, Cluster: M, Slice: M, Element: M> {
    /* ... */
    _marker: PhantomData<(D, Chip, Cluster, Slice, Element)>,
}
}

HBM tensors distribute data across chips for spatial parallelism: each chip processes its own portion of the data simultaneously. For example, HbmTensor<bf16, m![A], m![B]> distributes 8 × 512 = 4096 elements across 8 chips with 512 elements per chip. The i-th chip’s j-th element stores tensor index i![A: i, B: j].

SRAM tensor types add Cluster and Slice dimensions for finer-grained parallelism. TrfTensor additionally has a Lane dimension that distributes TRF data across the 8 lanes per slice. See Contraction Engine for details.

Every storage tensor (HostTensor, HbmTensor, and the SRAM types) places its element data at a starting address. For example, a DmTensor<D, ..., Element> at address addr occupies bytes addr..(addr + Element::SIZE * size_of::<D>()); TrfTensor and VrfTensor follow the same pattern.

Constraints

• Chip, Cluster, and Slice size: they must exactly match the hardware counts:

  Unit	Count	Constraint	Padding Example
  Chip	System-dependent	Chip::SIZE == NUM_CHIPS	m![1 # NUM_CHIPS]
  Cluster	2 / Chip	Cluster::SIZE == 2	m![1 # 2]
  Slice	256 / Cluster	Slice::SIZE == 256	m![X / N # 256]

  Any dimension can be padded with # when the kernel uses fewer units than the hardware provides. For example, type Cluster = m![1 # 2] uses 1 active cluster and 1 padding-only cluster, satisfying the hardware’s 2-cluster-per-chip requirement.

  Note

  The runtime operates at chip granularity (#[device(chip = N)]), so partial chip or cluster usage is not yet supported. This may be relaxed in future releases.

• Element size: Element::SIZE * size_of::<D>() must not exceed the per-unit SRAM capacity, which varies by tensor type:

  Type	Unit	Constraint
  DmTensor	512KB / Slice	Element::SIZE * size_of::<D>() <= 512KB
  TrfTensor	8KB / Lane	Lane::SIZE <= 8, Element::SIZE * size_of::<D>() <= 8KB
  VrfTensor	8KB / Slice	Element::SIZE * size_of::<D>() <= 8KB

• Element alignment: The starting address must be a multiple of size_of::<D>(), because misaligned writes require a read-modify-write cycle that can slow DM access by roughly 50×.

Temporal Dimension

TuTensor represents tensor data flowing through the Tensor Unit as a stream. It retains the same Chip, Cluster, and Slice dimensions as the SRAM types, and adds Time and Packet for streaming. Time is the temporal dimension: it sequences the delivery iterations. Unlike the spatial dimensions, Time has no hardware-imposed size limit and grows with the amount of data to process. Packet is an additional spatial dimension that determines how many elements each slice receives per temporal iteration.

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
use std::marker::ConstParamTy;
use std::marker::PhantomData;
axes![N = 4, C = 64, H = 32, W = 32];

/// Pipeline stage.
/// `Vector` is intentionally absent: the Vector Engine uses a separate typestate
/// (`VectorBranchTensor` and friends) that tracks branch, ALU, and other Vector-specific state.
/// `Commit` is intentionally absent: once the Commit Engine writes results back to DM,
/// the data is at rest and the type becomes `DmTensor`, not `TuTensor`.
#[derive(PartialEq, Eq, ConstParamTy)]
enum Position {
    Begin,       // After the start of the pipeline
    Fetch,       // After the Fetch Engine
    Switch,      // After the Switch Engine
    Collect,     // After the Collect Engine
    Contraction, // After the Contraction Engine
    Reduce,      // After the Reduce Engine
    Cast,        // After the Cast Engine
    Transpose,   // After the Transpose Engine
}

struct TuTensor<
    'l,                // Lifetime tied to the Tensor Unit context
    const P: Position,
    D: Scalar,
    Chip: M,
    Cluster: M,
    Slice: M,
    Time: M,
    Packet: M,
> {
    /* ... */
     _marker: PhantomData<&'l (D, Chip, Cluster, Slice, Time, Packet)>,
}

type T<'l> = TuTensor<
    'l,
    { Position::Fetch }, // Fetch Engine's output
    bf16,
    m![1],           // Chip: single chip
    m![1],           // Cluster: single cluster
    m![C / 2],       // Slice: distribute 64 channels across 32 slices
    m![N, H, W],     // Time: iterate over batch (N) and spatial (H, W) dimensions
    m![C % 2],       // Packet: 2 channels per cycle
>;
}

Type T streams a tensor with an aggregate shape of \(\{N=4, C=64, H=32, W=32\}\) across 32 slices (Slice::SIZE = m![C / 2]::SIZE = 32). The Time dimension (m![N, H, W]) has size 4 * 32 * 32 = 4096, which means there are 4,096 temporal iterations. For each temporal iteration, the Packet dimension m![C % 2] delivers 2 channels to each slice. Since 32 slices operate in parallel, each temporal iteration processes 32 * 2 = 64 channels total.

Tensor Semantics

Tensors reside in HBM, on-chip DM, or the pipeline stream, and operations transform them. This chapter defines their mathematical meaning: what it means for a tensor variable to hold a mathematical tensor, and what it means for an operation to specify a mathematical function. These definitions enable tensor-level reasoning about vISA programs: a function is correct when its output holds the right mathematical tensor, regardless of which mapping or memory tier is used.

Holding a Tensor

A tensor variable holds mathematical tensor \(T\) when each element stores the value of \(T\) at the tensor index formed by summing the partial indices produced by each dimension’s mapping.

HostTensor<D, E> is the simplest case: a single mapping E fully determines the correspondence between buffer positions and tensor indices. HostTensor<bf16, m![A, B]> with A = 8 and B = 512, for instance, stores 4,096 bf16 elements in A-major, B-minor order. It holds tensor \(T\) when:

• for every buffer index i where E::map(i) = Some(ti),
• the i-th element stores the value of \(T\) at ti.

HbmTensor<D, Chip, Element> extends this by splitting the single mapping into two: Chip maps chip indices to partial tensor indices, and Element maps per-chip element indices to the remaining partial indices, with each covering a disjoint subset of axes so their sum recovers the full tensor index. It holds \(T\) when:

• for every chip index i and element index j where Chip::map(i) = Some(ti) and Element::map(j) = Some(tj),
• the i-th chip’s j-th element stores \(T\) at the index ti + tj.

All other tensor types apply the same rule to more dimensions: each element stores \(T\) at the sum of the partial indices returned by all its mapping parameters.

Specifying a Function

Specifying a function means declaring what its output holds in terms of its inputs. For example, the function elementwise_add specifies the mathematical operation \(f(T_1, T_2) = T_1 + T_2\) in that:

• For every tensor \(T_1\) and \(T_2\),
• if lhs holds \(T_1\) and rhs holds \(T_2\),
• then the return value holds \(T_1 + T_2\).

#![allow(unused)]
fn main() {
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 8, B = 512];

fn elementwise_add(
    lhs: &HbmTensor<bf16, m![A], m![B]>,
    rhs: &HbmTensor<bf16, m![A], m![B]>,
) -> HbmTensor<bf16, m![A], m![B]> {
    // ... computes elementwise add ...
    todo!("elementwise add lhs and rhs")
}
}

A mathematical tensor move specifies \(f(T) = T\): the output holds the same mathematical tensor as the input, regardless of representation. .to_dm() is a mathematical tensor move. The .to_dm() method, for instance, specifies \(f(T) = T\) in that:

• if hbm holds \(T\),
• the return value holds \(T\).

#![allow(unused)]
fn main() {
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 8, B = 512];

fn hbm_to_dm(
    ctx: &mut Context,
    hbm: &HbmTensor<bf16, m![A], m![B]>,
) -> DmTensor<bf16, m![A], m![1], m![B / 2], m![B % 2]> {
    hbm.to_dm(&mut ctx.tdma)
}
}

Moving Tensors

Quick Start introduced TCP’s memory tiers. This chapter covers how tensors move between three of them: HBM, DM, and SPM, through three dedicated engines:

• Fetch: DM → Tensor Unit stream
• Commit: Tensor Unit stream → DM
• DMA: any pair among DM, SPM, HBM

TRF and VRF are populated by Tensor Unit primitives rather than dedicated move engines, and are covered in Computing Tensors.

flowchart TB
    HBM[(HBM)] <--> DMA[DMA]
    SPM[(SPM)] <--> DMA[DMA]
    DMA <--> DM[(DM)]

    subgraph TU[Tensor Unit]
        direction TB
        FE[Fetch] --> DOT1[...] --> CT[Contraction] --> VE[Vector] --> DOT2[...] --> CM[Commit]
    end

    DM -->|stream| FE
    CM -->|stream| DM

    click DMA "./dma-engine.html" "DMA Engine"
    click FE "./fetch-engine.html" "Fetch Engine"
    click CT "../computing-tensors/contraction-engine/index.html" "Contraction Engine"
    click VE "../computing-tensors/vector-engine/index.html" "Vector Engine"
    click CM "./commit-engine.html" "Commit Engine"
    click TU "../computing-tensors/index.html" "Tensor Unit"

Their APIs are designed around what the programmer controls: which engine moves each tensor and how axes map to hardware dimensions. The compiler translates these declarations into low-level hardware concerns such as memory bank scheduling, stride calculation, and access alignment.

The Sequencer is the shared mechanism all three engines use to convert between memory buffers and packet streams. Memory Performance covers how the choice of engine and axis mapping affects bandwidth utilization.

Sequencer

A sequencer reads a memory buffer as a packet stream and writes a packet stream back to memory. The Fetch and Commit Engines each use one sequencer to address DM. The DMA Engine chains a read sequencer and a write sequencer to move data among DM, SPM, and HBM without intermediate buffers.

Interface

MemTensor and StreamTensor are pedagogical pseudo types that capture the buffer-stream pattern in isolation, so this page can explain sequencer mechanics without dragging in each engine’s full type machinery. The real engine APIs use different types (DmTensor, HbmTensor, TuTensor, …), but every concrete pair maps onto the same MemTensor → StreamTensor shape illustrated below.

MemTensor holds data in some memory mapping Buf, and any concrete buffer tensor (DM, SPM, HBM) plays this role.

/// A generic buffer-backed tensor.
/// Anything that holds data in a memory `Buf` and can be streamed in or out.
#[derive(Debug)]
pub struct MemTensor<D: Scalar, Buf: M> {
    inner: Tensor<D, Buf>,
}

StreamTensor is a tensor in flight. The lifetime 'l ties the stream to its source buffer so a stream cannot outlive its data. Time is the temporal mapping (iteration over time) and Packet is the spatial mapping (contents of a single packet).

/// A streaming view of a tensor in flight.
/// `Packet` is the per-cycle shape and `Time` is the multi-cycle shape.
#[derive(Debug)]
pub struct StreamTensor<'l, D: Scalar, Time: M, Packet: M> {
    inner: Tensor<D, Pair<Time, Packet>>,
    _marker: PhantomData<&'l ()>,
}

read converts a MemTensor into a StreamTensor and write reverses it, both preserving values. Each engine’s full API adds spatial dimensions (Chip, Cluster, Slice) on top, covered in the engine-specific pages.

impl<D: Scalar, Buf: M> MemTensor<D, Buf> {
    /// Reads a stream from this buffer with the supplied `Time` and `Packet` mapping.
    /// `(Time, Packet)` may be a broadcast of `Buf` (matches Fetch / Switch / DMA-read behavior).
    pub fn read<'l, Time: M, Packet: M>(&'l self) -> StreamTensor<'l, D, Time, Packet> {
        StreamTensor {
            inner: self.inner.transpose(true),
            _marker: PhantomData,
        }
    }

    /// Writes a stream back into this buffer.
    /// Broadcast is rejected: each `Buf` slot must have exactly one source position in `(Time, Packet)` (matches Commit behavior).
    pub fn write<'l, Time: M, Packet: M>(&mut self, stream: StreamTensor<'l, D, Time, Packet>) {
        self.inner = stream.inner.transpose(false);
    }
}

For any MemTensor, many valid Time and Packet combinations exist, each producing a different StreamTensor. Among valid choices, larger Packet sizes improve bandwidth utilization, and Memory Performance covers the trade-offs in detail.

Examples

The following examples show common read and write patterns using the core API above. Architecture below explains how the compiler derives each pattern’s hardware configuration.

#![allow(unused)]
fn main() {
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
use furiosa_opt_std::pseudo::{MemTensor, StreamTensor};
axes![A = 8, B = 512, N = 4, C = 3, H = 8, W = 8, T = 4, P = 4];

/// Strided access: read 8×512 tensor as 128 packets of 32 elements.
/// Time = m![A, B / 32] produces 8 * 16 = 128 time steps.
/// Packet = m![B % 32] delivers 32 consecutive elements per packet.
fn strided_read<'l>(
    buf: &'l MemTensor<bf16, m![A, B]>,
) -> StreamTensor<'l, bf16, m![A, B / 32], m![B % 32]> {
    buf.read()  // Automatic type inference
}

/// Strided write: write 128 packets of 32 elements back to 8×512 tensor.
fn strided_write(
    buf: &mut MemTensor<bf16, m![A, B]>,
    stream: StreamTensor<bf16, m![A, B / 32], m![B % 32]>,
) {
    buf.write(stream)
}

/// Axis reordering read: change traversal from [N, C, H, W] to [W, H, C, N].
/// Time = m![W, H, C, N] iterates in reversed axis order.
/// Packet = m![1] delivers single-element packets.
fn axis_reordering_read<'l>(
    buf: &'l MemTensor<bf16, m![N, C, H, W]>,
) -> StreamTensor<'l, bf16, m![W, H, C, N], m![1]> {
    buf.read()
}

/// Axis reordering write: write [W, H, C, N] stream back to [N, C, H, W] buffer.
fn axis_reordering_write(
    buf: &mut MemTensor<bf16, m![N, C, H, W]>,
    stream: StreamTensor<bf16, m![W, H, C, N], m![1]>,
) {
    buf.write(stream)
}

/// Tiling read: break axes into sub-blocks for cache efficiency.
/// Time = m![A % 2, B % 4, A / 2, B / 4] tiles A into 2 × 4, B into 4 × 128 blocks.
/// Packet = m![C # 32] pads C to 32 elements per packet.
fn tiling_read<'l>(
    buf: &'l MemTensor<i8, m![A, B, C # 8]>,
) -> StreamTensor<'l, i8, m![A % 2, B % 4, A / 2, B / 4], m![C # 32]> {
    buf.read()
}

/// Tiling write: write tiled stream back to buffer.
fn tiling_write(
    buf: &mut MemTensor<i8, m![A, B, C # 8]>,
    stream: StreamTensor<i8, m![A % 2, B % 4, A / 2, B / 4], m![C # 32]>,
) {
    buf.write(stream)
}

/// Broadcasting read: replicate elements absent from `Buf`.
/// Time = m![T, A] broadcasts T temporally (same data repeated T times).
/// Packet = m![P] broadcasts P spatially (same element fills packet).
fn broadcasting_read<'l>(
    buf: &'l MemTensor<i8, m![A]>,
) -> StreamTensor<'l, i8, m![T, A], m![P]> {
    buf.read()
}

/// Broadcasting write: write broadcast stream back to buffer.
/// This is rejected as each `Buf` slot must have exactly one source position in `(Time, Packet)`
/// This code will panic when run
fn broadcasting_write(
    buf: &mut MemTensor<i8, m![A]>,
    stream: StreamTensor<i8, m![T, A], m![P]>,
) {
    buf.write(stream)
}

let buf_read = MemTensor::<bf16, m![A, B]>::from_vec(vec![bf16::from_f32(1f32); 8 * 512]);
let mut buf_write = MemTensor::<bf16, m![A, B]>::from_vec(vec![bf16::from_f32(1f32); 8 * 512]);

let stream = strided_read(&buf_read);
strided_write(&mut buf_write, stream);

// -----------------------------------------------------------------------------------

let buf_read = MemTensor::<bf16, m![N, C, H, W]>::from_vec(vec![bf16::from_f32(1f32); 4 * 3 * 8 * 8]);
let mut buf_write = MemTensor::<bf16, m![N, C, H, W]>::from_vec(vec![bf16::from_f32(0f32); 4 * 3 * 8 * 8]);

let stream = axis_reordering_read(&buf_read);
axis_reordering_write(&mut buf_write, stream);

// -----------------------------------------------------------------------------------

let buf_read = MemTensor::<i8, m![A, B, C # 8]>::from_vec(vec![1i8; 8 * 512 * 8]);
let mut buf_write = MemTensor::<i8, m![A, B, C # 8]>::from_vec(vec![0i8; 8 * 512 * 8]);

let stream = tiling_read(&buf_read);
tiling_write(&mut buf_write, stream);

// -----------------------------------------------------------------------------------

let buf_read = MemTensor::<i8, m![A]>::from_vec(vec![1i8; 8 ]);
let mut buf_write = MemTensor::<i8, m![A]>::from_vec(vec![0i8; 8 ]);

let stream = broadcasting_read(&buf_read);
let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
    broadcasting_write(&mut buf_write, stream);
}));
assert!(result.is_err()); 

}

Architecture

Each sequencer call is compiled from its input and output tensor mappings into a nested-loop configuration that the sequencer hardware executes. Each configuration takes the form [size_0 : stride_0, size_1 : stride_1, ...] : packet_size, where subscript 0 is the outermost loop:

#![allow(unused)]
fn main() {
struct Config {
    /// Each entry defines a nested loop level.
    entries: Vec<Entry>,
    /// Number of elements per packet.
    packet_size: usize,
}

struct Entry {
    /// Number of iterations for this loop level.
    size: usize,
    /// Memory address distance (in elements) to skip after each iteration.
    stride: isize,
}
}

Each entry encodes one dimension of tensor traversal. The size field determines how many times this loop iterates, while the stride field determines the memory offset between consecutive iterations.

Access Size

max_access_size = gcd(Packet::SIZE, contiguous_run) is the number of elements per hardware access.

Here, contiguous_run is the element count of the innermost physically contiguous run of Config entries. A larger max_access_size means fewer accesses per packet. The following shows how max_access_size is computed from a Config:

#![allow(unused)]
fn main() {
struct Config { entries: Vec<Entry>, packet_size: usize }
struct Entry { size: usize, stride: isize }
fn gcd(mut a: usize, mut b: usize) -> usize { while b != 0 { (a, b) = (b, a % b); } a }
impl Config {
    fn contiguous_run(&self) -> usize {
        // Walk pairs from innermost outward; stop at the first non-contiguous pair.
        // Two adjacent entries (n_outer : s_outer) and (n_inner : s_inner)
        // are contiguous when s_outer == n_inner * s_inner.
        let mut contiguous_run = self.entries.last().map_or(1, |e| e.size);
        for w in self.entries.windows(2).rev() {
            if w[0].stride == w[1].size as isize * w[1].stride {
                contiguous_run *= w[0].size;
            } else {
                break;
            }
        }
        contiguous_run
    }

    fn max_access_size(&self) -> usize {
        gcd(self.packet_size, self.contiguous_run())
    }
}

let config = Config {
    entries: vec![
        Entry { size: 4, stride: 192 },
        Entry { size: 3, stride: 64 },
        Entry { size: 8, stride: 8 },
        Entry { size: 8, stride: 1 },
    ],
    packet_size: 8,
};

assert_eq!(config.contiguous_run(), 768);
assert_eq!(config.max_access_size(), 8);
}

In most cases the packet layout is fully contiguous in DM and max_access_size == Packet::SIZE. See Non-Contiguous Packets for a case where max_access_size < Packet::SIZE.

How It Works

The Config for m![N, C, H, W] → m![W, H, C, N] has one entry per axis in the stream, each with a stride equal to that axis’s span in the source buffer. Since Packet = m![1], Packet::SIZE = max_access_size = 1 and the sequencer issues one DM access per loop iteration.

extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
use furiosa_opt_std::pseudo::{MemTensor, StreamTensor};
struct Config {
    entries: Vec<Entry>,
    packet_size: usize,
}
struct Entry {
    size: usize,
    stride: isize,
}
axes![N = 4, C = 3, H = 8, W = 8];

fn read_nchw_whcn(buf: &MemTensor<bf16, m![N, C, H, W]>) ->
                     StreamTensor<bf16, m![W, H, C, N], m![1]> {
    // Compiler-generated configuration: [8 : 1, 8 : 8, 3 : 64, 4 : 192] : 1
    let config = Config {
        entries: vec![
            Entry { size: 8, stride: 1 },   // W
            Entry { size: 8, stride: 8 },   // H
            Entry { size: 3, stride: 64 },  // C
            Entry { size: 4, stride: 192 }, // N
        ],
        packet_size: 1,
    };

    // The hardware executes the configuration as nested loops:
    for w in 0..8 {
        for h in 0..8 {
            for c in 0..3 {
                for n in 0..4 {
                    // Read each address
                    let addr = 1 * w + 8 * h + 64 * c + 192 * n;
                    // yield buf[addr];
                }
            }
        }
    }

    buf.read()
}

fn write_whcn_nchw(buf: &mut MemTensor<bf16, m![N, C, H, W]>,
                  stream: StreamTensor<bf16, m![W, H, C, N], m![1]>) {
    // The compiler generates an identical config for writing
    // The hardware executes the configuration as nested loops:
    for w in 0..8 {
        for h in 0..8 {
            for c in 0..3 {
                for n in 0..4 {
                    // Write to each address
                    let addr = 1 * w + 8 * h + 64 * c + 192 * n;
                    // buf[addr] = stream.next();
                }
            }
        }
    }
}

Configurations

The following patterns cover most configurations a kernel writer is likely to encounter.

Transposing Axes

Axes may be transposed so that the stream visits them in a different order than the buffer, and the compiler computes the strides needed to traverse memory in that order.

#![allow(unused)]
fn main() {
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
use furiosa_opt_std::pseudo::{MemTensor, StreamTensor};
axes![A = 8, B = 8, C = 8];

fn read_rearranging<'l>(
    buf: &'l MemTensor<i8, m![A, B, C # 32]>,  // Buf
) -> StreamTensor<'l, i8, m![B, A], m![C # 16]> {  // Time, Packet
    buf.read()
}

let buf_read = MemTensor::<i8, m![A, B, C # 32]>::from_vec(vec![1i8; 8 * 8 * 32]);
let _stream = read_rearranging(&buf_read);
}

The compiler generates configuration entries by processing the combined mapping m![B, A, C # 16] term by term, transforming Buf along the way. For each term, the entry size equals the term size, and the stride equals the volume that term occupies within the current Buf. After processing a term, Buf is updated to reflect that the axis has been consumed:

Packet::SIZE = max_access_size = 16. The innermost entry 16 : 1 is contiguous, so the hardware transfers the full packet in one access.

Splitting Axes

Tiling breaks a logical axis into sub-blocks for cache efficiency or to match tensor unit buffer sizes, and the compiler achieves this by splitting the axis into multiple entries.

#![allow(unused)]
fn main() {
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
use furiosa_opt_std::pseudo::{MemTensor, StreamTensor};
axes![A = 8, B = 8, C = 4];

fn read_splitting<'l>(
    buf: &'l MemTensor<i8, m![A, B, C # 8]>,  // Buf
) -> StreamTensor<'l, i8, m![A % 2, B % 4, A / 2, B / 4], m![C # 32]> {  // Time, Packet
    buf.read()
}

let buf_read = MemTensor::<i8, m![A, B, C # 8]>::from_vec(vec![1i8; 8 * 8 * 8]);
let _stream = read_splitting(&buf_read);
}

Expressions like A % 2 and A / 2 split axis A into separate entries. The compiler processes m![A % 2, B % 4, A / 2, B / 4, C # 32] term by term:

Packet::SIZE = max_access_size = 32.

Slicing Axes

Slicing reads only a partial range of indices from the memory layout, a condition that arises when an indexed view selects a subset of the original tensor.

extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
use furiosa_opt_std::pseudo::{MemTensor, StreamTensor};
axes![A = 16, B = 8, C = 8];

fn read_slicing<'l>(
    buf: &'l MemTensor<i8, m![A, B, C]>,  // Buf
) -> StreamTensor<'l, i8, m![A / 4, A % 4 = 3, B / 4, B % 4 = 2], m![C]> {  // Time, Packet
    buf.read()
}

let buf_read = MemTensor::<i8, m![A, B, C]>::from_vec(vec![1i8; 16 * 8 * 8]);
let _stream = read_slicing(&buf_read);

The = 3 notation limits A % 4 to only 3 iterations instead of 4, restricting the hardware to a sub-region of the tensor. The compiler processes m![A / 4, A % 4 = 3, B / 4, B % 4 = 2, C] term by term:

Packet::SIZE = max_access_size = 8.

Broadcasting Axes

Broadcasting replicates an element across multiple packets or time steps when the stream visits axes that Buf does not carry. Any axis (or partial-axis fragment like N / 512) present in Time or Packet but absent from Buf becomes a broadcast entry, shown as : 0 in the stride table (the hardware revisits the same address on each iteration).

#![allow(unused)]
fn main() {
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
use furiosa_opt_std::pseudo::{MemTensor, StreamTensor};
axes![A = 16, T = 4, P = 4];

fn read_broadcasting<'l>(
    buf: &'l MemTensor<i8, m![A]>,  // Buf
) -> StreamTensor<'l, i8, m![T, A], m![P]> {  // Time, Packet
    buf.read()
}

let buf_read = MemTensor::<i8, m![A]>::from_vec(vec![1i8; 16]);
let _stream = read_broadcasting(&buf_read);
}

The compiler processes m![T, A, P] term by term:

Packet::SIZE = max_access_size = 4. P is broadcast, so the same element is replicated across the packet (spatial broadcast). T is broadcast, so the same data is repeated across time steps (temporal broadcast).

The same rule applies when Time or Packet references a fragment of an axis that Buf does not carry. For example, a buffer of m![N % 512] read as StreamTensor<m![N / 512], m![N % 512]> broadcasts on the N / 512 time entry: the buffer’s 512 elements are reused across each of the N / 512 outer iterations.

Merging Entries

The hardware supports at most 8 entries per configuration, so when a transformation produces more, the compiler merges adjacent entries to satisfy that limit. Adjacent entries (n1 : s1) and (n2 : s2) merge into (n1 * n2 : s2) when physically contiguous: s1 == n2 * s2.

#![allow(unused)]
fn main() {
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
use furiosa_opt_std::pseudo::{MemTensor, StreamTensor};
axes![N = 8, C = 8, H = 8, W = 32];

fn read_merging<'l>(
    buf: &'l MemTensor<i8, m![N, C, H, W]>,  // Buf
) -> StreamTensor<'l, i8, m![W / 16, H % 2, H / 2, C / 2, C % 2, N / 2, N % 2, W / 8 % 2], m![W % 8]> {  // Time, Packet
    buf.read()
}

let buf_read = MemTensor::<i8, m![N, C, H, W]>::from_vec(vec![1i8; 8 * 8 * 8 * 32]);
let _stream = read_merging(&buf_read);
}

The compiler processes m![W / 16, H % 2, H / 2, C / 2, C % 2, N / 2, N % 2, W / 8 % 2, W % 8] term by term, producing 9 initial entries:

Since 9 entries exceed the hardware limit of 8, the compiler merges contiguous pairs where s1 == n2 * s2. The entries for H % 2 -> (2 : 32) and H / 2 -> (4 : 64) are not merged because they are not physically contiguous (\(s_1 \neq n_2 \times s_2 \iff 32 \neq 4 \times 64\)). The final configuration has 6 entries. The last merge crosses the time/packet boundary: W/8%2 (2:8) and W%8 (8:1) merge into W%16 (16:1).

Packet::SIZE = max_access_size = 8.

Non-Contiguous Packets

When the DM layout has stride discontinuities within the packet span, max_access_size < Packet::SIZE and the hardware issues one access per contiguous sub-block rather than one per packet. The example below writes a packet of 32 elements (m![A, B]) to a buffer where each B row is padded to 16 slots in DM. A’s stride is 16 rather than 8, so the packet span is not contiguous and the hardware issues 4 accesses instead of 1:

#![allow(unused)]
fn main() {
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
use furiosa_opt_std::pseudo::{MemTensor, StreamTensor};
axes![A = 4, B = 8];

fn write_padded(
    buf: &mut MemTensor<i8, m![A, B # 16]>,
    stream: StreamTensor<i8, m![1], m![A, B]>,
) {
    // Compiler-generated configuration: [
    //   A -> 4 : 16,   (16 != 8 * 1, NOT contiguous — padding gap after each B row)
    //   B -> 8 : 1,    (packet sub-block, contiguous)
    // ] : 32
    buf.write(stream)
}

let buf_read = MemTensor::<i8, m![A, B]>::from_vec(vec![1i8; 4 * 8]);
let mut buf_write = MemTensor::<i8, m![A, B # 16]>::from_vec(vec![0i8; 4 * 16]);
let stream = buf_read.read();
write_padded(&mut buf_write, stream);
}

Packet::SIZE = 32, contiguous_run = 8, max_access_size = 8.

Non-contiguous strides also arise when Packet contains non-adjacent axes from the source layout. The example below reads the same m![N, C, H, W] buffer with two different Packet choices. Placing only the innermost axis W in Packet gives max_access_size = Packet::SIZE = 8, one access per packet. Placing m![N, H, W] in Packet skips C, so N’s stride in source (96) does not equal H×W (32): contiguous_run = 32, max_access_size = 32, and the hardware issues 4 accesses per packet instead of 1.

#![allow(unused)]
fn main() {
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
use furiosa_opt_std::pseudo::{MemTensor, StreamTensor};
axes![N = 4, C = 3, H = 4, W = 8];

// Compiler-generated configuration: [
//   N -> 4 : 96,   (96 == 3 × 32, contiguous)
//   C -> 3 : 32,   (32 == 4 × 8,  contiguous)
//   H -> 4 : 8,    (8  == 8 × 1,  contiguous)
//   W -> 8 : 1,    (packet dimension)
// ] : 8
// contiguous_run = 8 (W); ×4 (H): 8==8×1 ✓; ×3 (C): 32==4×8 ✓; ×4 (N): 96==3×32 ✓; all axes contiguous
// max_access_size = gcd(packet_size, contiguous_run) = packet_size = 8
fn read_contiguous<'l>(
    buf: &'l MemTensor<i8, m![N, C, H, W]>,
) -> StreamTensor<'l, i8, m![N, C, H], m![W]> {
    buf.read()
}

let buf_read = MemTensor::<i8, m![N, C, H, W]>::from_vec(vec![1i8; 4 * 3 * 4 * 8]);
let _stream = read_contiguous(&buf_read);

// Compiler-generated configuration: [
//   C -> 3 : 32,   (time dimension)
//   N -> 4 : 96,   (96 != 4 × 8 = 32, NOT contiguous — C axis interspersed)
//   H -> 4 : 8,    (8  == 8 × 1, contiguous)
//   W -> 8 : 1,    (packet dimension)
// ] : 128
// contiguous_run = 8 (W); ×4 (H): 8==8×1 ✓ → 32; ×4 (N): 96!=4×8 ✗ stop → 32
// max_access_size = gcd(128, 32) = 32; hardware issues 128/32 = 4 accesses per packet
fn read_non_contiguous<'l>(
    buf: &'l MemTensor<i8, m![N, C, H, W]>,
) -> StreamTensor<'l, i8, m![C], m![N, H, W]> {
    buf.read()
}

let buf_read = MemTensor::<i8, m![N, C, H, W]>::from_vec(vec![1i8; 4 * 3 * 4 * 8]);
let _stream = read_non_contiguous(&buf_read);
}

Constraints

In RNGD, exceeding any of the following hardware limits causes a compilation error:

• Entry limit: Maximum 8 entries, so the compiler merges adjacent entries where possible (see Merging Entries in Configurations).
• Iteration limit: size <= 65,536 per entry.
• Packet size: Must be 1, 2, 4, 8, 16, or 32 bytes.
• Packet fetch: The innermost entry n : s must satisfy one of:
  • Contiguous access (adjacent elements): (s == 0 || s == 1) && n % packet_size == 0
  • Discrete access (single-element packets): packet_size == 1

If merging fails or limits are exceeded, redesign the tensor mapping or split the operation across multiple sequencer calls.

Compatible Axis Decompositions

Each axis named in both Buf and the stream must use the same decomposition. The compiler walks the stream term by term and consumes axes from Buf (see Architecture). When Buf splits an axis one way and the stream splits it another with no common refinement, no traversal order works and the configuration is rejected, even when both sides have the same total element count.

#![allow(unused)]
fn main() {
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
use furiosa_opt_std::pseudo::{MemTensor, StreamTensor};
axes![A = 15];

fn read_incompatible<'l>(
    buf: &'l MemTensor<i8, m![A % 5, A / 5]>,  // Buf
) -> StreamTensor<'l, i8, m![1], m![A % 3, A / 3]> {  // Time, Packet
    buf.read() // Compilation error: incompatible decomposition
}

let buf_read = MemTensor::<i8, m![A % 5, A / 5]>::from_vec(vec![1i8; 15]);
let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { read_incompatible(&buf_read) }));
assert!(result.is_err());
}

Buf decomposes A as 5 × 3 while the stream decomposes it as 3 × 5. Since gcd(5, 3) = 1, neither decomposition refines the other: the compiler cannot consume A % 3 from a Buf that has already committed to a 5-block split.

Indirect Access

All entries above use fixed strides: the memory offset between iterations is constant. IndirectLoop extends this by allowing variable offsets per iteration, enabling gather operations with data-dependent access patterns.

The standard pattern (limit, stride) becomes (limit, [offset0, offset1, ...]), where each iteration uses a different offset from the provided sequence. This supports operations like embedding lookups where indices are determined at runtime.

Fetch Engine

The Fetch Engine reads a DM tensor and produces a packet stream for the Tensor Unit, a mathematical tensor move that reads DM with per-slice sequencers and emits a FetchTensor.

Interface

BeginTensor represents a tensor resident in DM, at the entry of the Tensor Unit pipeline. Its Time is m![1] (no temporal iteration before the pipeline starts) and Packet is the element layout in DM.

BeginTensor::fetch() runs the sequencer and produces a FetchTensor packet stream that feeds the Fetch Adapter, the Switch Engine, or the Collect Engine. The assert_eq! calls enforce hardware constraints on Cluster::SIZE, Slice::SIZE, and packet alignment (see Constraints).

impl<'l, const T: Tu, P: CanApplyFetch, D: Scalar, Chip: M, Cluster: M, Slice: M, Time: M, Packet: M, B: Backend>
    TuTensor<'l, T, P, D, Chip, Cluster, Slice, Time, Packet, B>
{
    /// Runs the Fetch Sequencer.
    ///
    /// Configures per-slice DM reads and produces a `FetchTensor` with the
    /// chosen `OutTime` / `OutPacket`. The element type is unchanged. Type
    /// casts and other adapter transforms are applied by the per-stage
    /// `fetch_mask` / `fetch_table_lookup` / `fetch_cast` methods.
    #[primitive(TuTensor::fetch)]
    pub fn fetch<OutTime: M, OutPacket: M>(self) -> FetchTensor<'l, T, D, Chip, Cluster, Slice, OutTime, OutPacket, B> {
        verify_fetch::<Cluster, Slice, Time, Packet, OutTime, OutPacket>();
        FetchTensor::new(self.ctx, self.inner.transpose(true))
    }
}

As introduced in Mapping Tensors, the Chip, Cluster, Slice, Time, Packet mapping distributes data across space and time. .fetch() preserves the Chip, Cluster, and Slice dimensions unchanged from the input, because each slice independently reads its own DM partition. Later the Switch Engine changes the Slice mapping by moving data across slices.

fetch() takes OutTime and OutPacket type parameters that configure the Fetch Sequencer. OutTime sets the number of time steps in the output stream, and OutPacket sets the element layout within each packet. For performance implications of OutPacket choices, see Optimizations.

The following example fetches an i8 matrix from DM as an i8 packet stream. The output FetchTensor streams 512 time steps, each a 32-element i8 packet (32 bytes). Here OutTime = m![A] and OutPacket = m![B].

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![CH = 4, CL = 2, S = 256, A = 512, B = 32];

fn fetch_matrix_example<'l, const T: Tu>(
    input: BeginTensor<'l, T, i8, m![CH], m![CL], m![S], m![1], m![A, B]>,
) -> FetchTensor<'l, T, i8, m![CH], m![CL], m![S], m![A], m![B]> {
    input.fetch::<m![A], m![B]>()
}
}

Chip, Cluster, and Slice are the hardware spatial parallelism dimensions. A Fetch Sequencer runs independently in every slice, each operating on its own local DM partition. In the example above, Chip = m![CH], Cluster = m![CL], and Slice = m![S] (with CH = 4, CL = 2, S = 256) reflect a 4-chip RNGD system with 2 clusters per chip and 256 slices per cluster (2,048 slices total), each running the same sequencer pattern on its own A×B sub-tensor.

Constraints

• Hardware dimensions: Chip::SIZE, Cluster::SIZE, and Slice::SIZE must match the hardware configuration (see Sequencer).

Multi-Read Packet

Preparing a packet may require multiple hardware reads because packet axes may not be contiguous in DM, and the hardware reads at most 32 bytes at once. In the main-context, read_size is the largest divisor of the sequencer’s max_access_size (see Sequencer Architecture for max_access_size) such that D[read_size] is 1, 2, 4, 8, 16, or 32 bytes. In the sub-context, read_size is fixed at 8 bytes. The compiler derives read_size from the input element type of fetch() (and from any downstream Fetch Adapter cast) and users do not set it directly. Multi-read occurs whenever Packet::SIZE > read_size. For example, a 24-byte packet in the main-context forces read_size = 8 and 3 reads per packet. The total cycle count is Time::SIZE * (Packet::SIZE / read_size).

The following examples fetch the same i4 tensor of shape m![N, C, H, W] (with N=4, C=3, H=4, W=16) using four different OutPacket choices.

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![N = 4, C = 3, H = 4, W = 16];

/// Sequencer config: [N = 4 : 192, C = 3 : 64, H = 4 : 16, W = 16 : 1].
/// max_access_size = 16; read_size = 16 (8 bytes); reads per packet = 1; cycles = 48
fn fetch_batch_1<'l, const T: Tu>(
    input: BeginTensor<'l, T, i4, m![1], m![1 # 2], m![1 # 256], m![1], m![N, C, H, W]>,
) -> FetchTensor<'l, T, i4, m![1], m![1 # 2], m![1 # 256], m![N, C, H], m![W]> {
    input.fetch()
}

/// Sequencer config: [N = 4 : 192, C = 3 : 64, H / 2 = 2 : 32, H % 2 = 2 : 16, W = 16 : 1].
/// max_access_size = 32; read_size = 32 (16 bytes); reads per packet = 1; cycles = 24
fn fetch_batch_2<'l, const T: Tu>(
    input: BeginTensor<'l, T, i4, m![1], m![1 # 2], m![1 # 256], m![1], m![N, C, H, W]>,
) -> FetchTensor<'l, T, i4, m![1], m![1 # 2], m![1 # 256], m![N, C, H / 2], m![H % 2, W]> {
    input.fetch()
}

/// Sequencer config: [N = 4 : 192, C = 3 : 64, H = 4 : 16, W = 16 : 1].
/// max_access_size = 64; read_size = 64 (32 bytes); reads per packet = 1; cycles = 12
fn fetch_batch_3<'l, const T: Tu>(
    input: BeginTensor<'l, T, i4, m![1], m![1 # 2], m![1 # 256], m![1], m![N, C, H, W]>,
) -> FetchTensor<'l, T, i4, m![1], m![1 # 2], m![1 # 256], m![N, C], m![H, W]> {
    input.fetch()
}

/// Sequencer config: [N = 4 : 192, C = 3 : 64, H = 4 : 16, W = 16 : 1].
/// max_access_size = 192; read_size = 64 (32 bytes); reads per packet = 3; cycles = 12
fn fetch_batch_4<'l, const T: Tu>(
    input: BeginTensor<'l, T, i4, m![1], m![1 # 2], m![1 # 256], m![1], m![N, C, H, W]>,
) -> FetchTensor<'l, T, i4, m![1], m![1 # 2], m![1 # 256], m![N], m![C, H, W]> {
    input.fetch()
}

let mut ctx = Context::acquire();

let b: BeginTensor<'_, _, i4, m![1], m![1 # 2], m![1 # 256], m![1], m![N, C, H, W]> = BeginTensor::new(&mut ctx.main, Tensor::zero());
let _o = fetch_batch_1(b);

let b: BeginTensor<'_, _, i4, m![1], m![1 # 2], m![1 # 256], m![1], m![N, C, H, W]> = BeginTensor::new(&mut ctx.main, Tensor::zero());
let _o = fetch_batch_2(b);

let b: BeginTensor<'_, _, i4, m![1], m![1 # 2], m![1 # 256], m![1], m![N, C, H, W]> = BeginTensor::new(&mut ctx.main, Tensor::zero());
let _o = fetch_batch_3(b);

let b: BeginTensor<'_, _, i4, m![1], m![1 # 2], m![1 # 256], m![1], m![N, C, H, W]> = BeginTensor::new(&mut ctx.main, Tensor::zero());
let _o = fetch_batch_4(b);
}

Interleaving

Interleaving combines two tensors with identical mappings into a single sequencer operation, reducing overhead when both tensors are needed for the same computation. An explicit Time axis encodes alternation between the two tensors.

In the following example, the main-context creates an interleaved tensor using begin_interleaved(). The first temporal iteration fetches from lhs, the second from rhs, the third from lhs again, and so on. At most two tensors can be interleaved in a single fetch operation.

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 16, B = 32, I = 2];

/// Interleaves two input tensors into a single packet stream.
/// Useful for operations like 'input1 + input2' in the Vector Engine.
/// The interleaved BeginTensor is created via Tu.begin_interleaved().
/// The `I = 2` axis in Time encodes alternation between the two tensors.
fn fetch_interleaved<'l>(
    ctx: &'l mut Context,
    lhs: &'l DmTensor<i8, m![1], m![1 # 2], m![1 # 256], m![A, B]>,
    rhs: &'l DmTensor<i8, m![1], m![1 # 2], m![1 # 256], m![A, B]>,
) -> FetchTensor<'l, { Tu::Main }, i8, m![1], m![1 # 2], m![1 # 256], m![A, I], m![B]> {
    ctx.main.begin_interleaved::<I, _, _, _, _, _>(lhs.view(), rhs.view()).fetch()
}

let mut ctx = Context::acquire();

let lhs = unsafe { DmTensor::from_addr(0) };
let rhs = unsafe { DmTensor::from_addr(0) };
let _o = fetch_interleaved(&mut ctx, &lhs, &rhs);
}

Optimizations

Three factors determine Fetch Sequencer throughput.

• Input bandwidth: read_size is limited by axis contiguity in DM and packet size. Non-adjacent axes reduce max_access_size and therefore read_size (see Non-Contiguous Packets). A packet smaller than the contiguous run also limits read_size. Padding to a larger power-of-two raises it (see Packet padding).

  Furthermore, access patterns that hit the same bank 64 or more times consecutively starve the lower-priority Commit Engine and DMA Engine and can cause catastrophic NoC timeouts.

  See Memory Performance for details.

• Output bandwidth: the downstream Collect Engine converts Fetch’s packets to 32-byte flits, so packet sizes that don’t align to 32 bytes waste bandwidth. A 20-byte packet fills one flit with 12 bytes of zero-padding, wasting 12 / 32 = 37.5%. A 40-byte packet spans two flits (64 bytes total) and zero-pads the final 24 bytes of the second flit, wasting 24 / 64 = 37.5%.

• Spatial parallelism: Distributing fetches across slices maximizes throughput.

Example: Packet padding

Padding OutPacket to a larger power-of-two element count can increase read_size. The three examples below fetch the same 30-byte tensor in 15, 3, and 1 cycles by growing the packet from 2 to 16 to 32 bytes:

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 3, B = 5, C = 2];

/// Smallest packet: only C dimension padded to 8bytes. Takes 15 cycles.
fn fetch_packet_C<'l, const T: Tu>(
    input: BeginTensor<'l, T, f8e4m3, m![1], m![1 # 2], m![1 # 256], m![1], m![A, B, C]>,
) -> FetchTensor<'l, T, f8e4m3, m![1], m![1 # 2], m![1 # 256], m![A, B], m![C # 8]> {
    input.fetch()
}

/// Medium packet: B and C dimensions padded to 16 bytes. Takes 3 cycles.
fn fetch_packet_BC<'l, const T: Tu>(
    input: BeginTensor<'l, T, f8e4m3, m![1], m![1 # 2], m![1 # 256], m![1], m![A, B, C]>,
) -> FetchTensor<'l, T, f8e4m3, m![1], m![1 # 2], m![1 # 256], m![A], m![[B, C] # 16]> {
    input.fetch()
}

/// Largest packet: all dimensions padded to 32 bytes. Takes 1 cycle.
fn fetch_packet_ABC<'l, const T: Tu>(
    input: BeginTensor<'l, T, f8e4m3, m![1], m![1 # 2], m![1 # 256], m![1], m![A, B, C]>,
) -> FetchTensor<'l, T, f8e4m3, m![1], m![1 # 2], m![1 # 256], m![1], m![[A, B, C] # 32]> {
    input.fetch()
}


let mut ctx = Context::acquire();
let x: BeginTensor<'_, _, f8e4m3, m![1], m![1 # 2], m![1 # 256], m![1], m![A, B, C]> = BeginTensor::new(&mut ctx.main, Tensor::zero());
let _o = fetch_packet_C(x);
let y: BeginTensor<'_, _, f8e4m3, m![1], m![1 # 2], m![1 # 256], m![1], m![A, B, C]> = BeginTensor::new(&mut ctx.main, Tensor::zero());
let _o = fetch_packet_BC(y);
let z: BeginTensor<'_, _, f8e4m3, m![1], m![1 # 2], m![1 # 256], m![1], m![A, B, C]> = BeginTensor::new(&mut ctx.main, Tensor::zero());
let _o = fetch_packet_ABC(z);
}

In these examples, padding reads beyond the actual data, but this is safe because padding values do not affect computation. Different padding strategies produce different FetchTensor mappings, which may affect downstream components.

Commit Engine

The Commit Engine writes a Tensor Unit stream packet to DM, the inverse of the Fetch Engine. It is the Commit Sequencer: a mathematical tensor move that runs independently in every slice, each writing to its own local DM partition.

Interface

A TuTensor carries Chip, Cluster, Slice, Time, and Packet dimensions at the end of the Tensor Unit pipeline. Its Time reflects the temporal unrolling of the computation, and Packet is the element layout in the output stream.

.commit() writes the stream to a DmTensor in DM.

impl<'l, const T: Tu, P: CanApplyCommit, D: Scalar, Chip: M, Cluster: M, Slice: M, Time: M, Packet: M, B: Backend>
    TuTensor<'l, T, P, D, Chip, Cluster, Slice, Time, Packet, B>
{
    /// Commits to data memory.
    #[primitive(TuTensor::commit)]
    pub fn commit<Element: M>(self) -> DmTensor<D, Chip, Cluster, Slice, Element, B> {
        verify_commit::<D, Time, Packet, Element>();
        DmTensor::new(self.inner.transpose(false), None)
    }

    /// Commits to data memory at `address`.
    #[primitive(TuTensor::commit_at)]
    pub fn commit_at<Element: M>(self, address: Address) -> DmTensor<D, Chip, Cluster, Slice, Element, B> {
        verify_commit::<D, Time, Packet, Element>();
        DmTensor::new(self.inner.transpose(false), Some(address))
    }

    /// Commits to a mutable tensor view in data memory.
    #[primitive(TuTensor::commit_view)]
    pub fn commit_view<Element: M>(self, mut dst: DmTensorViewMut<'l, D, Chip, Cluster, Slice, Element, B>) {
        verify_commit::<D, Time, Packet, Element>();
        dst.inner.transpose(self.inner.view(), false);
    }
}

.commit() preserves the Chip, Cluster, and Slice dimensions unchanged, because each slice independently writes to its own DM partition. The output Element mapping replaces Time and Packet, defining how the stream is laid out in DM. Element configures both the Commit Sequencer and the Commit Adapter, and can reorder Time axes relative to the input stream, performing a transpose during the commit. For performance implications of the Element mapping, see Optimizations.

The following example commits a cast accumulation result to DM as bf16. The output DmTensor stores 16 time steps × 8 bf16 elements across 256 slices. Here D = bf16 and Element = m![M, N # 16].

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![P = 256, M = 16, N = 8];

fn cast_commit<'l, const T: Tu>(
    input: ContractTensor<'l, T, f32, m![1], m![1 # 2], m![P], m![M], m![N]>,
) -> DmTensor<bf16, m![1], m![1 # 2], m![P], m![M, N # 16]> {
    // Cast f32 to bf16 (Cast Engine), then commit to DM (Commit Engine).
    // Input: M = 16 time steps, N = 8 f32 elements per packet (32 bytes).
    // After cast: N = 8 bf16 elements padded to 16 (32 bytes).
    // After trim: N # 16 trimmed into N = 8.
    // The sequencer writes across P = 256 slices.
    input.cast::<bf16, m![N # 16]>().commit_trim::<m![N]>().commit()
}

let mut ctx = Context::acquire();

let c: ContractTensor<'_, _, f32, m![1], m![1 # 2], m![P], m![M], m![N]> = ContractTensor::new(&mut ctx.main, Tensor::zero());
let _o = cast_commit(c);
}

Constraints

• Hardware dimensions: Chip::SIZE, Cluster::SIZE, and Slice::SIZE must match the hardware configuration (see Sequencer).
• Address alignment: All sequencer strides must be multiples of 8 bytes.
• Write unit alignment: D[valid_size] must be 8, 16, 24, or 32 bytes (see the Commit Adapter’s Trimming stage).

Multi-Write Packet

Writing a packet may require multiple hardware writes because packet axes may not be contiguous in DM. The per-write element count write_size = gcd(valid_size, access_size) is derived by the compiler, where valid_size comes from the Commit Adapter and access_size from the Sequencer Architecture. In the sub-context, D[write_size] is fixed at 8 bytes. The total cycle count is Time::SIZE * (valid_size / write_size). The division is always exact: in the main-context, valid_size == write_size, so each packet commits in a single cycle. In the sub-context, write_size is fixed at 8 bytes and valid_size is one of 8, 16, 24, or 32 bytes (from the trimming constraint), so valid_size / write_size is always 1, 2, 3, or 4.

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![M = 4, K = 2, W = 8, N = 16, L = 32];

// Compiler-generated configuration: [
//   M -> 4 : 64,  (64 == 2 * 32,  contiguous)
//   K -> 2 : 32,   (32  == 32 * 1,  contiguous)
//   M -> 32 : 1    (packet dimension, contiguous)
// ] : 8
// access_size = 64; valid_size = 8; write_size = gcd(64, 8) = 8; writes per packet = 1
fn no_transpose<'l, const T: Tu>(
    input: CastTensor<'l, T, i8, m![1], m![1 # 2], m![1 # 256], m![M, K], m![L]>,
) -> DmTensor<i8, m![1], m![1 # 2], m![1 # 256], m![M, K, L]> {
    input.commit_trim::<m![L]>().commit()
}

// Compiler-generated configuration: [
//   M -> 4 : 8,   (8  != 2 * 32, NOT contiguous)
//   K -> 2 : 32,  (32 != 8 * 1,  NOT contiguous)
//   W -> 8 : 1    (packet dimension, contiguous)
// ] : 32
// access_size = 8; valid_size = 8; write_size = gcd(8, 8) = 8; writes per packet = 1
fn transpose<'l, const T: Tu>(
    input: ContractTensor<'l, T, f32, m![1], m![1 # 2], m![1 # 256], m![M, K], m![W]>,
) -> DmTensor<f32, m![1], m![1 # 2], m![1 # 256], m![K, M, W]> {
    input.commit_trim::<m![W]>().commit()
}

// Compiler-generated configuration: [
//   M -> 4 : 8,   (8  != 2 * 32, NOT contiguous)
//   K -> 2 : 32,  (32 != 8 * 1,  NOT contiguous)
//   N -> 8 : 1    (trimmed packet dimension, contiguous)
// ] : 16
// access_size = 8; valid_size = 8 (trimmed from 16); write_size = gcd(8, 8) = 8; writes per packet = 1
fn transpose_with_trimming<'l, const T: Tu>(
    input: CastTensor<'l, T, i8, m![1], m![1 # 2], m![1 # 256], m![M, K], m![N # 32]>,
) -> DmTensor<i8, m![1], m![1 # 2], m![1 # 256], m![K, M, N]> {
    input.commit_trim::<m![N]>().commit()
}


let mut ctx = Context::acquire();
let a: CastTensor<'_, _, i8, m![1], m![1 # 2], m![1 # 256], m![M, K], m![L]> = CastTensor::new(&mut ctx.main, Tensor::zero());
let _o = no_transpose(a);
let b: ContractTensor<'_, _, f32, m![1], m![1 # 2], m![1 # 256], m![M, K], m![W]> = ContractTensor::new(&mut ctx.main, Tensor::zero());
let _o = transpose(b);
let c: CastTensor<'_, _, i8, m![1], m![1 # 2], m![1 # 256], m![M, K], m![N # 32]> = CastTensor::new(&mut ctx.main, Tensor::zero());
let _o = transpose_with_trimming(c);
}

Slice Bitmap

The slice bitmap is a 256-bit mask covering one full cluster (one bit per slice, 256 slices per cluster) that gates which slices receive commit data. For example, bitmap = 00000000...01 enables commit only to slice 0, and bitmap = 11111111...10 enables commit to all slices except slice 0.

Optimizations

Three factors determine Commit Sequencer throughput.

• Sequential Addresses: Writing to sequential DM addresses within each slice enables parallel bank access (128 B/cycle per DMN, 256 B/cycle with DMN interleaving). Patterns that hit the same bank 64+ times consecutively trigger DM Bank Starvation.
• Spatial parallelism: Distributing writes across all active slices maximizes throughput.
• Aligned writes (invariant): Partial bank writes never occur, because both the write address and the write unit are always 8-byte aligned. Sequencer strides are multiples of 8 bytes (see Constraints), and the Commit Adapter’s Trimming stage holds D[valid_size] to a multiple of 8 bytes.

DMA Engine

The DMA Engine moves tensors directly between memory tiers without engaging the Tensor Unit pipeline. Each transfer pairs two coordinated stages:

• Read Sequencer: Reads from the source tier.
• Write Sequencer: Writes to the destination tier, possibly with a layout transformation.

A DMA transfer is a mathematical tensor move: the output holds the same mathematical tensor as the input even when the layouts differ. Tensor DMA spans cross-DMN, cross-cluster, and cross-chip transfers, with chip IDs globally agreed across the system.

See Optimizations for transfer throughput considerations.

Interface

A DMA transfer takes a tensor in one memory tier and produces a tensor in another (or the same) tier. The kernel writer calls .to_dm(), .to_hbm(), or related methods on the source tensor, passing in a DmaContext:

• Context::tdma: Tensor DMA context for on-chip transfers (HBM ↔ HBM, HBM ↔ DM, DM ↔ DM).
• Context::pdma: PCIe DMA context for host ↔ HBM transfers (see PCIe DMA).

impl<D: Scalar, Chip: M, Element: M, B: Backend> HbmTensor<D, Chip, Element, B> {
    /// Converts to data memory tensor.
    #[primitive(HbmTensor::to_dm)]
    pub fn to_dm<Cluster: M, Slice: M, Element2: M>(
        &self,
        _dma: &mut DmaContext<{ Dma::Tensor }>,
    ) -> DmTensor<D, Chip, Cluster, Slice, Element2, B> {
        assert_dma_layout::<
            D,
            m![{ Chip }, { Element }],
            Element,
            m![{ Chip }, { Cluster }, { Slice }, { Element2 }],
            Element2,
        >(DMA_SRAM_WRITE_WIDTH);
        DmTensor::new(self.inner.transpose(true), None)
    }

    /// Converts to data memory tensor at `address`.
    #[primitive(HbmTensor::to_dm_at)]
    pub fn to_dm_at<Cluster: M, Slice: M, Element2: M>(
        &self,
        _dma: &mut DmaContext<{ Dma::Tensor }>,
        address: Address,
    ) -> DmTensor<D, Chip, Cluster, Slice, Element2, B> {
        assert_dma_layout::<
            D,
            m![{ Chip }, { Element }],
            Element,
            m![{ Chip }, { Cluster }, { Slice }, { Element2 }],
            Element2,
        >(DMA_SRAM_WRITE_WIDTH);
        DmTensor::new(self.inner.transpose(true), Some(address))
    }

    /// Reshapes the tensor to a different mapping at the same HBM address, consuming `self`.
    /// The HBM analogue of [`DmTensor::reshape`]; both delegate to [`Tensor::reshape`].
    ///
    /// # Safety
    ///
    /// The per-level sizes (`Chip::SIZE == Chip2::SIZE`, `Element`) are asserted at compile time below
    /// (see [`constraints::assert_hbm_reshape_dimension_preserved`]); the genuine precondition is
    /// [`Tensor::reshape`]'s: the old and new mappings must lay the elements out in the SAME physical
    /// (wire) order, so the relabel moves no data. Axis regrouping (merge/split) preserves wire order
    /// and is valid; a permutation is not (use a transpose). Equal sizes do not guarantee this.
    /// Consuming `self` is the safety contract made explicit: the old-shaped handle cannot survive to
    /// alias the same HBM bytes under a conflicting mapping.
    #[primitive(HbmTensor::reshape)]
    pub unsafe fn reshape<Chip2: M, Element2: M>(self) -> HbmTensor<D, Chip2, Element2, B> {
        constraints::assert_hbm_reshape_dimension_preserved::<Chip, Chip2, Element, Element2>();
        let reshaped = unsafe { self.inner.reshape::<m![{ Chip2 }, { Element2 }]>() };
        HbmTensor::new(reshaped, self.address)
    }
}

The compiler derives the read and write sequencer configurations from the source and destination tensor types. The kernel writer specifies the destination type’s Cluster, Slice, and Element (for DM tensors) or Element (for HBM tensors), which encode the layout transformation.

The example below transposes a tensor from [A, B, C] to [C, A, B] using two HBM-to-HBM transfers:

#![allow(unused)]
fn main() {
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 8, B = 16, C = 32];

fn transpose_simple(
    ctx: &mut Context,
    input: &HbmTensor<f32, m![1], m![A, B, C]>,
) -> HbmTensor<f32, m![1], m![C, A, B]> {
    // Step 1: [A, B, C] → [A, C, B]
    let intermediate: HbmTensor<f32, m![1], m![A, C, B]> = input.to_hbm(&mut ctx.tdma);

    // Step 2: [A, C, B] → [C, A, B]
    intermediate.to_hbm(&mut ctx.tdma)
}

let mut ctx = Context::acquire();
let in_hbm = unsafe { HbmTensor::<f32, m![1], m![A, B, C]>::from_addr(0) };
let _out_hbm = transpose_simple(&mut ctx, &in_hbm);
}

A transfer that crosses tiers also takes a layout transformation through the destination type’s mapping. For an HBM-to-DM transfer, the destination DM tensor adds Cluster and Slice axes that distribute and broadcast the tensor across hardware partitions.

#![allow(unused)]
fn main() {
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 2048];

fn hbm_to_dm(
    ctx: &mut Context,
    input: &HbmTensor<i8, m![1], m![A]>,
) -> DmTensor<i8, m![1], m![1 # 2], m![A / 8], m![A % 8]> {
    input.to_dm::<m![1 # 2], m![A / 8], m![A % 8]>(&mut ctx.tdma)
}

let mut ctx = Context::acquire();
let in_hbm = unsafe { HbmTensor::<i8, m![1], m![A]>::from_addr(0) };
let _out_dm = hbm_to_dm(&mut ctx, &in_hbm);
}

Here the 2,048-element vector is distributed as 256 elements per slice (Slice = m![A / 8]) with 8 elements per slice (Element = m![A % 8]), spread across 2 clusters.

Architecture

Each RNGD chip holds 8 DMA Engines, one per pair of DMNs, running up to 8 independent transfers in parallel. A single DMA Engine runs paired read and write sequencers, and a tensor move spreads across multiple engines through an aggregate. The subsections below describe its static structure, sequencer representation, dynamic behavior, compiler derivation, and aggregate operations.

Note

The Tensor Unit (via Fetch and Commit Engines) is often more efficient than DMA for SRAM-to-SRAM transfers, since DMA may underutilize SRAM slice bandwidth. HBM bandwidth, however, is typically the bottleneck in practice, making this gap less critical for HBM ↔ DM transfers.

Static Structure

Chip, Cluster, and Slice are the hardware spatial parallelism dimensions. The 8 DMA Engines per chip transfer between different memory components in parallel (e.g., engine #0 handles HBM ↔ DM while engine #1 handles DM ↔ DM).

Each DMA Engine runs paired read and write sequencers in lockstep. The read sequencer traverses source addresses, the write sequencer traverses destination addresses. Both share the same loop count but use different strides and base addresses, since the layout transformation reorders how the same logical elements appear in source vs. destination memory. The compiler represents the pair compactly as a single sequencer with paired strides per loop entry, exploiting the matched read and write counts.

The compiler distributes a single tensor move across the available DMA Engines, partitioning the work along chip, cluster, and slice dimensions and assigning each partition to a DMA Engine. Any DMA Engine handles any transfer. By default the compiler picks the source DM’s local DMA Engine, since local DMN access is faster than cross-DMN access. The kernel writer can also specify an engine explicitly.

Sequencer Representation

The compiler represents each DMA Engine’s work as a DmaSequencer paired with source and destination addressing:

struct DmaSequencer {
    entries: Vec<DmaEntry>,
    stride0: u16,        // 1..=4096, per-iteration packet size in bytes
    source_base: usize,
    dest_base: usize,
}

struct DmaEntry {
    axis: AxisName,
    size: usize,
    source_stride: isize,
    dest_stride: isize,
}

Each entry specifies a loop with a shared size but separate source_stride and dest_stride, since the layout transformation reorders the same logical elements between source and destination memory. The innermost-loop stride stride0 (1 to 4,096 bytes) sets the per-iteration packet size. A full DMA command bundles the sequencer with the engine’s location and media:

struct DmaDescriptor {
    sequencer: DmaSequencer,
    source_media: Media,
    dest_media: Media,
}

struct DmnIndex {
    chip: ChipIndex,
    cluster_in_chip: ClusterInChipIndex,
    slice_in_cluster: SliceInClusterIndex,
}

enum Media {
    Hbm(ChipIndex),
    Dm(DmnIndex),
    Spm(DmnIndex),
}

enum Dtype {
    I4, I8, F8E4M3, F8E5M2, I16, Bf16, F16, I32, F32,
}

A homogeneous aggregate uses one descriptor template parameterized across all participating DMA Engines. A heterogeneous aggregate uses a HashMap<DmnIndex, DmaDescriptor> that pairs each DMN with its specific descriptor. DM tensor specifications must include chip, cluster, and slice in the mapping expression to identify the exact memory location.

Dynamic Behavior

Each loop in the DmaSequencer advances a counter in row-major order, deriving the read and write addresses from the paired strides. For the sequencer below with base = (0, 256³):

[
  A -> 256 : (65,536, 256),
  B -> 256 : (256, 65,536),
  C -> 256 : (1, 1),
] : 256

With stride0 = 256, the hardware reads and writes 256 bytes per iteration, so iteration 0 processes all values for (A, B, C) = (0, 0, 0..255) as a single packet. The transfer completes in approximately 500 cycles of startup latency plus 256 × 256 cycles of data transfer.

Compiler Derivation

Given source and destination tensor mappings (In, Out) and a stream shape (Stream), the compiler derives the read and write sequencers:

• Read sequencer: Projects In onto the stream shape, producing per-loop strides into the source tier.
• Write sequencer: Projects Out onto the stream shape, producing per-loop strides into the destination tier.
• Unified sequencer: Merges the two so each entry pairs the read and write strides.
• Packet size: Infers stride0 from the consecutive read/write volume. When both read and write access 256 consecutive bytes, the optimal stride0 is 256.

For the layout transformation m![A, B, C] → m![B, A, C] over axes![A=256, B=256, C=256] with Stream = m![A, B, C], the compiler uses the index relation m![A, B, C]::map(i) = i![A: i / 65,536, B: (i % 65,536) / 256, C: i % 256] (see Mapping Expressions for the notation) to derive:

read_sequencer  = [
  A -> 256 : 65,536,
  B -> 256 : 256,
  C -> 256 : 1,
] : 256, HBM @ 0
write_sequencer = [
  A -> 256 : 256,
  B -> 256 : 65,536,
  C -> 256 : 1,
] : 256, HBM @ 256³

These combine into a single DmaSequencer with paired strides per entry. Each side of the unified DmaSequencer (read or write) can be displayed as a single-stride sequencer for that direction, omitting the paired-stride bracket for clarity.

Aggregate Operations

The aggregate takes one of two forms based on whether tensor shapes divide evenly across DMNs.

When shapes divide evenly, all participating engines run a homogeneous aggregate. Every DMA Engine uses the same parametric stream environment (Stream = { chip, cluster, slice, time, packet }), differing only by base address.

When shapes do not divide evenly, the compiler falls back to a heterogeneous aggregate. Each DMN gets its own stream environment via StreamFn(chip, cluster, slice), and boundary DMNs split their work across multiple DMA commands to avoid writing past the valid region. The input and output mapping environments (In and Out) remain structurally identical to the homogeneous case, so the overall logical tensor move is well-defined.

Two correctness invariants apply across both forms:

• Same media types: All participating DMA Engines must use the same source and destination media.
• Single unified mapping: One input and one output tensor mapping govern the overall transfer.

Each command incurs its own startup latency, so prefer tensor shapes that divide evenly across DMNs to keep the aggregate homogeneous.

Constraints

The DMA Engine enforces hardware-level alignment and packet-size rules. Violations cause correctness errors or hardware exceptions, not just performance degradation.

• Address alignment:

  Tier	Read	Write
  HBM	1 byte	1 byte
  DM (SRAM)	1 byte	8 bytes

  HBM ↔ DM transfers additionally require 8-byte alignment for the read address, write address, and packet size, regardless of the table above. The asymmetric DM rule reflects asymmetric SRAM hardware. Read ports use byte-select logic to extract arbitrary byte ranges, but write ports operate on full 8-byte bank-width units. Misaligned DM writes therefore trigger a Read-Modify-Write operation that triples the write time and blocks other operations on the affected bank.

  The compiler enforces these constraints as hardware invariants.

• Packet size: The maximum packet size is 4,096 bytes, set by the AXI protocol constraint that transactions cannot exceed 256 beats × 16-byte data width.

Optimizations

Three factors determine DMA throughput: memory bandwidth, channel and DMN interleaving, and startup latency with packet splitting.

Memory Bandwidth

Each tier has a peak bandwidth that bounds achievable throughput, and the actual rate is limited by the slowest component on the streaming path.

Each DMA Engine moves up to 256 B/cycle on its own. HBM bandwidth is shared across all engines transferring HBM data, so an aggregate saturating HBM is bounded by the 1.5 TB/s HBM peak rather than by any per-engine sum.

Same-cluster DM-to-DM transfers serialize their reads and writes, since both phases contend for the same DM bank access. Cross-tier transfers like HBM ↔ DM pipeline the read and write phases.

Note

The Tensor Unit (via Fetch and Commit Engines) is often more efficient than DMA for SRAM-to-SRAM transfers, since DMA may underutilize SRAM slice bandwidth. HBM bandwidth, however, is typically the bottleneck in practice, making this gap less critical for HBM ↔ DM transfers.

Channel and DMN Interleaving

Sustaining peak bandwidth requires interleaving access patterns across the underlying memory partitions.

HBM channel selection uses address bits 9 to 28, and address bit 8 is the stack bit. Access patterns must toggle all of these bits to spread requests across all 32 channels. Missing the stack bit (address bit 8) alone halves effective bandwidth by routing all requests to only 16 of the 32 channels. Access patterns that hit the same HBM bank repeatedly (toggling row-address bits 21+ on consecutive accesses) trigger row-conflict penalties of approximately 40 cycles per access, degrading bandwidth by an order of magnitude. FR-FCFS memory scheduling recovers some throughput, but the fundamental cost remains severe.

DM bandwidth requires alternating between both DMNs (each 128 B/cycle), so a single-DMN access pattern halves DM bandwidth.

Startup Latency and Packet Splitting

Each DMA command incurs approximately 500 cycles of fixed startup latency before data transfer begins. Combining multiple transfers into fewer commands amortizes this cost, while heterogeneous aggregates split into per-DMN commands and pay the latency on each command.

Within a single command, the hardware splits each packet into 256-byte units, so an n-byte packet becomes ceil(n / 256) AXI requests. A 4,095-byte packet therefore costs 16 requests, while a 4,099-byte packet (a prime length, awkwardly placed past the 4,096-byte limit) requires splitting into multiple commands. The innermost-loop stride (stride0) determines packet alignment: when stride0 is 256-byte aligned, the cycle count is ceil(stride0 / 256). When stride0 is not 256-byte aligned, HBM writes additionally pay a Read-Modify-Write penalty for the partial 256-byte block. HBM reads incur only the ceil overhead, and DM operations are largely unaffected by this kind of misalignment.

DMA has the lowest DM bank-access priority, so 64+ consecutive same-bank accesses from the Fetch or Commit Engine can starve it and trigger a NoC timeout. See DM Bank Starvation for details.

Detailed Examples

The examples below give concrete sequencer configurations and cycle estimates for representative transfer patterns. Examples 1 to 3 cover well-tuned single-engine cases for each tier pair. Examples 4 and 5 contrast pathological access patterns that lose 10x or more. Example 6 illustrates heterogeneous segmentation when shapes do not divide evenly across DMNs.

The sequencer configurations use two address-stride symbols:

• slice_stride: The virtual address span of one in-slice DM partition (4 MB).
• DMN_stride: The address span between two DMNs within the same cluster.

Example 1: HBM ↔ HBM Layout Transformation

Arguments:

• axes![A = 8, B = 8, C = 256]
• dtype = i8
• Source: HBM at offset 0, mapping m![A, B, C]
• Destination: HBM at offset 16,384, mapping m![B, A, C]
• Stream: time m![A, B], packet m![C]

Generated sequencers:

read = [
  A -> 8   : 2,048,
  B -> 8   : 256,
  C -> 256 : 1,
] : 256, HBM @ 0

write = [
  A -> 8   : 256,
  B -> 8   : 2,048,
  C -> 256 : 1,
] : 256, HBM @ 16,384

The non-innermost strides (256 and 2,048) toggle HBM address bits 8 (stack) and 11 (channel), spreading every request across distinct HBM channels for parallel execution. A single 256-byte transfer takes 4 cycles per channel at 0.75 GHz, but parallel channel distribution sustains near-peak bandwidth. Total time: approximately 64 read requests + 64 write requests at 1 GHz, plus 500 cycles startup ≈ 628 cycles.

When 4 DMA Engines share HBM ↔ HBM traffic, each gets approximately 0.1875 TB/s out of the 0.75 TB/s read bandwidth. Even with stride0 = 256, no single engine completes one request per cycle under that share.

Example 2: HBM → DM with Full Bandwidth

This cross-tier transfer pipelines reads and writes by interleaving across HBM channels and both DMNs.

Arguments:

• axes![A = 256, B = 256, C = 256]
• dtype = i8
• Source: HBM at chip 0, mapping m![B, A, C]
• Destination: DM at chip 0, cluster 0, slice 0. Slice mapping m![A / 4], element mapping m![A % 4, B, C]
• Stream: time m![B, A % 4, A / 4 % 32, A / 128], packet m![C]

Generated sequencers:

read = [
  B      -> 256 : 65,536,
  A%4    -> 4   : 256,
  A/4%32 -> 32  : 1,024,
  A/128  -> 2   : 32,768,
  C      -> 256 : 1,
] : 256, HBM @ 0

write = [
  B      -> 256 : 256,
  A%4    -> 4   : 65,536,
  A/4%32 -> 32  : slice_stride,
  A/128  -> 2   : DMN_stride,
  C      -> 256 : 1,
] : 256, DM @ 0

On the HBM side, the 32,768-stride on A/128=2 interleaves access across channels, and the hardware command queue keeps all 65,536 requests (256 × 4 × 32 × 2) flowing. On the DM side, slice_stride and DMN_stride interleave consecutive 256-byte writes across the two DMNs, keeping both at one request per cycle. Reads and writes pipeline across tiers, so total time is approximately max(65,536 read cycles, 65,536 write cycles) + 500 startup ≈ 66,036 cycles.

Example 3: DM → DM Within One Cluster

Same-cluster DM-to-DM transfers serialize reads and writes, since both contend for the same DM bank access.

Arguments:

• axes![A = 256, B = 256, C = 256]
• dtype = i8
• Source: DM at chip 0, cluster 0, slice 0, element offset 0. Slice mapping m![A / 4], element mapping m![A % 4, B, C]
• Destination: DM at chip 0, cluster 0, slice 0, element offset 4·256·256. Slice mapping m![A / 4], element mapping m![B, A % 4, C]
• Stream: time m![B, A % 4, A / 4 % 32, A / 128], packet m![C]

Generated sequencers:

read = [
  B      -> 256 : 1,
  A%4    -> 4   : 65,536,
  A/4%32 -> 32  : slice_stride,
  A/128  -> 2   : DMN_stride,
  C      -> 256 : 1,
] : 256, DM @ 0

write = [
  B      -> 256 : 1,024,
  A%4    -> 4   : 256,
  A/4%32 -> 32  : slice_stride,
  A/128  -> 2   : DMN_stride,
  C      -> 256 : 1,
] : 256, DM @ (4·256·256)

DMN and slice interleaving give each phase the full 256 B/cycle, but the two phases serialize. Total time: approximately 131,072 cycles (65,536 reads + 65,536 writes) + 500 startup.

Note

Choose C to be a multiple of 256 when possible. For C = 256n + r with 0 < r < 256, the cycle count grows by a factor of n+1 because each access splits into more requests, even though the total data volume changes only slightly.

Example 4: HBM Bank-Conflict Pathology

DM interleaving is healthy here, but a pathological HBM access pattern costs roughly 10x the well-tuned cycle count.

Arguments:

• 1 chip (8 DMNs)
• axes![A = 64, B = 2,048, C = 1,024]
• dtype = i8
• Source: HBM, with cluster mapping m![B / 1024], slice mapping m![B / 256 % 4, A], element mapping m![B % 256, C]
• Destination: DM, with slice mapping m![A / 4], element mapping m![B, A % 4, C]
• Stream: cluster m![B / 1024], slice m![B / 256 % 4], time m![B % 256, C / 256, A % 32, A / 32], packet m![C % 256]

Generated sequencers per (cluster_i, dmn_j):

read = [
  B%256 -> 256 : 1,024,
  C/256 -> 4   : 256,
  A%32  -> 32  : 2²¹,
  A/32  -> 2   : 2²⁶,
  C%256 -> 256 : 1,
] : 256, HBM @ (i·2²⁰ + j·2¹⁸)

write = [
  B%256 -> 256 : 1,024,
  C/256 -> 4   : 256,
  A%32  -> 32  : slice_stride,
  A/32  -> 2   : DMN_stride,
  C%256 -> 256 : 1,
] : 256, DM @ (cluster_i, dmn_j, 0)

The strides on A%32 and A/32 toggle HBM address bits 21 and 26, which select the row within an HBM bank. Consecutive accesses within each channel therefore close one row and open the next on nearly every request, paying approximately 40 cycles per access. Channel interleaving via C / 256 = 4 (stride 256) does spread requests across all 32 channels, but cannot hide the row-conflict cost within each channel.

Performance breakdown:

• HBM reads: 524,288 total requests across 32 channels = 16,384 per channel × ~40 cycles ≈ 655,360 cycles.
• DM writes: 65,536 requests per DMN at one per cycle, hidden under the read latency.

Total time: approximately 655,360 cycles + 500 startup ≈ 655,860 cycles. FR-FCFS scheduling recovers some throughput, but the order-of-magnitude penalty remains.

Example 5: Missing Stack Bit Pathology

The access pattern fails to interleave across HBM’s stack dimension (address bit 8), routing all traffic to half the channels and halving effective bandwidth.

Arguments:

• 1 chip (8 DMNs)
• axes![A = 8, B = 64, C = 8, D = 512]
• dtype = i8
• Source: HBM, mapping m![A, B, C, D]
• Destination: DM, with cluster mapping m![A / 4], slice mapping m![A % 4, B], element mapping m![C, D % 256]
• Stream: cluster m![A / 4], slice m![A % 4], time m![C, B % 32, B / 32], packet m![D % 256]

Generated sequencers per (cluster_i, dmn_j):

read = [
  C     -> 8   : 512,
  B%32  -> 32  : 4,096,
  B/32  -> 2   : 131,072,
  D%256 -> 256 : 1,
] : 256, HBM @ (i·2²⁰ + j·2¹⁸)

write = [
  C     -> 8   : 256,
  B%32  -> 32  : slice_stride,
  B/32  -> 2   : DMN_stride,
  D%256 -> 256 : 1,
] : 256, DM @ (cluster_i, dmn_j, 0)

The C stride of 512 never toggles HBM address bit 8 (the stack bit), so the eight DMNs concentrate on 16 of the 32 HBM channels.

Performance breakdown:

• HBM reads (bottleneck): 4,096 total requests across 16 channels = 256 per channel × ~5.3 cycles per request at 1 GHz ≈ 1,357 cycles.
• DM writes: 512 requests per DMN, pipelined under the reads.

Total time: approximately 1,357 cycles + 500 startup ≈ 1,857 cycles. Restoring stack-bit interleaving across all 32 channels would halve the HBM cycle count.

Example 6: Heterogeneous DMN Segmentation

When tensor shapes do not divide evenly across DMNs, the compiler segments the boundary DMN’s work into multiple commands, each paying its own startup latency.

Arguments:

• 4 chips
• axes![A = 15, B = 32, C = 256, D = 8]
• dtype = i8
• Source: DM, with (writing A' = A + 1#) chip mapping m![D / 2], cluster mapping m![D % 2], slice mapping m![A' / 4, A' / 2 % 2, B], element mapping m![A' % 2, C]
• Destination: HBM, with chip mapping m![D / 2], element mapping m![D % 2, B, A, C]
• Stream (per-DMN, expressed as StreamFn(chip_i, cluster_j, slice_k)):

StreamFn(chip_i, cluster_j, slice_k) = let A' = A + 1# in
  { chip: m![(D / 2) @ i = 1], cluster: m![(D % 2) @ j = 1],
    slice: m![(A' / 4) @ k = 1],
    time: (k == 0,1,2): m![A' % 2, B, A' / 2 % 2, C]
          (k == 3, exec #0): m![A' % 2, B, A' / 2 = 1, C]
          (k == 3, exec #1): m![A' = 1, B, A' / 2 % 2 @ 1, C],
    packet: m![C] }

The dimension A = 15 does not divide across 4 DMNs (15 = 3·4 + 3), so DMNs 0 to 2 each handle 4 elements while DMN 3 handles only 3. A single descriptor on DMN 3 would write a fourth element past the valid region, so the compiler segments DMN 3’s work into two commands that together cover exactly 3 elements.

Performance breakdown:

• DMNs 0 to 2 (one command each): ~256 cycles + 500 startup ≈ 756 cycles.
• DMN 3 (two commands): ~192 data cycles + 1,000 startup (500 each) ≈ 1,192 cycles.

Total time: approximately 1,192 cycles, gated by DMN 3. Choose tensor shapes that divide evenly across DMNs to avoid this segmentation cost.

Shuffle Operations

Shuffle operations redistribute a tensor across clusters or chips according to a per-partition source pattern. The methods chain off the source tensor, matching the to_dm / to_hbm convention: dm_cluster_shuffle and dm_chip_shuffle live on DmTensorView, while hbm_cluster_shuffle and hbm_chip_shuffle live on HbmTensor. The shuffle pattern specifies, for each destination cluster or chip, which source cluster or chip provides its data.

#![allow(unused)]
fn main() {
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 256, B = 4096];

fn cluster_shuffle(
    ctx: &mut Context,
    input: &DmTensor<i32, m![A / 4 % 4], m![A / 2 % 2], m![B % 16, B / 16 % 16], m![B / 256, A % 2, A / 16]>,
) -> DmTensor<i32, m![A / 4 % 4], m![A / 2 % 2], m![B % 16, B / 16 % 16], m![B / 256, A % 2, A / 16]> {
    // Shuffle pattern [1, 0]: cluster 0 ↔ cluster 1
    input.view().dm_cluster_shuffle::<2>(&mut ctx.tdma, &[1, 0])
}
let mut ctx = Context::acquire();
let input_dm = unsafe { DmTensor::<i32, m![A / 4 % 4], m![A / 2 % 2], m![B % 16, B / 16 % 16], m![B / 256, A % 2, A / 16]>::from_addr(0) };
let _output_dm = cluster_shuffle(&mut ctx, &input_dm);
}

Inter-chip shuffles use the system-wide global chip IDs. hbm_chip_shuffle is generic over the DMA context (tdma or pdma) because the cross-chip operation is HBM ↔ HBM, and HBM ↔ HBM is the one DMA pair that both Tensor DMA and PCIe DMA support. The other shuffle methods, and DMA pairs like HBM ↔ DM and DM ↔ DM in general, are not context-generic.

Scatter and Gather

Scatter and gather move tensor elements at addresses computed from an index tensor rather than at fixed strides.

DmTensor::dma_scatter writes DM values to HBM rows chosen by an index tensor. HbmTensor::dma_gather_scaled and HbmTensor::dma_gather_unscaled read HBM rows into DM at rows chosen by an index tensor. The two gather variants differ only in where the index lives and how its values are read, described below.

extern crate furiosa_opt_std;
extern crate tokio;
use furiosa_opt_std::prelude::*;
axes![K = 512, D = 128, C = 612, G = 512, CL = 2];

fn scatter_minimal(
    ctx: &mut Context,
    data: &HbmTensor<bf16, m![1], m![K, D]>,
    index: &HbmTensor<i32, m![1], m![K]>,
    output: &mut HbmTensor<bf16, m![1], m![C, D]>,
) {
    let data_dm: DmTensor<bf16, m![1], m![1 # 2], m![K / 2], m![K % 2, D]> =
        data.to_dm(&mut ctx.tdma);

    data_dm.dma_scatter::<m![K], _, _>(index, output);
}

fn gather_minimal(
    table: &HbmTensor<bf16, m![1], m![K, D]>,
    index: &HbmTensor<i32, m![1], m![G]>,
    // The gather axis itself partitions into Slice x Element (`G / 2 = 256`, a valid slice count)
    // with `D` folded into the Element side alongside the `G % 2` remainder; `C = 612` (used above
    // for the scatter cache) has no divisor landing on a valid 64 | 128 | 256 slice count, so the
    // gather count here is the separate, slice-friendly `G` instead.
) -> DmTensor<bf16, m![1], m![1 # 2], m![G / 2], m![G % 2, D]> {
    table.dma_gather_scaled(index)
}

fn gather_unscaled(
    ctx: &mut Context,
    table: &HbmTensor<bf16, m![1], m![K, D]>,
    // Raw row positions per cluster. The gather reads the index off SPM, so the kernel first
    // stages it on-chip with `to_dm`; a real per-cluster (`CL`) partition avoids broadcast padding.
    index: &HbmTensor<i32, m![1], m![CL, G]>,
) -> DmTensor<bf16, m![1], m![CL], m![G / 2], m![G % 2, D]> {
    let index_dm: DmTensor<i32, m![1], m![CL], m![G / 2], m![G % 2]> =
        index.to_dm(&mut ctx.tdma);
    table.dma_gather_unscaled(&index_dm)
}

#[tokio::main]
async fn main() {
    let mut ctx = Context::acquire();

    let index = &(HostTensor::<i32, m![K]>::zero().to_hbm(&mut ctx.pdma).await);
    let data = unsafe { HbmTensor::<bf16, m![1], m![K, D]>::from_addr(0) };
    let mut output_hbm = unsafe { HbmTensor::<bf16, m![1], m![C, D]>::from_addr(0) };

    scatter_minimal(&mut ctx, &data, &index, &mut output_hbm);
    gather_minimal(&data, &(HostTensor::<i32, m![G]>::zero().to_hbm(&mut ctx.pdma).await));
    let placed_index = &(HostTensor::<i32, m![CL, G]>::zero().to_hbm(&mut ctx.pdma).await);
    gather_unscaled(&mut ctx, &data, placed_index);
}

The scaled variants (dma_gather_scaled, dma_scatter) take the index from an HbmTensor in DRAM and read its values as byte offsets along the gather/scatter axis: to address row r, pass r times one row’s byte size (its element count times the element’s byte size; e.g. 128 * 2 = 256 for a 128-wide bf16 row). dma_gather_unscaled instead takes an on-chip DmTensor index, staged from DRAM with to_dm, and reads its values as raw row positions, for indices computed on-chip such as paged-attention block tables. Note a side effect: the seed reads the index off SPM, so the compiler emits an extra DM to SPM DMA to stage the index (SPM is the on-chip tier noted above, not yet a user-facing type). Its scatter counterpart dma_scatter_unscaled is not yet implemented.

PCIe DMA

PCIe DMA (Context::pdma) moves tensors between host system memory and device HBM. It is a separate physical engine from the on-chip Tensor DMA. PCIe DMA handles only host ↔ HBM, while Tensor DMA handles all on-chip transfers.

The kernel writer calls .to_hbm() on a HostTensor (host → device) or .to_host() on an HbmTensor (device → host). Both are async operations.

extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
use rand::{rngs::SmallRng, SeedableRng};
axes![A = 8, B = 512];

async fn upload_and_download(ctx: &mut Context) {
    let mut rng = SmallRng::seed_from_u64(0);
    let host: HostTensor<i8, m![A, B]> = HostTensor::rand(&mut rng);

    // Host → HBM (allocator-assigned address)
    let hbm: HbmTensor<i8, m![A], m![B]> = host.to_hbm(&mut ctx.pdma).await;

    // HBM → host (back to system memory)
    let _round_tripped: HostTensor<i8, m![A, B]> = hbm.to_host(&mut ctx.pdma).await;
}

HostTensor carries only an Element mapping (host memory has no chip/cluster/slice partitioning), while the destination HbmTensor adds the Chip axis to distribute across chips. The destination element layout in HBM may differ from the host layout, since to_hbm accepts new Chip and Element type parameters.

PCIe DMA bandwidth is 30 B/cycle, an order of magnitude slower than on-chip Tensor DMA (256 B/cycle). Algorithms should minimize host ↔ device traffic, uploading data once and reusing it across many on-chip operations.

Memory Performance

Memory performance is critical to kernel throughput. The Fetch, Commit, and DMA Engines each expose API choices (such as Packet size and access ordering) that map directly to performance outcomes. This page documents the hardware specifications and constraint rationale that connect those choices to measured throughput.

Each memory type has a peak bandwidth per chip:

Reaching these peaks requires specific access patterns. The following table lists rules whose violation degrades throughput, and the sections below explain each memory type and factor in detail:

Data Memory (DM)

Data Memory (DM) holds 256MB per chip, organized hierarchically into clusters, Data Memory Networks (DMNs), slices, and banks. The following table summarizes the geometry:

Clusters can exchange data through the Switch Engine. See the dedicated section for details. The subsections below explain how this structure determines bandwidth and the bank access constraint.

Bank Structure in a Slice

Each slice provides 512KB of SRAM with a dedicated address space. The memory is organized into 16 parallel banks, each with an 8-byte data width, enabling a total data access rate of 128 B/cycle. Access to any individual bank is serialized, but the address space distributes 128 consecutive bytes across all 16 banks (8 bytes per bank) for parallel access. The following bit mapping defines this distribution:

Consecutive addresses map to different banks, enabling parallel access during sequential scans.

DMN and Slice Interleaving

Each DMN provides only 128 B/cycle bandwidth (its 32 slices share data paths). Since the standard 256-byte transfer unit requires two cycles per DMN, pipeline accesses across both DMNs to maintain continuous throughput:

Note

While command queues theoretically allow some burst access without interleaving, always interleave across DMNs when generating DMA streams, as this is the most natural approach.

Slices are shared by the DMA, Fetch, and Commit Engines, so spreading requests across the 32 slices within each DMN reduces contention. Data Memory Routers connect those slices in a ring topology within each DMN: slice0_in → slice31_out, slice32_in → slice63_out. Each Data Memory Slice has a 2-entry command queue for pending DMA requests. Distributing requests across M slices reduces required throughput per slice to 1/M, even when priority delays individual slices. DMN interleaving every n cycles achieves saturated 256 B/cycle.

Bank Starvation

Bank starvation occurs when the DMA Engine is indefinitely blocked waiting for a DM bank held by higher-priority engines. The 64-access rule prevents this. Violating this rule causes a Network-on-Chip (NoC) timeout and a full cluster reset, losing all computation state.

Each DM bank is a shared resource. When high-priority engines continuously access it, lower-priority requesters are indefinitely blocked, a form of priority inversion. The DM controller prioritizes requests in this order:

• Main-context Fetch Engine
• Main-context Commit Engine
• Sub-context Fetch Engine
• Sub-context Commit Engine
• DMA Engine

DMA has the lowest priority among all memory engines because computation engines must get first access to data during normal operation. However, this creates a dangerous scenario when high-priority engines continuously access the same bank: the DMA Engine’s request sits in the queue, unable to make progress, while the higher-priority engines monopolize that bank. Tensor DMA communicates with DRAM and DMN through a NoC hub where each port (DMA, DRAM, DMN) must acknowledge requests within 4,096 cycles. After 4,096 cycles without a response, the NoC protocol declares the transaction dead and enters an exception state as a safety mechanism to detect deadlocks and indefinitely hung transactions. When the timeout triggers, the hardware lacks a graceful recovery mechanism. The only recovery is a full cluster domain reset, losing all computation state and requiring complete reinitialization.

The 64-access rule prevents this catastrophe: the Fetch and Commit Engines must not access the same bank for 64 or more consecutive operations while DMA is active. Why 64? The constraint is (TDMA_IO_BYTE / DMN_IO_BYTE) * Max_Consecutive_Access * DMN_SIZE < 4096 (with TDMA_IO_BYTE = 256, DMN_IO_BYTE = 128, DMN_SIZE = 32), which yields Max_Consecutive_Access < 64. This ensures DMA requests complete before the NoC timeout even in the worst case.

For example, suppose the DMA Engine issues a request to bank 0 (along with 15 other banks), but the main-context’s Fetch Engine continuously requests bank 0. The DMA request stalls, and if this exceeds 4,096 cycles, a NoC timeout forces a hardware reset.

• Scheduling model: The scheduler uses context occupancy information: if operation A occupies a context (e.g., main-context), the next operation B using that context waits until A completes. Understanding which contexts operations occupy enables predicting parallel execution.
• Compiler scheduling behavior: When Tensor Unit operations would violate the 64-access limit, the compiler schedules them as if they occupy DMA, preventing concurrent DMA operations. This sacrifices the TCP architecture’s inherent main/sub/DMA context parallelism where data preparation and computation occur in parallel, but avoids catastrophic hardware resets. Treat this as a hard constraint: never use patterns with 64+ consecutive same-bank accesses.
• The 64-access limit details:
  • The limit is cumulative: total accesses from all engines to the same bank must stay below 64, since even interleaved accesses across commands accumulate toward this total. For example, main: 30, sub: 20, DMA: 1 totals 51 (safe), but main: 30, sub: 35, DMA: 1 totals 66 (triggers starvation).
  • The compiler keeps each individual command below 64 consecutive same-bank accesses, but cannot prevent the total from reaching 64 when multiple commands run concurrently.
  • In practice, sub-context rarely accesses the same bank consecutively (StoTrf, StoVrf operations typically use sequential addresses and tiling prevents same-bank access).
  • Sub-context operations that would exceed the limit are also not scheduled concurrently with DMA.
• Main/sub-context contention: Main-context can starve sub-context, but this is less severe:
  • Unlike DMA starvation, sub-context starvation does not cause NoC timeout or hardware reset and only increases processing time.
  • Collision probability is lower: DMA Engine occupies 16 banks at once, while sub fetch/commit engines occupy only one bank.
  • Starvation does not occur between fetch and commit engines within the same context due to pipeline back-pressure.
  • Performance impact example: If main-context exec command continuously accesses a specific bank while sub-context stos command is scheduled, sub-context processing is delayed. Worst case: total time = main-context time + sub-context time. Ideal case: main and sub access different banks, achieving total time = max(main-context time, sub-context time).

See Schedule Viewer for a scheduling visualization utility that shows which operations run in parallel and verifies actual context assignments.

Scratchpad Memory (SPM)

DM and SPM are both on-chip SRAM. They are distinguished by intended use (and corresponding compiler allocation policy) rather than by any documented latency or capacity difference.

DM (Data Memory) is the main working memory for tensor data flowing through the Tensor Unit pipeline. The DMA Engine populates DM from HBM (or other tiers), the Fetch Engine streams data from DM into the Tensor Unit, and the Commit Engine writes the pipeline’s results back to DM. DM allocation follows general-purpose policies driven by the program’s tensor lifetimes.

SPM (Scratchpad Memory) is a compiler-managed staging tier. The compiler explicitly chooses what lives in SPM, reserving it for small, frequently reused values that should not have to be refetched from DM on every access:

• scalar constants and configuration data,
• activation function lookup tables,
• small per-DMN working sets that are read many times.

SPM is most useful for per-DMN state that would otherwise force repeated DM reads through the Fetch or DMA Engines.

DM and SPM are both distinct from the per-slice register files that feed the Tensor Unit’s compute engines directly (see Computing Tensors):

• TRF (Tensor Register File) is a per-slice register file populated via .to_trf(); the Contraction Engine reads it each cycle.
• VRF (Vector Register File) is a per-slice register file populated via .to_vrf(); the Vector Engine reads it each cycle.

The pipeline’s data flow is therefore: DMA populates DM from HBM → Fetch streams data from DM into the Tensor Unit → Collect writes the stream into TRF / VRF → Contraction / Vector read directly from TRF / VRF → Commit writes results back into DM. SPM sits to the side as a compiler-controlled staging area for the small, high-locality data the kernel needs but does not want to refetch.

Each DMN contains SPM with a bandwidth of 128 B/cycle, and because each DMN has dedicated SPM there are no inter-DMN contention issues.

High-Bandwidth Memory (HBM)

HBM holds 48GB per chip and delivers 1.5 TB/s aggregate bandwidth, but reaching that peak requires 256-byte-aligned access and channel interleaving across all 32 channels. Misaligned writes and bank conflicts can degrade throughput by 30–50×. The following table summarizes the HBM geometry:

Peak Bandwidth

Saturating a single DMA Engine (256GB/s capacity) requires interleaving accesses across multiple channels. Peak HBM bandwidth reaches 1.5TB/s per chip through parallel operation of stacks and channels. The channel controller transfers 64B/cycle at 0.75GHz,¹ yielding 48GB/s per channel (0.75GHz x 64B/cycle) or 1.5TB/s per chip (48GB/s x 32 channels). The fundamental transfer unit is 256 bytes, requiring 4 clock cycles per channel.

Peak bandwidth is sensitive to access patterns. Misalignment, bank conflicts, and resource sharing can each severely degrade throughput. Each channel controller has a 64-entry command queue that interleaves accesses to minimize penalties, but pathological cases can still cause severe degradation. The following sections describe causes of performance degradation and how to avoid them.

Address Space in a Chip

The HBM address space uses a non-linear bit mapping optimized for parallel sequential access. This design maximizes parallelism and minimizes overhead:

The bit assignment for each component corresponds to the physical memory geometry. For instance, the byte component occupies 11 bits (bits 0-7, 14-16) to represent 2K (2^11) bytes per row. Three exceptions exist:

• Slice representation: Two bits (20 and 34) represent slice, even though there are only three slices.
• Contiguous address space: Bit 34 is influenced by the row component to ensure bits 34 and 35 are never both 1, guaranteeing a contiguous 48GB address space.
• Channel XOR mapping: The channel component equals the XOR of bits 9-12 and 13-28 (e.g., the channel’s first bit equals the XOR of bits 9, 13, 21, and 25).

This bit ordering ensures that sequential accesses are spread across stacks, channels, bank groups, and banks simultaneously, keeping multiple memory resources busy in parallel.

Misaligned Access

Misaligned access degrades HBM performance substantially. Reads crossing a 256-byte boundary require two transfers (2x penalty), and unaligned writes require a Read-Modify-Write (RMW) operation (roughly 50x penalty). The 256-byte minimum access unit is defined by bits 0-7 (the eight LSBs), so data that crosses this boundary incurs these penalties.

• Unaligned Read: Read requests crossing a 256-byte boundary require two NoC transfers, effectively halving bandwidth.
• Unaligned or Partial Write: An unaligned write arises because DMA packets are internally segmented into 256-byte transactions. When a packet’s size is not 256-byte aligned (e.g., a 2,800-byte packet splits into ten 256-byte requests plus one 240-byte request), the final “leftover” transaction requires an RMW operation. RMW reads the entire 256-byte unit, updates the requested bytes, then writes the entire unit back. RMW can slow writes by roughly 50× compared to aligned writes.

Bank Conflict

HBM banks hold one open row at a time. Switching to a different row within the same bank requires closing the current row and opening the new one. This adds 40-50 ns (60-75 cycles at 1.5 GHz) of latency, which is 30-40x slower than accessing an already-open row. This penalty occurs whenever consecutive accesses target different rows within the same bank. All rows start closed, so the first access to any row always pays the open-row cost.

Channel interleaving mitigates bank conflicts. Interleaving accesses across all 32 channels distributes load and reduces conflicts. Bits 8-12 (the next five LSBs) represent independent stacks and channels. Placing these at low addresses prevents interference between adjacent accesses, which is vital for parallelizing contiguous operations. Non-contiguous operations often benefit from natural channel interleaving because the channel component spans bits 9-28. However, the stack component corresponds only to bit 8, so the programmer must explicitly ensure accesses alternate between the two stacks to achieve full stack interleaving.

The controller hides row-switch latency through command interleaving. Within each channel, the controller automatically interleaves commands across banks, enabling useful transfers while other banks perform row switches. The controller manages bank states using its command queue. It employs FR-FCFS (First Ready-First Come First Served) scheduling, prioritizing commands targeting already-open rows.

Despite this sophisticated scheduling, access patterns that continuously switch rows within the same bank still degrade performance significantly. Compilers and programmers should estimate row-switch costs when generating code.

Column-to-Column Delay

tCCD (Column-to-Column Delay) is the minimum time between consecutive read or write commands on the same channel, which determines the maximum command issue rate. In most access patterns, bank conflicts or channel interleaving dominate before tCCD becomes the bottleneck. Vendor specifications set tCCD values based on analog constraints for accessing DRAM stack layers and shared resources.

The tCCD value depends on which memory resources consecutive commands target:

The optimal case is interleaving between different bank groups within the same slice (tCCD = 2 cycles at 1.5GHz), allowing a new 64B command to be issued every cycle at 0.75GHz, achieving back-to-back transmission and full channel speed. Any tCCD greater than 2 reduces the command rate and channel utilization.

Compared to bank conflicts, tCCD degradation is less severe because the worst-case patterns either coincide with bank conflicts (making tCCD the secondary effect) or are masked by channel interleaving:

• Different Slice (tCCD = 3): Slice ID corresponds to bit 20, and bit 21 corresponds to the row. Interleaving across slices therefore likely causes bank conflicts simultaneously.
• Same Slice, Same Bank Group (tCCD = 4): This pattern interleaves bits 8-35 except bits 13, 19, 20, and 34. Bits 29-35 relate to bank conflicts. Bits 8-28 relate to channel interleaving.

1. Although the channel controller operates at a frequency of 0.75GHz, it performs eight bursts per cycle, leading to an effective frequency of 0.75×8=6GHz. ↩

Computing Tensors

Tensor Unit

The Tensor Unit is the on-chip compute pipeline. It reads tensor data from DM, transforms it through ten engines, and writes results back to DM.

Each tensor flows through the pipeline as a stream of packets, one packet per cycle. The engines consume and produce these streams, reshaping the per-cycle layout and the iteration order along the way. The Collect Engine normalizes incoming packets to 32-byte flits. Every downstream engine (Contraction, Vector, Cast, Transpose, Commit Adapter, Commit) operates on these flits.

flowchart TB
    subgraph SRAM
        DM[(DM)] & TRF[(TRF)] & VRF[(VRF)]
    end

    subgraph TU[Tensor Unit]
        direction LR
        FE[Fetch] --> FA[Fetch Adapter] --> SW[Switching] --> CO[Collect] --> CE[Contraction] --> VE[Vector] --> CA[Cast] --> TR[Transpose] --> CMA[Commit Adapter] --> CM[Commit]
    end

    DM --> FE
    CM --> DM
    CO --> TRF --> CE
    CO --> VRF --> VE

    click FE "../moving-tensors/fetch-engine.html" "Fetch Engine"
    click FA "./fetch-adapter.html" "Fetch Adapter"
    click SW "./switch-engine.html" "Switch Engine"
    click CO "./collect-engine.html" "Collect Engine"
    click CE "./contraction-engine/index.html" "Contraction Engine"
    click VE "./vector-engine/index.html" "Vector Engine"
    click CA "./cast-engine.html" "Cast Engine"
    click TR "./transpose-engine.html" "Transpose Engine"
    click CMA "./commit-adapter.html" "Commit Adapter"
    click CM "../moving-tensors/commit-engine.html" "Commit Engine"

Each tensor stream inside the Tensor Unit carries five dimensions, [Chip, Cluster, Slice, Time, Packet], that split into two groups. Chip, Cluster, and Slice are spatial dimensions: each slice runs its own pipeline instance, with slices grouped by cluster and clusters grouped by chip. Time and Packet describe the per-slice stream (see Spatial and Temporal Dimensions for the definitions). The engines above reshape Time / Packet along the pipeline. The spatial dimensions are preserved by every engine except two: Switch changes Slice by moving data across slices, and Vector’s inter-slice reducer collapses Slice by aggregating across the 256 slices in a cluster.

The Contraction and Vector Engines each take one operand from the pipeline stream and the other operand from a dedicated per-slice register file. TRF (Tensor Register File) feeds the Contraction Engine, and VRF (Vector Register File) feeds the Vector Engine. The Collect Engine writes into TRF via .to_trf() and into VRF via .to_vrf(). For an end-to-end example using both files, see Quick Start.

Fetch reads from DM and Commit writes back to DM. Their detailed sequencer behavior is documented in Moving Tensors rather than here.

Execution Context

The scheduler treats each execution context as an independent stream of operations. The hardware exposes three:

• Main drives the Tensor Unit pipeline for the kernel’s primary computation.
• Sub drives a subset of the same pipeline, typically prefetching operands into TRF / VRF while main computes.
• DMA drives the DMA Engine alone, external to the Tensor Unit (HBM ↔ DM, HBM ↔ SPM, DM ↔ SPM).

The main context can drive every Tensor Unit engine. The sub context drops the Contraction Engine and a handful of other features; everything else carries over from main.

Operations serialize within a single context but run in parallel across different contexts. For example, sub prefetches the next operand batch into TRF / VRF while main computes the current one (double-buffering), and the DMA Engine moves bulk data between HBM and DM / SPM independently of either Tensor Unit context (overlap).

Some Tensor Unit engines form a single unit of scheduling that can be driven by only one context at a time. For example, the Vector Engine and the Cast Engine form one such unit of scheduling. So when sub is running Vector Engine work, main runs its type casting through the Commit Adapter’s Type Casting stage instead of the Cast Engine, to avoid serializing with sub.

The scheduler may also assign a Tensor Unit operation to the DMA context defensively when its DM access pattern would otherwise risk a hardware-level memory conflict (see Memory Performance for the rules that trigger this).

Fetch Adapter

The Fetch Adapter applies element-wise transformations (type casting, masking, table lookup, zero-point subtraction) to the packet stream emitted by the Fetch Engine, before the Switch Engine routes it across slices. The Fetch Engine itself does not run any of these transforms; they live here under Computing Tensors and are applied as separate stages between the Fetch Engine and the Switch Engine. The kernel writer composes the per-stage methods directly on a FetchTensor, and each call advances to the next stage.

The adapter has four stages, each optional and invoked by calling its method on the stream in hardware pipeline order. A FetchTensor may flow directly into the Switch Engine or the Collect Engine with no adapter call at all.

• Masking zeros the padded slots of the sequencer’s right pad.
• Table Lookup replaces values via a hardware lookup table.
• Type Casting converts the element type.
• Zero-Point Subtraction subtracts a quantization zero point, widening an integer stream to the Contraction Engine’s staging type (i4 to i5, i8 to i9).

The main-context adapter supports all four stages, while the sub-context adapter supports only fetch_cast.

The example below pads 63 elements to 64, masks the 64th slot to zero, then casts i8 → i32. The two stages are chained as two method calls on the FetchTensor produced by fetch(), in the mask → cast order matching the hardware pipeline.

#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 63];

fn fetch_mask_then_cast<'l, const T: Tu>(
    input: BeginTensor<'l, T, i8, m![1], m![1], m![1], m![1], m![A]>,
) -> FetchCastTensor<'l, T, i32, m![1], m![1], m![1], m![1], m![A # 64]> {
    input
        .fetch::<m![1], m![A # 64]>()
        //   Time      = m![1]
        //   Packet    = m![A # 64]
        //   OutTime   = m![1]
        //   OutPacket = m![A # 64]    (#{0} cannot yet ride the type, see UC below)
        .fetch_mask::<m![1], m![A # 64]>()
        .fetch_cast::<i32>()
}

let mut ctx = Context::acquire();
let x: BeginTensor<'_, _, i8, m![1], m![1], m![1], m![1], m![A]> = BeginTensor::new(&mut ctx.main, Tensor::zero());
let _o = fetch_mask_then_cast(x);

Masking

The Tensor Unit’s internal data paths operate on fixed-width units (32-byte flits of 8 elements at 32 bits each), so axes whose sizes do not align must be padded (for example, 63 rounds up to 64). The padded slots in m![A # 64] hold arbitrary values, which corrupts downstream computations. Masking overwrites those slots with zero, tightening the mapping from A # 64 to A #{0} 64.

impl<'l, const T: Tu, P: CanApplyFetchMask, D: Scalar, Chip: M, Cluster: M, Slice: M, Time: M, Packet: M, B: Backend>
    TuTensor<'l, T, P, D, Chip, Cluster, Slice, Time, Packet, B>
{
    /// Runs the Fetch Adapter's masking stage.
    ///
    /// Zeroes the padded slots described by the book chapter. `OutTime`
    /// and `OutPacket` carry the pad-kind change at the type level (e.g.
    /// `m![D # n]` → `m![D #{0} n]`). Callers spell them out explicitly
    /// because downstream methods do not constrain their input shape. This
    /// method takes no runtime argument (see [`FetchMaskConfig`]).
    #[primitive(TuTensor::fetch_mask)]
    pub fn fetch_mask<OutTime: M, OutPacket: M>(
        self,
    ) -> FetchMaskTensor<'l, T, D, Chip, Cluster, Slice, OutTime, OutPacket, B> {
        verify_fetch_mask::<Time, Packet, OutTime, OutPacket>();
        FetchMaskTensor::new(self.ctx, self.inner.transpose(true))
    }
}

The padded # m annotations on the input become #{0} m on the output. fetch_mask takes no runtime argument. The compiler derives the mask config (FetchMaskConfig, holding last_axis, valid_count_dim, and rightmost_valid_count) from the difference between the input and output mappings.

The three cases below walk through the configuration the compiler will eventually emit for each mapping shape, using the same B-axis pattern under three different sequencer layouts.

Contiguous right mask

This case shows the basic shape: one contiguous right pad sits along the innermost axis, and the DM tensor’s B # 96 carries 4 trailing slots of arbitrary values.

#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 32, B = 92];

fn fetch_mask_contiguous_right<'l, const T: Tu>(
    input: BeginTensor<'l, T, i8, m![1], m![1], m![1], m![1], m![A, B # 96]>,
) -> FetchMaskTensor<'l, T, i8, m![1], m![1], m![1], m![A, B #{0} 96 / 32], m![B #{0} 96 % 32]> {
    input
        .fetch::<m![A, B # 96 / 32], m![B # 96 % 32]>()
        //   Time      = m![A, B # 96 / 32]
        //   Packet    = m![B # 96 % 32]
        //   OutTime   = m![A, B #{0} 96 / 32]   (chunk axis tightens to #{0})
        //   OutPacket = m![B #{0} 96 % 32]      (packet axis inherits #{0})
        .fetch_mask::<m![A, B #{0} 96 / 32], m![B #{0} 96 % 32]>()
}

let mut ctx = Context::acquire();
let x: BeginTensor<'_, _, i8, m![1], m![1], m![1], m![1], m![A, B # 96]> = BeginTensor::new(&mut ctx.main, Tensor::zero());
let _o = fetch_mask_contiguous_right(x);

The diagram below shows the intent: the chunk axis B # 96 / 32 carries the pad, and the mask zeroes its last 4 slots.

From the input and output mappings the compiler emits these sequencer loops (innermost first):

[
    B # 96 % 32 = 32 : 1,    // loop 0: packet axis
    B # 96 / 32 = 3 : 32,    // loop 1: chunk axis (carries the pad)
    A = 32 : 96,             // loop 2: outer
] : 32 @ base_addr = 0

It then programs the hardware mask with these parameters:

• last_axis = the chunk axis B # 96 / 32 (recorded as an axis, not a raw loop index, since the sequencer optimizer may merge entries).
• valid_count_dim = Rightmost (only the rightmost cell of last_axis is masked).
• rightmost_valid_count = [4, 0, 0, 0, 0, 0, 0, 0] (4 slots zeroed at the tail of the rightmost iteration).

At run time the hardware streams every packet untouched until the chunk axis hits its last iteration, where it zeroes the trailing 4 slots of that final packet. The output type therefore tightens from B # 96 to B #{0} 96.

Split right mask

This case keeps the same right pad as the previous case, but a non-padded axis splits the padded region across the stream.

#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 32, B = 92];

fn fetch_mask_split_right<'l, const T: Tu>(
    input: BeginTensor<'l, T, i8, m![1], m![1], m![1], m![1], m![A, B # 96]>,
) -> FetchMaskTensor<'l, T, i8, m![1], m![1], m![1], m![B #{0} 96 / 32, A], m![B #{0} 96 % 32]> {
    input
        .fetch::<m![B # 96 / 32, A], m![B # 96 % 32]>()
        //   Time      = m![B # 96 / 32, A]
        //   Packet    = m![B # 96 % 32]
        //   OutTime   = m![B #{0} 96 / 32, A]
        //   OutPacket = m![B #{0} 96 % 32]   (innermost packet axis tightens)
        .fetch_mask::<m![B #{0} 96 / 32, A], m![B #{0} 96 % 32]>()
}

let mut ctx = Context::acquire();
let x: BeginTensor<'_, _, i8, m![1], m![1], m![1], m![1], m![A, B # 96]> = BeginTensor::new(&mut ctx.main, Tensor::zero());
let _o = fetch_mask_split_right(x);

The diagram below shows the intent: A now sits between B # 96 / 32 and B # 96 % 32, so the pad lives on the innermost packet axis instead of an outer chunk axis.

The compiler emits these sequencer loops:

[
    B # 96 % 32 = 32 : 1,    // loop 0: packet axis (carries the pad)
    A = 32 : 96,              // loop 1: non-padded axis splits the chunks
    B # 96 / 32 = 3 : 32,    // loop 2: outer chunk axis
] : 32 @ base_addr = 0

It then programs the hardware mask with these parameters:

• last_axis = the innermost packet axis (the only axis carrying the pad in this layout).
• valid_count_dim = Rightmost (a single cell at the tail of the outermost iteration covers the pad).
• rightmost_valid_count = [4, 0, 0, 0, 0, 0, 0, 0] (4 slots zeroed per affected packet).

While the outermost B # 96 / 32 loop runs its final iteration, the hardware zeroes the last 4 elements of every packet it emits. The input keeps B # 96 and the masked return type is the same B #{0} 96 as the previous case, with only the Time-axis order differing.

Variable per-chunk mask

This case lifts the per-cell cap (255 for 4-bit data, 31 for f32) by giving each chunk its own valid count. The DM tensor’s B # 128 carries 31 trailing pad slots split across 8 chunks of 16, so the cap does not suffice on its own.

#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 32, B = 97];

fn fetch_mask_variable_per_chunk<'l, const T: Tu>(
    input: BeginTensor<'l, T, f32, m![1], m![1], m![1], m![1], m![A, B # 128]>,
) -> FetchMaskTensor<'l, T, f32, m![1], m![1], m![1], m![A, B #{0} 128 / 16, 1], m![B #{0} 128 % 16]> {
    input
        .fetch::<m![A, B # 128 / 16, 1], m![B # 128 % 16]>()
        //   Time      = m![A, B # 128 / 16, 1]
        //   Packet    = m![B # 128 % 16]
        //   OutTime   = m![A, B #{0} 128 / 16, 1]
        //   OutPacket = m![B #{0} 128 % 16]
        .fetch_mask::<m![A, B #{0} 128 / 16, 1], m![B #{0} 128 % 16]>()
}

let mut ctx = Context::acquire();
let x: BeginTensor<'_, _, f32, m![1], m![1], m![1], m![1], m![A, B # 128]> = BeginTensor::new(&mut ctx.main, Tensor::zero());
let _o = fetch_mask_variable_per_chunk(x);

The diagram below shows the intent: 8 chunks each get their own valid count, so the per-chunk cap no longer constrains the pad.

The compiler emits these sequencer loops:

[
    B # 128 % 16 = 16 : 1,    // loop 0: packet axis
    1 = 1 : 0,                 // loop 1: unit loop (placeholder)
    B # 128 / 16 = 8 : 16,    // loop 2: chunk axis (one count per iteration)
    A = 32 : 128,              // loop 3: outer
] : 16 @ base_addr = 0

It then programs the hardware mask with these parameters:

• last_axis = the B # 128 / 16 chunk axis.
• valid_count_dim = Iterator(2) (the count array indexes along the same loop, so entry i covers chunk i; the only variant that consumes more than one array entry).
• rightmost_valid_count = [16, 16, 16, 16, 16, 16, 1, 0] (counts sum to 97, the valid B length: chunks 0-5 fully live, chunk 6 keeps only its first element, chunk 7 is fully zeroed).

The hardware applies one cell’s count per chunk iteration, in lockstep with the loop. The output type’s B #{0} 128 records the remaining 31 trailing slots as zero.

Left Padding

Left padding zeroes the first few elements when last_axis is the innermost axis. The sequencer’s base_addr is shifted by -left_pad so reads start in pre-data memory, which the mask then overwrites. This is the only place where masking depends on the sequencer configuration.

Contiguous left+right mask

This case combines one contiguous left pad with one contiguous right pad along the innermost axis. Input (# 2 + B) # 96 has 2 arbitrary head slots and 4 arbitrary tail slots.

axes![A = 32, B = 90];

fn fetch_mask_contiguous_left_right<'l, const T: Tu>(
    input: BeginTensor<'l, T, i8, m![1], m![1], m![1], m![1], m![A, (# 2 + B) # 96]>,
) -> FetchMaskTensor<'l, T, i8, m![1], m![1], m![1], m![A, (#{0} 2 + B) #{0} 96 / 32], m![(#{0} 2 + B) #{0} 96 % 32]> {
    input
        .fetch::<m![A, (# 2 + B) # 96 / 32], m![(# 2 + B) # 96 % 32]>()
        //   Time      = m![A, (# 2 + B) # 96 / 32]
        //   Packet    = m![(# 2 + B) # 96 % 32]
        //   OutTime   = m![A, (#{0} 2 + B) #{0} 96 / 32]
        //   OutPacket = m![(#{0} 2 + B) #{0} 96 % 32]
        .fetch_mask::<m![A, (#{0} 2 + B) #{0} 96 / 32], m![(#{0} 2 + B) #{0} 96 % 32]>()
}

The diagram below shows the intent.

The sequencer shape matches the contiguous-right case, but the base address shifts:

[
    (# 2 + B) # 96 % 32 = 32 : 1,    // loop 0: packet axis
    (# 2 + B) # 96 / 32 = 3 : 32,    // loop 1: chunk axis (carries the right pad)
    A = 32 : 96,                       // loop 2: outer
] : 32 @ base_addr = -2

It then programs the hardware mask with these parameters:

• last_axis = the chunk axis carrying the right pad (same as the contiguous-right case).
• valid_count_dim = Rightmost (one cell at the tail of the rightmost iteration covers the right pad).
• rightmost_valid_count = [4, 0, 0, 0, 0, 0, 0, 0] (4 slots zeroed at the tail of the rightmost iteration).
• left_pad = 2 (planned; zero the first 2 slots of every packet).

base_addr = -2 reads 2 bytes earlier so the head pad enters the stream, and left_pad = 2 overwrites those leading slots before they reach the consumer. The return type tightens to (#{0} 2 + B) #{0} 96.

Split left+right mask

This case combines a left pad and a right pad whose regions are split across the stream by a non-padded axis.

axes![A = 32, B = 90];

fn fetch_mask_split_left_right<'l, const T: Tu>(
    input: BeginTensor<'l, T, i8, m![1], m![1], m![1], m![1], m![A, (# 2 + B) # 96]>,
) -> FetchMaskTensor<'l, T, i8, m![1], m![1], m![1], m![(#{0} 2 + B) #{0} 96 / 32, A], m![(#{0} 2 + B) #{0} 96 % 32]> {
    input
        .fetch::<m![(# 2 + B) # 96 / 32, A], m![(# 2 + B) # 96 % 32]>()
        //   Time      = m![(# 2 + B) # 96 / 32, A]
        //   Packet    = m![(# 2 + B) # 96 % 32]
        //   OutTime   = m![(#{0} 2 + B) #{0} 96 / 32, A]
        //   OutPacket = m![(#{0} 2 + B) #{0} 96 % 32]
        .fetch_mask::<m![(#{0} 2 + B) #{0} 96 / 32, A], m![(#{0} 2 + B) #{0} 96 % 32]>()
}

The diagram below shows the intent.

The sequencer shape matches the split-right case, but the base address shifts:

[
    (# 2 + B) # 96 % 32 = 32 : 1,    // loop 0: packet axis (carries both pads)
    A = 32 : 96,                       // loop 1: non-padded axis splits the chunks
    (# 2 + B) # 96 / 32 = 3 : 32,    // loop 2: outer chunk axis
] : 32 @ base_addr = -2

It then programs the hardware mask with these parameters:

• last_axis = the innermost packet axis, carrying both head and tail pads.
• valid_count_dim = Rightmost (one cell at the tail of the outermost iteration covers the right pad).
• rightmost_valid_count = [4, 0, 0, 0, 0, 0, 0, 0] (4 slots zeroed at the tail of every affected packet).
• left_pad = 2 (planned; zero the first 2 slots of every packet).

The masked return type is the same (#{0} 2 + B) #{0} 96 as the previous case, with only the Time-axis order differing.

Table Lookup

Table lookup provides hardware-accelerated lookup tables during the fetch stage. Each value is treated as an index into a pre-configured table, and the corresponding table entry is output instead. This is useful for operations that cannot be efficiently implemented with standard arithmetic, such as non-linear activation functions like Sigmoid and GeLU, or quantization schemes that use custom encoding tables. This enables:

• Non-linear activations: Implements Sigmoid, GeLU, and other functions through pre-computed lookup tables.
• Custom type casting: Translates specialized encodings like MXFP4 to standard formats using conversion tables.

Sigmoid and GeLU can also be expressed directly in the Vector Engine, so table lookup is one option among several for these activations rather than the only path.

impl<
    'l,
    const T: Tu,
    P: CanApplyFetchTableLookup,
    D: Scalar,
    Chip: M,
    Cluster: M,
    Slice: M,
    Time: M,
    Packet: M,
    B: Backend,
> TuTensor<'l, T, P, D, Chip, Cluster, Slice, Time, Packet, B>
{
    /// Runs the Fetch Adapter's table-lookup stage.
    #[primitive(TuTensor::fetch_table_lookup)]
    #[allow(unreachable_code)]
    pub fn fetch_table_lookup<OutD: Scalar>(
        self,
    ) -> FetchTableLookupTensor<'l, T, OutD, Chip, Cluster, Slice, Time, Packet, B> {
        verify_fetch_table_lookup::<D, OutD, Time, Packet>();
        FetchTableLookupTensor::new(self.ctx, todo!())
    }
}

#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 8];

/// Fetches with table lookup: each input value indexes into a pre-configured table.
/// Input [0, 1, 2, 3, 4, 5, 6, 7] with table[x] = 2*x
/// Output [0, 2, 4, 6, 8, 10, 12, 14]
fn fetch_with_table<'l, const T: Tu>(
    input: BeginTensor<'l, T, i8, m![1], m![1], m![1], m![1], m![A]>,
    table: &LookupTable<i8, i8>,
) -> FetchTableLookupTensor<'l, T, i8, m![1], m![1], m![1], m![1], m![A]> {
    input.fetch::<m![1], m![A]>().fetch_table_lookup::<i8>()
}

Type Casting

fetch_cast::<OutD>() converts the element type from D to OutD, preserving the Time and Packet mapping. Type casting adds 1 to 2 cycles of latency. fetch_cast performs only the type conversion; the integer widenings that hold a zero-point offset (i4 to i5, i8 to i9) are a separate stage, Zero-Point Subtraction, so fetch_cast never produces an i5/i9.

impl<
    'l,
    const T: Tu,
    P: CanApplyFetchCast,
    D: MaterializableScalar,
    Chip: M,
    Cluster: M,
    Slice: M,
    Time: M,
    Packet: M,
    B: Backend,
> TuTensor<'l, T, P, D, Chip, Cluster, Slice, Time, Packet, B>
{
    /// Runs the Fetch Adapter's type-casting stage.
    ///
    /// Converts the stream's element type from `D` to `OutD`. The mapping
    /// shape is preserved.
    #[primitive(TuTensor::fetch_cast)]
    pub fn fetch_cast<OutD: Scalar>(self) -> FetchCastTensor<'l, T, OutD, Chip, Cluster, Slice, Time, Packet, B>
    where
        D: FetchCast<OutD>,
    {
        FetchCastTensor::new(self.ctx, self.inner.map(|v| v.cast()))
    }
}

RNGD supports the following fetch_cast conversions (the i4 to i5 and i8 to i9 widenings are Zero-Point Subtraction, not type casts):

The example below fetches an 8-element i8 stream and casts it to i32. The Time and Packet mapping is unchanged across the call.

#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 8];

/// Fetches with type casting: converts i8 storage to i32 for computation.
/// Input:   i8 [0, 1, 2, 3, 4, 5, 6, 7]
/// Output: i32 [0, 1, 2, 3, 4, 5, 6, 7]
fn fetch_with_type_cast<'l, const T: Tu>(
    input: BeginTensor<'l, T, i8, m![1], m![1], m![1], m![1], m![A]>,
) -> FetchCastTensor<'l, T, i32, m![1], m![1], m![1], m![1], m![A]> {
    input.fetch::<m![1], m![A]>().fetch_cast::<i32>()
}

let mut ctx = Context::acquire();
let x: BeginTensor<'_, _, i8, m![1], m![1], m![1], m![1], m![A]> = BeginTensor::new(&mut ctx.main, Tensor::zero());
let _o = fetch_with_type_cast(x);

Type casting adds an additional limit on read_size. The cast output per fetch must fit in a single 32-byte flit (see Collect Engine).

• Valid:
  • i4 -> i32, read_size = 8 (4 bytes): produces 8 × 4 = 32 B
  • i8 -> i32, read_size = 8 (8 bytes): produces 8 × 4 = 32 B
• Invalid:
  • i4 -> i32, read_size = 16 (8 bytes): produces 16 × 4 = 64 B
  • i8 -> i32, read_size = 16 (16 bytes): produces 16 × 4 = 64 B

Zero-Point Subtraction

fetch_zero_point_sub::<OutD>(zero_point) subtracts the quantization zero_point from each element and widens the stream to the Contraction Engine’s staging type: i4 to i5, i8 to i9. It is the only stage that produces an i5/i9.

impl<
    'l,
    const T: Tu,
    P: CanApplyFetchZeroPointSub,
    D: MaterializableScalar,
    Chip: M,
    Cluster: M,
    Slice: M,
    Time: M,
    Packet: M,
    B: Backend,
> TuTensor<'l, T, P, D, Chip, Cluster, Slice, Time, Packet, B>
{
    /// Runs the Fetch Adapter's zero-point-subtraction stage.
    ///
    /// Subtracts `zero_point` and widens the stream from `D` to its contraction-engine staging
    /// type `OutD` (`i4 -> i5`, `i8 -> i9`), the only way to produce an i5/i9
    /// stream. The result may only feed `contract_outer`; it is not
    /// [`MaterializableScalar`], so committing or re-routing it is a compile
    /// error. The mapping shape is preserved.
    ///
    /// Panics if `zero_point` is outside the source type's range
    /// ([`FetchZeroPointSub::ZERO_POINT_RANGE`]); a zero point in range keeps
    /// every widened residual within `OutD`, so this one check (independent of
    /// the stream data) is enough.
    #[primitive(TuTensor::fetch_zero_point_sub)]
    pub fn fetch_zero_point_sub<OutD: Scalar>(
        self,
        zero_point: i32,
    ) -> FetchZeroPointSubTensor<'l, T, OutD, Chip, Cluster, Slice, Time, Packet, B>
    where
        D: FetchZeroPointSub<OutD>,
    {
        let zero_point_range = <D as FetchZeroPointSub<OutD>>::ZERO_POINT_RANGE;
        assert!(
            zero_point_range.contains(&zero_point),
            "zero_point {zero_point} is outside the source type's quantized range {zero_point_range:?}",
        );
        FetchZeroPointSubTensor::new(self.ctx, self.inner.map(|v| v.zero_point_sub(zero_point)))
    }
}

Why the extra bit

Subtracting the zero point turns an unsigned-around-zero_point quantized value into a signed residual whose range no longer fits the input width. For a symmetric-signed input the residual is a difference of two same-width values:

• i4 residual: [-8, 7] - [-8, 7] = [-15, 15], which needs i5’s [-16, 15].
• i8 residual: [-128, 127] - [-128, 127] = [-255, 255], which needs i9’s [-256, 255].

The subtraction therefore produces one more bit than it consumes. The conversion checks this at runtime: a residual outside the i5/i9 range (an out-of-range zero_point or input) is rejected rather than silently wrapped.

Contraction Engine specification

i5 and i9 exist only as Contraction Engine operands. The engine multiplies operand pairs drawn from the same integer precision family and accumulates in i32:

Either operand may be the raw form (i4/i8) or its zero-point-subtracted staging (i5/i9); the two operands need not match within a family, but they may not cross families (no i4 against i8) or kinds (no integer against float). Floating-point contraction pairs (bf16, f8e4m3, f8e5m2) must match exactly and are never zero-point-subtracted.

Staging is contraction-only

An i5/i9 stream may flow through the Switch Engine and Collect Engine, but from there its only legal consumer is contract_outer. It cannot be committed to memory, stored to a register file (to_trf/to_vrf), transposed, or fed to any other engine.

This restriction is enforced at compile time, not by a runtime check: passing an i5/i9 stream to any consumer other than contract_outer is a compile error.

Switch Engine

While every other Tensor Unit engine runs per-slice on its own DM partition, the Switch Engine moves data across slices through a 256-slice ring network: broadcasting one slice’s value to a group, swapping values between slices, or permuting which slice holds which value.

Interface

FetchTensor::switch() produces a SwitchTensor, preserving Chip, Cluster, Packet, and the underlying data values. Only the Slice and Time mappings change to reflect the selected configuration.

impl<'l, const T: Tu, P: CanApplySwitch, D: Scalar, Chip: M, Cluster: M, Slice: M, Time: M, Packet: M, B: Backend>
    TuTensor<'l, T, P, D, Chip, Cluster, Slice, Time, Packet, B>
{
    /// Applies switching network routing only. The packet passes through
    /// unchanged, no padding, no reshaping. Use `collect` afterwards to
    /// normalize the packet to flit-sized chunks.
    #[primitive(TuTensor::switch)]
    pub fn switch<OutSlice: M, OutTime: M>(
        self,
        config: SwitchConfig,
    ) -> SwitchTensor<'l, T, D, Chip, Cluster, OutSlice, OutTime, Packet, B> {
        verify_switch::<Slice, Time, OutSlice, OutTime>(&config);
        SwitchTensor::new(self.ctx, self.inner.transpose(true))
    }
}

The kernel writer picks OutSlice, OutTime, and a SwitchConfig argument that selects the configuration and its parameters. SwitchConfig is one of the predefined variants (Broadcast01, Broadcast1, Transpose, InterTranspose, TransposedBroadcast1) for common patterns, or a CustomBroadcast for more general patterns. The numeric suffix in each variant name lists the slice sub-dimensions that move out of Slice. For example, Broadcast01 broadcasts both slice0 and slice1, while Broadcast1 broadcasts only slice1.

The compiler verifies that OutSlice and OutTime match the configuration’s required dimension structure (each per-config section below shows the required structure with input/output diagrams), and compilation fails when they do not. Every configuration also requires InSlice::SIZE == OutSlice::SIZE: the switch preserves the total slice count.

Regular Configurations

Each regular configuration partitions the 256 slices on a chip into parallel sub-rings of ring_size slices each. ring_size is derived from the configuration’s parameters (typically slice1 × slice0) and determines both the partitioning granularity and the cycle cost. See Architecture below for the ring topology and per-router decision logic, and Performance for the cycle-factor breakdown.

Regular configurations cover six common patterns. Arbitrary Slice sub-dimension permutations not expressible by one of these patterns require CustomBroadcast instead.

Configurations that introduce broadcast axes (X or Y in Broadcast01, Broadcast1, TransposedBroadcast1) require those axes to be new: they must not already appear in input Slice or input Time.

Forwarding

Forwarding leaves the Slice and Time mappings unchanged: every router outputs its own slice’s input directly, with no cross-slice movement.

SwitchConfig has no Forwarding variant. When no inter-slice exchange is needed, skip .switch() and call .collect() directly on the FetchTensor.

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 256, B = 64, C = 32];

fn forwarding<'l, const T: Tu>(
    input: FetchTensor<'l, T, f32, m![1], m![1 # 2], m![A], m![B], m![C]>,
) -> CollectTensor<'l, T, f32, m![1], m![1 # 2], m![A], m![B, C / 8], m![C % 8]> {
    input.collect()
}

let mut ctx = Context::acquire();

let f: FetchTensor<'_, _, f32, m![1], m![1 # 2], m![A], m![B], m![C]> = FetchTensor::new(&mut ctx.main, Tensor::zero());
let _o = forwarding(f);
}

Broadcast01

Broadcast01 broadcasts each slice’s data to every slice in a sub-ring, moving both inner Slice sub-dimensions (slice0 and slice1) into Time.

The input dimension structure (outermost to innermost, left to right):

┌──────────────────────────┬───────────────┐
│          Slice           │      Time     │
├────────┬────────┬────────┼───────┬───────┤
│ slice2 │ slice1 │ slice0 │ time1 │ time0 │
└────────┴────────┴────────┴───────┴───────┘

After switching, slice1 and slice0 move from Slice into Time to broadcast across the sub-ring. Two new broadcast dimensions, labeled X and Y, fill their vacated Slice positions:

┌──────────────────────┬─────────────────────────────────┐
│        Slice         │              Time               │
├────────┬──────┬──────┼───────┬────────┬───────┬────────┤
│ slice2 │  X   │  Y   │ time1 │ slice1 │ time0 │ slice0 │
└────────┴──────┴──────┴───────┴────────┴───────┴────────┘

The sub-ring spans ring_size = slice1 × slice0 slices, one for each (slice1, slice0) combination at fixed slice2. Every slice sends its packet around the sub-ring, and every slice receives all ring_size packets, so each output slice ends up holding the full broadcast group’s data, indexed along the new X and Y axes in the output Slice.

Example

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 256, B = 64, C = 63, D = 2, X = 2, Y = 2];

fn broadcast01<'l, const T: Tu>(
    input: FetchTensor<'l, T, f32, m![1], m![D], m![A], m![B], m![C # 64]>,
) -> SwitchTensor<'l, T, f32, m![1], m![D], m![A / 4, X, Y], m![B / 4, A / 2 % 2, B % 4, A % 2], m![C # 64]> {
    input.switch::<m![A / 4, X, Y], m![B / 4, A / 2 % 2, B % 4, A % 2]>(
        SwitchConfig::Broadcast01 {
            slice1: 2,
            slice0: 2,
            time0: 4
        }
    )
}

let mut ctx = Context::acquire();

let f: FetchTensor<'_, _, f32, m![1], m![D], m![A], m![B], m![C # 64]> = FetchTensor::new(&mut ctx.main, Tensor::zero());
let _o= broadcast01(f);
}

With slice1 = 2 (size of broadcast X), slice0 = 2 (size of broadcast Y), and time0 = 4, the compiler derives slice2 = 64, time1 = 16, and ring_size = 4 (64 sub-rings span 256 slices).

Sub-dimensions resolve to slice2 = A / 4, slice1 = A / 2 % 2, slice0 = A % 2, time1 = B / 4, time0 = B % 4, giving OutSlice = m![A / 4, X, Y] and OutTime = m![B / 4, A / 2 % 2, B % 4, A % 2].

Cycle estimate: ring_size × Time::SIZE × flits_per_packet = 4 × 64 × 8 = 2048, where Time::SIZE = 64 and flits_per_packet = sizeof(f32) × Packet::SIZE / 32 = 4 × 64 / 32 = 8.

Broadcast1

The input dimension structure (outermost to innermost, left to right):

┌──────────────────────────┬────────┐
│           Slice          │  Time  │
├────────┬────────┬────────┼────────┤
│ slice2 │ slice1 │ slice0 │ time0  │
└────────┴────────┴────────┴────────┘

After switching, slice1 moves from Slice into Time to broadcast across the sub-ring while slice0 stays in Slice. A new broadcast dimension labeled X fills slice1’s vacated Slice position:

┌────────────────────────┬────────────────┐
│          Slice         │      Time      │
├────────┬──────┬────────┼───────┬────────┤
│ slice2 │  X   │ slice0 │ time0 │ slice1 │
└────────┴──────┴────────┴───────┴────────┘

The sub-ring spans ring_size = slice1 × slice0 slices, the same physical extent as Broadcast01’s. But broadcasting happens only along slice1: each output slice receives the slice1 packets from sources at the same slice0 position, sequenced along the innermost Time. The new X axis in the output Slice (sized slice1) replicates this collected data, while slice0 itself is preserved at its original position.

Example

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 256, B = 64, C = 63, X = 4];

fn broadcast1<'l, const T: Tu>(
    input: FetchTensor<'l, T, i8, m![1], m![1 # 2], m![A], m![B], m![C # 64]>,
) -> SwitchTensor<'l, T, i8, m![1], m![1 # 2], m![A / 32, X, A % 8], m![B, A / 8 % 4], m![C # 64]> {
    input.switch::<m![A / 32, X, A % 8], m![B, A / 8 % 4]>(
        SwitchConfig::Broadcast1 {
            slice1: 4,
            slice0: 8,
        }
    )
}

let mut ctx = Context::acquire();

let f: FetchTensor<'_, _, i8, m![1], m![1 # 2], m![A], m![B], m![C # 64]> = FetchTensor::new(&mut ctx.main, Tensor::zero());
let _o = broadcast1(f);
}

With slice1 = 4 (size of broadcast X) and slice0 = 8, the compiler derives slice2 = 8 and ring_size = 32 (8 sub-rings span 256 slices).

Sub-dimensions resolve to slice2 = A / 32, slice1 = A / 8 % 4, slice0 = A % 8, time0 = B, giving OutSlice = m![A / 32, X, A % 8] and OutTime = m![B, A / 8 % 4].

Cycle estimate: ring_size × Time::SIZE × flits_per_packet = 32 × 64 × 2 = 4096, where Time::SIZE = 64 and flits_per_packet = sizeof(i8) × Packet::SIZE / 32 = 1 × 64 / 32 = 2.

Transpose

Transpose swaps slice1 and slice0 within the innermost part of Slice.

The input and output Slice orderings:

┌──────────────────────────┐         ┌──────────────────────────┐
│           Slice          │         │           Slice          │
├────────┬────────┬────────┤   ──►   ├────────┬────────┬────────┤
│ slice2 │ slice1 │ slice0 │         │ slice2 │ slice0 │ slice1 │
└────────┴────────┴────────┘         └────────┴────────┴────────┘

Each sub-ring spans slice0 × slice1 slices and circulates data so every slice in the sub-ring ends up holding the value previously held by its swap partner.

Transpose requires input Time and output Time to match (after normalization).

Example

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 256, B = 64, C = 63];

fn transpose<'l, const T: Tu>(
    input: FetchTensor<'l, T, i8, m![1], m![1 # 2], m![A], m![B], m![C # 64]>,
) -> SwitchTensor<'l, T, i8, m![1], m![1 # 2], m![A / 64, A % 2, A / 2 % 32], m![B], m![C # 64]> {
    input.switch::<m![A / 64, A % 2, A / 2 % 32], m![B]>(SwitchConfig::Transpose {
        slice1: 32,
        slice0: 2,
    })
}

let mut ctx = Context::acquire();

let f: FetchTensor<'_, _, i8, m![1], m![1 # 2], m![A], m![B], m![C # 64]> = FetchTensor::new(&mut ctx.main, Tensor::zero());
let _o = transpose(f);
}

With slice1 = 32 and slice0 = 2, the compiler derives slice2 = 4 and ring_size = 64 (4 sub-rings span 256 slices).

Sub-dimensions resolve to slice2 = A / 64, slice1 = A / 2 % 32, slice0 = A % 2, time0 = B, giving OutSlice = m![A / 64, A % 2, A / 2 % 32] and OutTime = m![B] (slice0 and slice1 swap; Time is unchanged).

Cycle estimate: ring_size × Time::SIZE × flits_per_packet = 64 × 64 × 2 = 8192, where Time::SIZE = 64 and flits_per_packet = sizeof(i8) × Packet::SIZE / 32 = 1 × 64 / 32 = 2.

InterTranspose

InterTranspose swaps a dimension between Slice and Time: slice1 moves into Time and time1 moves into Slice (regular Transpose stays within Slice).

The input dimension structure (outermost to innermost, left to right):

┌──────────────────────────┬───────────────────────┐
│           Slice          │         Time          │
├────────┬────────┬────────┼───────┬───────┬───────┤
│ slice2 │ slice1 │ slice0 │ time2 │ time1 │ time0 │
└────────┴────────┴────────┴───────┴───────┴───────┘

After switching, slice1 and time1 swap positions across the Slice/Time boundary:

┌─────────────────────────┬────────────────────────┐
│          Slice          │          Time          │
├────────┬───────┬────────┼───────┬───────┬────────┤
│ slice2 │ time1 │ slice0 │ time2 │ time0 │ slice1 │
└────────┴───────┴────────┴───────┴───────┴────────┘

Each sub-ring spans slice1 × slice0 slices and circulates data over time1 time steps so each slice’s value previously indexed by slice1 ends up indexed by time1, and vice versa.

InterTranspose enforces three sizing constraints:

• InSlice spans all 256 slices: slice2 × slice1 × slice0 == 256.
• The swapped dimensions match in size: time1.SIZE == slice1.
• InTime::SIZE is divisible by slice1 × time0 so the time2 decomposition is integral.

Example

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 256, B = 8, C = 32];

fn inter_transpose<'l, const T: Tu>(
    input: FetchTensor<'l, T, i8, m![1], m![1 # 2], m![A], m![B], m![C # 32]>,
) -> SwitchTensor<'l, T, i8, m![1], m![1 # 2], m![A / 32, B / 2 % 2, A % 16], m![B / 4, B % 2, A / 16 % 2], m![C # 32]> {
    input.switch::<m![A / 32, B / 2 % 2, A % 16], m![B / 4, B % 2, A / 16 % 2]>(
        SwitchConfig::InterTranspose {
            slice1: 2,
            slice0: 16,
            time0: 2,
        })
}

let mut ctx = Context::acquire();

let f: FetchTensor<'_, _, i8, m![1], m![1 # 2], m![A], m![B], m![C # 32]> = FetchTensor::new(&mut ctx.main, Tensor::zero());
let _o = inter_transpose(f);
}

With slice1 = 2, slice0 = 16, and time0 = 2, the compiler derives slice2 = 8, time2 = 2, and ring_size = 32 (8 sub-rings span 256 slices).

Sub-dimensions resolve to slice2 = A / 32, slice1 = A / 16 % 2, slice0 = A % 16, time2 = B / 4, time1 = B / 2 % 2, time0 = B % 2, giving OutSlice = m![A / 32, B / 2 % 2, A % 16] and OutTime = m![B / 4, B % 2, A / 16 % 2] (slice1 and time1 swap between Slice and Time).

Cycle estimate: ring_size × Time::SIZE × flits_per_packet = 32 × 8 × 1 = 256, where Time::SIZE = 8 and flits_per_packet = sizeof(i8) × Packet::SIZE / 32 = 1 × 32 / 32 = 1.

TransposedBroadcast1

TransposedBroadcast1 broadcasts slice0 to the innermost Time position and shifts slice1 to the innermost Slice position, equivalent to applying Transpose followed by Broadcast1. An input tensor structured as:

┌──────────────────────────┬────────┐
│           Slice          │  Time  │
├────────┬────────┬────────┼────────┤
│ slice2 │ slice1 │ slice0 │ time0  │
└────────┴────────┴────────┴────────┘

becomes the output below, where slice0 moves to the innermost Time position to broadcast across the sub-ring, slice1 shifts to the innermost Slice position, and a broadcast dimension fills slice1’s vacated middle slot:

┌────────────────────────┬────────────────┐
│          Slice         │      Time      │
├────────┬──────┬────────┼───────┬────────┤
│ slice2 │  Y   │ slice1 │ time0 │ slice0 │
└────────┴──────┴────────┴───────┴────────┘

Each sub-ring spans slice0 × slice1 slices and circulates data so every slice ends up with its swap partner’s value, broadcast across the slice0 positions at the innermost Time sub-dimension.

Example

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 256, B = 16, C = 32, Y = 8];

fn transposed_broadcast1<'l, const T: Tu>(
    input: FetchTensor<'l, T, i8, m![1], m![1 # 2], m![A], m![B], m![C # 32]>,
) -> SwitchTensor<'l, T, i8, m![1], m![1 # 2], m![A / 64, Y, A / 8 % 8], m![B, A % 8], m![C # 32]> {
    input.switch::<m![A / 64, Y, A / 8 % 8], m![B, A % 8]>(
        SwitchConfig::TransposedBroadcast1 {
            slice1: 8,
            slice0: 8,
        }
    )
}

let mut ctx = Context::acquire();

let f: FetchTensor<'_, _, i8, m![1], m![1 # 2], m![A], m![B], m![C # 32]> = FetchTensor::new(&mut ctx.main, Tensor::zero());
let _o = transposed_broadcast1(f);
}

With slice1 = 8 and slice0 = 8 (size of broadcast Y), the compiler derives slice2 = 4 and ring_size = 64 (4 sub-rings span 256 slices).

Sub-dimensions resolve to slice2 = A / 64, slice1 = A / 8 % 8, slice0 = A % 8, time0 = B, giving OutSlice = m![A / 64, Y, A / 8 % 8] and OutTime = m![B, A % 8].

Cycle estimate: ring_size × Time::SIZE × flits_per_packet = 64 × 16 × 1 = 1024, where Time::SIZE = 16 and flits_per_packet = sizeof(i8) × Packet::SIZE / 32 = 1 × 32 / 32 = 1.

Architecture

To execute the configurations introduced above, the Switch Engine arranges all 256 slices on a chip into a single physical ring (one router per slice), partitioned into 256 / ring_size parallel sub-rings of ring_size slices each (one such sub-ring is shown below). For regular configurations the compiler derives ring_size from the configuration’s parameters (slice1, slice0, time0); for CustomBroadcast the kernel writer sets ring_size directly.

   ┌────────────────────┐    ┌────────────────────┐    ┌────────────────────┐    ┌────────────────────┐
   │      Router 0      │◀──▶│      Router 1      │◀──▶│        ...         │◀──▶│ Router ring_size-1 │
   └────────────────────┘    └────────────────────┘    └────────────────────┘    └────────────────────┘
             ▲                                                                                          ▲
             └──────────────────────────────────────────────────────────────────────────────────────────┘
                                            wrap-around (links are bidirectional)

The Switch Engine is configured by a snoop bitmap: 256 entries (one per slice), each naming the source slices whose data should arrive at that output slice. Regular configurations come with built-in bitmap generators. CustomBroadcast instead lets the compiler synthesize an arbitrary bitmap from the kernel writer’s input/output mappings.

Based on its bitmap entry, every router decides per incoming packet which combination of three actions to take:

• Output: deliver the packet to the local slice’s downstream pipeline, when the packet’s source slice is selected for delivery here.
• Forward right: pass the packet to the right neighbor’s router.
• Forward left: pass the packet to the left neighbor’s router.

In each sub-ring, the leftmost router sends its own data rightward, and the rightmost router sends its own data leftward. When ring_size > 2, the intermediate routers output incoming left-neighbor data and forward it rightward. Every router also outputs any data that arrives from a neighbor.

The trace below illustrates per-router execution in one 2-slice sub-ring (ring_size = 2), running a 2-slice broadcast pattern. The remaining 127 sub-rings behave identically and are omitted. Each link has a 1-cycle traversal latency, so the leftmost router initiates at cycle 0 while the rightmost begins at cycle 1, once the first packet from its left neighbor arrives. With axes![A = 256, B = 2, C = 32], Slice = m![A], Time = m![B], and Packet = m![C], the shown sub-ring contains slices 0 and 1: the leftmost slice holds packets [0, 1] and the rightmost slice holds [2, 3]. Each cell reads as <packet>: from <source>, to <action>, where source is input/left/right and action is one or more of output/right/left.

After the trace, both slices in the sub-ring hold all four packets, completing the broadcast.

The bitmap encodes transformations through the shape of its entries:

• Broadcast shape: multiple output slices receiving the same source data have identical bitmap entries.
• Slice-to-Time shape: one output slice listing several source slices means that output collects data from all of them across consecutive time steps.

For instance, the bitmap that reproduces the Broadcast01 example above looks like this:

Rows 0-3 share identical entries because slices {0, 1, 2, 3} all receive data from input slices {0, 1, 2, 3} (the broadcast shape), and each row lists all four sources because slice1 and slice0 collapse from Slice into Time (the Slice-to-Time shape). This pattern repeats every 4 rows for the 64 sub-rings. SwitchConfig::Broadcast01 generates this bitmap automatically.

Performance

A switch operation takes roughly ring_size × Time::SIZE × flits_per_packet cycles. All sub-rings advance in parallel, so this per-ring cycle count is also the chip-wide latency. The three factors are:

• ring_size: cycles for one flit to traverse one sub-ring. A larger ring_size reaches more slices per ring at a higher per-ring cost, while a smaller ring_size partitions the cluster into more parallel rings at a lower per-ring cost.
• Time::SIZE: number of time steps in the input tensor. The sub-ring traversal repeats once per time step.
• flits_per_packet: flits per packet, equal to size of D[Packet::SIZE] divided by 32 bytes. The traversal also repeats once per flit of the packet.

Custom Configurations

Custom configurations handle movement patterns no regular configuration expresses, such as arbitrary dimension permutations or partial dimension extractions. This flexibility comes with configuration overhead and the constraints listed at the end of the section.

The SwitchConfig::CustomBroadcast variant carries a single field:

/// Routes data across slices using a custom snoop bitmap.
/// The bitmap is computed by the compiler from the input shape and
/// topology parameters.
CustomBroadcast {
    /// Ring group size for the custom routing.
    ring_size: usize,
},

Where regular configurations supply built-in generators for the snoop bitmap, CustomBroadcast lets the compiler synthesize the bitmap directly from the kernel writer’s input/output mappings together with ring_size.

Supported Transformation Patterns

Custom bitmaps cover two patterns regular configurations cannot:

• Free transpose with broadcast: arbitrary permutation and broadcast of partitioning dimensions, beyond the fixed forms in Transpose or TransposedBroadcast1.
• Partial dimension extraction: only a subset of a dimension’s values moves to Time during broadcasting, whereas regular configurations like Broadcast01 always move the whole dimension.

The examples below illustrate these patterns.

Configuration Overhead

Writing a custom snoop bitmap streams configuration data into the Switch Engine’s Special Function Registers (SFRs), and this SFR write occupies both the DMA Engine and the sub-context for the duration. While the bitmap is loading, the DMA and sub contexts cannot run any other operation, so the cost manifests as reduced scheduling parallelism rather than a fixed-cycle stall.

Example 1: Arbitrary Permutation

This example reverses the four innermost slice sub-dimensions (A / 4, A % 4, B / 4, B % 4) into [3, 2, 1, 0], a pattern no regular configuration expresses.

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 16, B = 16, C = 8, D = 8, E = 8];

fn arbitrary_permutation<'l, const T: Tu>(
    input: FetchTensor<'l, T, f32, m![1], m![1 # 2], m![A, B], m![C], m![D, E]>,
) -> SwitchTensor<'l, T, f32, m![1], m![1 # 2], m![B % 4, B / 4, A % 4, A / 4], m![C], m![D, E]> {
    input.switch::<m![B % 4, B / 4, A % 4, A / 4], m![C]>(
        SwitchConfig::CustomBroadcast { ring_size: 256 }
    )
}

let mut ctx = Context::acquire();

let f: FetchTensor<'_, _, f32, m![1], m![1 # 2], m![A, B], m![C], m![D, E]> = FetchTensor::new(&mut ctx.main, Tensor::zero());
let _o = arbitrary_permutation(f);
}

The output Slice = m![B % 4, B / 4, A % 4, A / 4] permutes the input slice shape [0, 1, 2, 3] into [3, 2, 1, 0], which no regular configuration covers but a custom bitmap does.

Plugging into the cycle formula, cycles ≈ ring_size × Time::SIZE × flits_per_packet = 256 × 8 × 8 = 16384:

• ring_size = 256
• Time::SIZE = C::SIZE = 8
• flits_per_packet = sizeof(f32) × Packet::SIZE / 32 = 4 × 64 / 32 = 8 (Packet = m![D, E], D::SIZE × E::SIZE = 64)

The maximum ring_size = 256 is necessary because the permutation creates dependencies across all slices with no repeating structure, so input and output slices can be arbitrarily far apart in the ring index, and any smaller sub-ring would fail to cover at least one such pair.

Example 2: Multi-dimension Broadcast

Unlike Example 1’s pure permutation, this example moves two non-contiguous dimensions (A % 2 and B % 2) from Slice to Time, broadcasting at their original positions.

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 16, B = 16, C = 8, D = 8, E = 8, X = 2, Y = 2];

fn multi_axis_broadcast<'l, const T: Tu>(
    input: FetchTensor<'l, T, f32, m![1], m![1 # 2], m![A, B], m![C], m![D, E]>,
) -> SwitchTensor<'l, T, f32, m![1], m![1 # 2], m![A / 2, X, B / 2, Y], m![C, A % 2, B % 2], m![D, E]> {
    input.switch::<m![A / 2, X, B / 2, Y], m![C, A % 2, B % 2]>(
        SwitchConfig::CustomBroadcast { ring_size: 32 }
    )
}

let mut ctx = Context::acquire();

let f: FetchTensor<'_, _, f32, m![1], m![1 # 2], m![A, B], m![C], m![D, E]> = FetchTensor::new(&mut ctx.main, Tensor::zero());
let _o = multi_axis_broadcast(f);
}

The output moves A % 2 and B % 2 from Slice to Time, broadcasting at their original positions via the broadcast dimensions X and Y. Broadcast01 supports a similar form but requires the broadcast dimensions (slice0, slice1) to be contiguous in the input slice, so it cannot express non-contiguous dimensions moving from Slice to Time. A custom bitmap expresses it instead.

Plugging into the cycle formula, cycles ≈ ring_size × Time::SIZE × flits_per_packet = 32 × 8 × 8 = 2048:

• ring_size = 32 (the outermost A / 2 partition needs no inter-sub-ring exchange, so only the innermost 32 slices within each sub-ring communicate)
• Time::SIZE = 8
• flits_per_packet = sizeof(f32) × Packet::SIZE / 32 = 4 × 64 / 32 = 8 (Packet = m![D, E], D::SIZE × E::SIZE = 64)

Example 3: Partial Axis Extraction (Slicing)

Unlike Examples 1 and 2, which include every value of the moved dimensions, here only the 3 valid values of C # 4 move from Slice to Time; its padding cell stays behind.

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 16, B = 4, C = 3, D = 8, E = 8, X = 4];

fn partial_axis_extraction<'l, const T: Tu>(
    input: FetchTensor<'l, T, f32, m![1], m![1 # 2], m![A, B, C # 4], m![D], m![E]>,
) -> SwitchTensor<'l, T, f32, m![1], m![1 # 2], m![A, B, X], m![D, C], m![E]> {
    input.switch::<m![A, B, X], m![D, C]>(
        SwitchConfig::CustomBroadcast { ring_size: 4 }
    )
}

let mut ctx = Context::acquire();

let f: FetchTensor<'_, _, f32, m![1], m![1 # 2], m![A, B, C # 4], m![D], m![E]> = FetchTensor::new(&mut ctx.main, Tensor::zero());
let _o = partial_axis_extraction(f);
}

The output moves C # 4 from Slice to Time, placing a broadcast axis X at its vacated Slice position, and the fourth value (C # 4 = 3), a pure padding cell, is dropped, so only the three valid values C are extracted. Broadcast1 supports a similar form but always moves the entire dimension, so it cannot express a subset.

This partial extraction is allowed only because the discarded value is padding: C # 4 holds C = 3 valid elements in a slot padded to 4, so slicing it down to C (C # 4 → C) throws away nothing real. Slicing a valid range would instead discard live input, which the Dropped values must be padding constraint forbids. A custom bitmap expresses this padding-only extraction instead.

Plugging into the cycle formula, cycles ≈ ring_size × Time::SIZE × flits_per_packet = 4 × 24 × 1 = 96:

• ring_size = 4 (the outer A, B partition needs no inter-sub-ring exchange, so only the innermost 4 slices (one C # 4 group) within each sub-ring communicate)
• Time::SIZE = D::SIZE × C::SIZE = 8 × 3 = 24
• flits_per_packet = sizeof(f32) × Packet::SIZE / 32 = 4 × 8 / 32 = 1 (Packet = m![E], E::SIZE = 8)

The bitmap shows padding-only extraction directly: bitmap[0] = {0, 1, 2} means output slice 0 receives from the 3 valid input slices in its C # 4 group while skipping index 3, the padding cell. Reading {0, 1, 2, 3} would pull that padding in as if it were real data.

Constraints

Custom configurations come with seven constraints that bound this flexibility.

Broadcast axes must be new

Same rule as the Regular Configurations intro: each broadcast axis introduced in the output Slice must not appear in the input Slice or input Time.

For instance, with axes![A = 256, B = 64, C = 32] and input Slice = m![A], Time = m![B], Packet = m![C # 32], an output Slice = m![A / 4, B / 32, A % 4] violates this constraint because B already appears in the input Time.

Each broadcast axis used exactly once

Each broadcast axis must appear exactly once in the output Slice. Repeating the same axis at two output positions has no defined meaning for the routing bitmap.

For instance, output Slice = m![A / 4, X, X] (where X is a new axis used twice) violates this constraint.

Broadcast axes must not be padded

Broadcast axes in the output Slice must not carry padding (no Axis # N form). Padding on a broadcast axis would leave routing destinations undefined for the padded positions.

For instance, output Slice = m![A / 4, X # 4, Y] violates this constraint because X is a broadcast axis with padding.

Order preservation

Axes moving from Slice to Time must preserve their relative order from the input slice dimension, and the verifier in SwitchConfig::CustomBroadcast panics at kernel compile time when they do not. Each router has minimal buffering (one packet) and must immediately decide to output locally or forward, with no opportunity to buffer multiple packets and reorder them.

For instance, with axes![A = 16, B = 16, C = 8, D = 8, E = 8] and dtype = i8, mapping input Slice = m![A, B], Time = m![C], Packet = m![D, E] to output Slice = m![A, B / 4, 4], Time = m![C, B % 2, B / 2], Packet = m![D, E] violates this constraint. Here B % 2 and B / 2 appear in reversed order relative to their input slice arrangement. Output Time = m![C, B / 2, B % 2] would be valid, since B / 2, B % 2 match the input order.

Innermost time position

Axes moving from Slice to Time must occupy the innermost positions of the output Time. Data from other slices arrives last within each packet in the pipeline, so Slice-to-Time sub-dimensions naturally land at the innermost time dimensions. Placing them elsewhere would demand buffering and reordering full time sequences, which the hardware cannot do.

For instance, with the same axes! and dtype as above, mapping input Slice = m![A, B], Time = m![C], Packet = m![D, E] to output Slice = m![A / 2, 2, B / 2, 2], Time = m![A % 2, C, B % 2], Packet = m![D, E] violates this constraint. Here A % 2 and B % 2 preserve their relative order correctly, but C sits between them.

Note

Broadcast01 works around this constraint via the time0 parameter, but custom configurations lack that mechanism and must follow the constraint strictly.

Dropped values must be padding

When fewer values of a dimension move from Slice to Time than the dimension spans (partial extraction, as in Example 3), every value left behind must be padding. Dropping a valid value would silently discard live input, so the verifier rejects it at kernel compile time.

For instance, with axes![A = 16, B = 4, C = 3, D = 8, E = 8, X = 4] and input Slice = m![A, B, C # 4], extracting C # 4 → C is allowed because the dropped fourth value is a padding cell. Slicing a fully valid axis, for instance B = 4 → B = 3, violates this constraint, since the dropped value carries real data.

Ring size

The ring_size parameter must be a power of 2. The compiler also derives the expected ring_size from the input/output mappings (the outermost non-direct-cast boundary) and rejects any user-supplied value that does not match.

Collect Engine

All downstream engines (Contraction Engine, Vector Engine, Cast Engine, Transpose Engine, and Commit Engine) consume exactly 32-byte flits. The Collect Engine normalizes arbitrary-sized packets to one flit in two steps:

1. Pad the input packet up to the next 32-byte boundary. Skipped if the packet is already 32-byte aligned.
2. Split at the flit boundary: the inner 32 bytes become Packet2, and the outer flit count is absorbed into Time2. Skipped if the packet is already 32 bytes.

The resulting CollectTensor either flows down the pipeline to a downstream engine or is stored in the Register Files.

Interface

SwitchTensor and FetchTensor both expose .collect() with the same semantics. The FetchTensor entry point bypasses the Switch Engine when no slice distribution is needed.

impl<'l, const T: Tu, P: CanApplyCollect, D: Scalar, Chip: M, Cluster: M, Slice: M, Time: M, Packet: M, B: Backend>
    TuTensor<'l, T, P, D, Chip, Cluster, Slice, Time, Packet, B>
{
    /// Normalizes packet to exactly 32 bytes (one flit).
    ///
    /// Pads to flit-aligned boundary, then splits: inner 32 bytes become
    /// `Packet2`, outer flit portion is absorbed into `Time2`. For packets
    /// already ≤ 32 bytes, only padding is added.
    #[primitive(TuTensor::collect)]
    pub fn collect<Time2: M, Packet2: M>(self) -> CollectTensor<'l, T, D, Chip, Cluster, Slice, Time2, Packet2, B> {
        verify_collect::<D, Time, Packet, Time2, Packet2>();
        CollectTensor::new(self.ctx, self.inner.transpose(false))
    }
}

Examples

Single-Flit Packet

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 8, B = 32];

fn collect_identity<'l, const T: Tu>(
    input: SwitchTensor<'l, T, i8, m![1], m![1 # 2], m![1 # 256], m![A], m![B]>,
) -> CollectTensor<'l, T, i8, m![1], m![1 # 2], m![1 # 256], m![A], m![B # 32]> {
    // B=32 elements × 1 byte (i8) = 32 bytes = one flit.
    // Time and Packet pass through unchanged.
    input.collect()
}

let mut ctx = Context::acquire();

let c: SwitchTensor<'_, _, i8, m![1], m![1 # 2], m![1 # 256], m![A], m![B]> = SwitchTensor::new(&mut ctx.main, Tensor::zero());
let _o = collect_identity(c);
}

When the input packet is already exactly 32 bytes, collect passes it through unchanged (B = 32 elements × 1 byte for i8 = 32 bytes).

Before:   Time = m![A]
          Packet = m![B]
          ┌──────────────────────────┐
          │            B             │  32 bytes
          └──────────────────────────┘

After:    Time = m![A]
          Packet = m![B # 32]
          ┌──────────────────────────┐
          │          B # 32          │  32 bytes
          └──────────────────────────┘

Sub-Flit Packet

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 8, B = 16];

fn collect_padding<'l, const T: Tu>(
    input: SwitchTensor<'l, T, i8, m![1], m![1 # 2], m![1 # 256], m![A], m![B]>,
) -> CollectTensor<'l, T, i8, m![1], m![1 # 2], m![1 # 256], m![A], m![B # 32]> {
    // B=16 elements × 1 byte = 16 bytes < 32 bytes.
    // Padded to 32 bytes: Packet2 = m![B # 32].
    // Time unchanged since it fits in one flit.
    input.collect()
}

let mut ctx = Context::acquire();

let c: SwitchTensor<'_, _, i8, m![1], m![1 # 2], m![1 # 256], m![A], m![B]> = SwitchTensor::new(&mut ctx.main, Tensor::zero());
let _o = collect_padding(c);
}

When the input packet is smaller than 32 bytes, collect pads to 32 bytes (B = 16 elements × 1 byte for i8 = 16 bytes).

Before:   Time = m![A]
          Packet = m![B]
          ┌────────────┐
          │     B      │  16 bytes
          └────────────┘

After:    Time = m![A]
          Packet = m![B # 32]
          ┌────────────┬─────────────┐
          │     B      │     pad     │  32 bytes
          └────────────┴─────────────┘

Multi-Flit Packet

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 8, B = 32];

fn collect_multi_flit<'l, const T: Tu>(
    input: SwitchTensor<'l, T, bf16, m![1], m![1 # 2], m![1 # 256], m![A], m![B]>,
) -> CollectTensor<'l, T, bf16, m![1], m![1 # 2], m![1 # 256], m![A, B / 16], m![B % 16]> {
    // B=32 elements × 2 bytes (bf16) = 64 bytes = 2 flits.
    // Inner 16 elements = 32 bytes → Packet2 = m![B % 16].
    // Outer 2 flits → absorbed into Time2 = m![A, B / 16].
    input.collect()
}

let mut ctx = Context::acquire();

let c: SwitchTensor<'_, _, bf16, m![1], m![1 # 2], m![1 # 256], m![A], m![B]> = SwitchTensor::new(&mut ctx.main, Tensor::zero());
let _o = collect_multi_flit(c);
}

When the input packet exceeds 32 bytes, collect splits into flits and absorbs the outer flit count into Time (B = 32 elements × 2 bytes for bf16 = 64 bytes, so B / 16 = 2 flits).

Before:   Time = m![A]
          Packet = m![B]
          ┌──────────────────────────┬──────────────────────────┐
          │       B / 16 == 0        │       B / 16 == 1        │  64 bytes
          └──────────────────────────┴──────────────────────────┘
                    32 bytes                   32 bytes

After:    Time = m![A, B / 16]
          Packet = m![B % 16]
          ┌──────────────────────────┐
          │          B % 16          │  32 bytes  × B/16 time steps
          └──────────────────────────┘

Multi-Flit Packet With Padding

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 8, B = 56];

fn collect_multi_flit_padded<'l, const T: Tu>(
    input: SwitchTensor<'l, T, i8, m![1], m![1 # 2], m![1 # 256], m![A], m![B]>,
) -> CollectTensor<'l, T, i8, m![1], m![1 # 2], m![1 # 256], m![A, B # 64 / 32], m![B # 64 % 32]> {
    // B is not 32-byte aligned; first pad B to a multiple of 32 bytes.
    // B # 64=64 elements × 1 byte (i8) = 64 bytes = 2 flits.
    // Inner 32 elements = 32 bytes → Packet2 = m![B # 64 % 32].
    // Outer 2 flits → absorbed into Time2 = m![A, B # 64 / 32].
    input.collect()
}

let mut ctx = Context::acquire();

let c: SwitchTensor<'_, _, i8, m![1], m![1 # 2], m![1 # 256], m![A], m![B]> = SwitchTensor::new(&mut ctx.main, Tensor::zero());
let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { collect_multi_flit_padded(c) }));
}

When the input packet is not aligned to 32 bytes, it is first padded (B = 51 elements × 1 byte for i8 = 51 bytes, padded to 64). Then, collect splits into flits and absorbs the outer flit count (B # 64 / 32 = 2) into Time.

Before:   Time = m![A]
          Packet = m![B]
          ┌──────────────────────────┬───────────────┐
          │       B / 32 == 0        │  B / 32 == 1  │  51 bytes
          └──────────────────────────┴───────────────┘
                    32 bytes             19 bytes

Padded:   Time = m![A]
          Packet = m![B # 64]
          ┌──────────────────────────┬───────────────┬──────────┐
          │       B / 32 == 0        │  B / 32 == 1  │   pad    │  64 bytes
          └──────────────────────────┴───────────────┴──────────┘
                    32 bytes                   32 bytes

After:    Time = m![A, B # 64 / 32]
          Packet = m![B # 64 % 32]
          ┌──────────────────────────┐
          │       B # 64 % 32        │  32 bytes  × B # 64 / 32 time steps
          └──────────────────────────┘

Register File Loading

After normalization, store the CollectTensor into the Tensor Register File via .to_trf() or the Vector Register File via .to_vrf(). The “To TRF” / “To VRF” subsections below describe the store mechanism (time_inner derivation, [time_inner, Packet] sequenced into Element).

To TRF

.to_trf::<Row, Element>() partitions the TRF along its row dimension. The kernel writer chooses Row (the row layout in the TRF, with Row::SIZE in {1, 2, 4, 8}) and Element (the per-row element layout). The compiler then finds a time_inner such that Time decomposes into [Row, time_inner] and [time_inner, Packet] is sequenced into Element, so each row of the TRF is filled by time_inner consecutive flits.

.to_trf() uses the entire TRF (TrfAddress::Full). To let two tensors occupy the TRF independently, use .to_trf_at::<Row, Element>(address) with a TrfAddress that selects the region:

• Full: the entire TRF.
• FirstHalf / SecondHalf: the TRF split into two halves.

The compiler bounds the resulting tensor’s total byte size by the chosen region’s capacity.

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![B = 32];

fn load_trf<'l, const T: Tu>(
    input: CollectTensor<'l, T, i8, m![1], m![1 # 2], m![1 # 256], m![1], m![B]>,
) -> TrfTensor<i8, m![1], m![1 # 2], m![1 # 256], m![1], m![B]> {
    input.to_trf()
}

let mut ctx = Context::acquire();

let c: CollectTensor<'_, _, i8, m![1], m![1 # 2], m![1 # 256], m![1], m![B]> = CollectTensor::new(&mut ctx.main, Tensor::zero());
let _o = load_trf(c);
}

To VRF

.to_vrf::<Element>() stores the flits into the VRF; .to_vrf_at::<Element>(address) stores at a raw Address (no bounded-region selection). The kernel writer chooses Element, the destination element layout in the VRF. Unlike .to_trf (which accepts any Scalar element type), .to_vrf requires a VeScalar element type (i.e., i32 or f32) because the Vector Engine downstream consumes these types only.

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![B = 64];

fn load_vrf<'l, const T: Tu>(
    input: CollectTensor<'l, T, i32, m![1], m![1 # 2], m![1 # 256], m![B / 8], m![B % 8]>,
) -> VrfTensor<i32, m![1], m![1 # 2], m![1 # 256], m![B]> {
    input.to_vrf()
}

let mut ctx = Context::acquire();

let c: CollectTensor<'_, _, i32, m![1], m![1 # 2], m![1 # 256], m![B / 8], m![B % 8]> = CollectTensor::new(&mut ctx.main, Tensor::zero());
let _o = load_vrf(c);
}

Register Files

The Collect Engine streams into the Contraction Engine and the Vector Engine. Both engines also take input from a register file (one per slice): the Tensor Register File (TRF) feeds the Contraction Engine, and the Vector Register File (VRF) feeds the Vector Engine. These register files must be populated before their consumer engine runs.

Tensor Register File

Interface

A TrfTensor is a tensor stored in the TRF:

/// Tensor stored in the tensor register file.
#[primitive(TrfTensor)]
#[derive(Debug)]
pub struct TrfTensor<D: Scalar, Chip: M, Cluster: M, Slice: M, Lane: M, Element: M, B: Backend = CurrentBackend> {
    pub(crate) inner: Tensor<D, Pair<Chip, Pair<Cluster, Pair<Slice, Pair<Lane, Element>>>>, B>,
    #[expect(dead_code)]
    address: Option<TrfAddress>,
    _marker: PhantomData<(D, Chip, Cluster, Slice, Lane, Element)>,
}

Chip / Cluster / Slice pass through from the source. Lane indexes the spatial parallelism (1, 2, 4, or 8 active lanes). Element holds the per-lane layout.

From Collect Engine

.to_trf::<Lane, Element>() on CollectTensor produces a TrfTensor in the full TRF; .to_trf_at::<Lane, Element>(address) targets a TrfAddress region:

impl<'l, const T: Tu, P: CanApplyToTrf, D: Scalar, Chip: M, Cluster: M, Slice: M, Time: M, Packet: M, B: Backend>
    TuTensor<'l, T, P, D, Chip, Cluster, Slice, Time, Packet, B>
{
    /// Stores to the tensor register file.
    #[primitive(TuTensor::to_trf)]
    pub fn to_trf<Lane: M, Element: M>(self) -> TrfTensor<D, Chip, Cluster, Slice, Lane, Element, B> {
        verify_to_trf::<D, Lane, Time, Packet, Element>(&TrfAddress::Full);
        TrfTensor::new(self.inner.transpose(false), None)
    }

    /// Stores to the tensor register file at `address`.
    #[primitive(TuTensor::to_trf_at)]
    pub fn to_trf_at<Lane: M, Element: M>(
        self,
        address: TrfAddress,
    ) -> TrfTensor<D, Chip, Cluster, Slice, Lane, Element, B> {
        verify_to_trf::<D, Lane, Time, Packet, Element>(&address);
        TrfTensor::new(self.inner.transpose(false), Some(address))
    }
}

.to_trf reshapes the streaming Time / Packet into Lane / Element:

Lane    = Time / FlitsPerLane
Element = [Time % FlitsPerLane, Packet]

for some FlitsPerLane that the compiler derives from Lane and Time, so each lane is filled by FlitsPerLane consecutive flits.

For example, in a matmul kernel Lane holds output channels and Element holds the contracted axis.

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![V = 32, M = 32, N = 8, K = 32];

type Chip    = m![1];
type Cluster = m![V / 16];
type Slice   = m![V % 16 # 256];
type Lane    = m![N];

/// Stores matmul weights into TRF for consumption by `bmatmul` in
/// [Contraction Engine: Example: Batched MatMul](./contraction-engine/index.md#example-batched-matmul).
fn store_bmatmul_trf<'l, const T: Tu>(
    input: CollectTensor<'l, T, bf16, Chip, Cluster, Slice, m![N, K / 16], m![K % 16]>,
) -> TrfTensor<bf16, Chip, Cluster, Slice, Lane, m![K]> {
    input.to_trf_at(TrfAddress::FirstHalf)
}

let mut ctx = Context::acquire();

let c: CollectTensor<'_, _, bf16, Chip, Cluster, Slice, m![N, K / 16], m![K % 16]> = CollectTensor::new(&mut ctx.main, Tensor::zero());
let _o = store_bmatmul_trf(c);
}

From Data Memory

For completely contiguous input access (no gaps or reordering), TRF supports a short command (StoTRF), a compact hardware instruction that loads data from Data Memory directly into the TRF, bypassing the full Fetch → Switch → Collect → to_trf() pipeline. The shortcut trades arbitrary-layout support for lower setup overhead.

To Contraction Engine

Each read covers 8 lanes × (1 or 2) banks × 1 row × 320 bits per bank, producing 320 bits per lane per cycle for a narrow read (one bank) or 640 bits per lane per cycle for a wide read (both banks): all active lanes access one or both banks of the same row in parallel (one bank for narrow reads, both for wide reads). Per slice that totals 320 bytes/cycle (narrow) or 640 bytes/cycle (wide) across all 8 lanes. See TRF Sequencer for how the sequencer iterates these reads across rows and broadcasts.

Architecture

The TRF is a banked SRAM with the structure 8 lanes × 2 banks × 128 rows × 320 bits = 80 KB per slice. The 8 lanes operate in parallel, with 1, 2, 4, or 8 active per access.

How many elements pack into a single 320-bit row depends on the data type:

Only i4 elements promote on store: i4 → i5 (5 bits) and i4 → i9 (9 bits) leave room for the fetch adapter’s optional zero-point subtraction, which can widen i4 intermediates by one bit per nibble. i8 / f8 and bf16 stay at their native widths (8 and 16 bits respectively); the 320-bit row holds extra slack relative to a flat 8- or 16-bit packing so the same physical row width serves all types.

With fewer than 8 active lanes, each active lane sees more rows (as if the row count grew). Halving the active count doubles the rows per active lane (e.g., 4 active → 256 rows per bank, 1 active → 1024).

Double Buffering

The TRF enables double-buffering by splitting each bank into two halves: the TRF Sequencer loads from one half while a store fills the other, and the two can be flipped between iterations. Three address modes select the region, fixed at store time: Full uses all 128 rows per bank, FirstHalf uses rows 0–63, and SecondHalf uses rows 64–127. The half modes cap per-slice capacity at 40 KB.

See Scheduling: Double-Buffering Pattern for the kernel pattern that uses these halves across the main and sub contexts.

Both halves share the same banks, so reads and writes contend at the bank level even though they target different rows. When both target the same bank in a cycle, the read takes priority because the contraction pipeline needs the data this cycle while the store can wait.

The TRF mitigates this contention with a read cache and bank alternation.

TRF reads have heavy reuse: the same data is typically broadcast across many cycles, so a direct-mapped read cache (8 lanes × 2 banks × 4 rows × 320 bits = 2.5 KB) sits in front of the banks and absorbs repeated reads. The cache also relieves contention with the concurrent store. On a hit the read skips the bank, so the store can use it that cycle. On a miss the cache refills from the bank, occupying it for that cycle.

For narrow reads (≤ 32 bytes), bank alternation adds a second mitigation. Reads use only one bank, so they can alternate at 32-byte granularity across the two banks. Reads and writes then end up on different banks on successive cycles, avoiding contention even on cache misses. Wide reads (64 bytes) occupy both banks every cycle, so each cache miss blocks the concurrent store; narrow reads preserve half-bandwidth alternation even when the cache misses.

Vector Register File

The VRF is written either from the Collect Engine or directly from Data Memory, and read by the Vector Engine.

Interface

A VrfTensor is a tensor stored in the VRF:

/// Tensor stored in the vector register file (VRF).
#[primitive(VrfTensor)]
#[derive(Debug, Clone)]
pub struct VrfTensor<D: VeScalar, Chip: M, Cluster: M, Slice: M, Element: M, B: Backend = CurrentBackend> {
    pub(crate) inner: Tensor<D, Pair<Chip, Pair<Cluster, Pair<Slice, Element>>>, B>,
    #[expect(dead_code)]
    address: Option<Address>,
    _marker: PhantomData<(D, Chip, Cluster, Slice, Element)>,
}

Chip / Cluster / Slice pass through from the source. Element holds the per-(slice) layout.

From Collect Engine

.to_vrf::<Element2>() on CollectTensor stores the flits into the VRF and produces a VrfTensor; .to_vrf_at::<Element2>(address) stores at a raw Address:

impl<'l, const T: Tu, P: CanApplyToVrf, D: VeScalar, Chip: M, Cluster: M, Slice: M, Time: M, Packet: M, B: Backend>
    TuTensor<'l, T, P, D, Chip, Cluster, Slice, Time, Packet, B>
{
    /// Stores to the vector register file.
    #[primitive(TuTensor::to_vrf)]
    pub fn to_vrf<Element: M>(self) -> VrfTensor<D, Chip, Cluster, Slice, Element, B> {
        VrfTensor::new(self.inner.transpose(false), None)
    }

    /// Stores to the vector register file at `address`.
    #[primitive(TuTensor::to_vrf_at)]
    pub fn to_vrf_at<Element: M>(self, address: Address) -> VrfTensor<D, Chip, Cluster, Slice, Element, B> {
        VrfTensor::new(self.inner.transpose(false), Some(address))
    }
}

.to_vrf flattens the streaming Time / Packet into Element2:

Element2 = [Time, Packet]

The user picks Element2.

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![B = 64];

fn store_vrf<'l, const T: Tu>(
    input: CollectTensor<'l, T, i32, m![1], m![1 # 2], m![1 # 256], m![B / 8], m![B % 8]>,
) -> VrfTensor<i32, m![1], m![1 # 2], m![1 # 256], m![B]> {
    input.to_vrf()
}

let mut ctx = Context::acquire();

let c: CollectTensor<'_, _, i32, m![1], m![1 # 2], m![1 # 256], m![B / 8], m![B % 8]> = CollectTensor::new(&mut ctx.main, Tensor::zero());
let _o = store_vrf(c);
}

From Data Memory

For completely contiguous input access (no gaps or reordering), VRF supports a short command (StoVRF), a compact hardware instruction that loads data from Data Memory directly into the VRF, bypassing the full Fetch → Switch → Collect → to_vrf() pipeline. The shortcut trades arbitrary-layout support for lower setup overhead.

Architecture

Contraction Engine

The Contraction Engine performs binary tensor contractions such as matmul and convolution. Recall from Quick Start:

• A tensor contraction takes two input tensors and reduces along their shared (contracted) axes. Dot product, GEMV, and GEMM are the canonical examples.
• A contraction decomposes into three steps: Broadcast, Multiply, Reduce.
• One operand streams from the Collect Engine, and the other sits in the TRF (Tensor Register File).
• Contraction runs in the main context. TRF preparation runs in the sub context via .to_trf().

Architecture

Four pipeline stages factor the workload: one for Broadcast and Multiply, three for Reduce. Each stage handles its own non-overlapping dimension.

%%{init: {'flowchart': {'htmlLabels': true}, 'themeCSS': '.cluster-label .nodeLabel { font-size: 16px; font-weight: 600; }'}}%%
flowchart TB
    CO[Collect Engine] --> SA
    TRF[(TRF)] --> TS

    subgraph CE[Contraction Engine]
        direction TB
        subgraph BC[Outer]
            direction LR
            SA[Stream Adapter]
            TS[TRF Sequencer]
            MUL[Elementwise Multiply]
            SA --> MUL
            TS --> MUL
        end
        SC[Packet Reducer]
        TR[Time Reducer]
        RR[Lane Folder]
        MUL --> SC
        SC --> TR
        TR --> RR
    end

    RR --> VE[Vector Engine]

    click SA "./outer.html#stream-adapter" "Stream Adapter"
    click TS "./outer.html#trf-sequencer" "TRF Sequencer"
    click SC "./packet-reducer.html" "Packet Reducer"
    click TR "./time-reducer.html" "Time Reducer"
    click RR "./lane-folder.html" "Lane Folder"
    click CO "../collect-engine.html" "Collect Engine"
    click TRF "../register-files.html#tensor-register-file" "Tensor Register File"
    click VE "../vector-engine/index.html" "Vector Engine"

• Outer (Broadcast and Multiply): broadcasts the two operands to a matching shape [Chip, Cluster, Slice, Lane, Time, Packet] and multiplies them elementwise into a single product tensor. Chip / Cluster / Slice pass through. Lane indexes the spatial parallelism shared by the TRF and downstream reducers. Time and Packet together represent packet streams. Three sub-stages run in series: the Stream Adapter broadcasts the streaming operand, the TRF Sequencer broadcasts the TRF operand, and the Multiplier widens to the contraction output type (i4/i8 -> i32, f8/bf16 -> f32) and multiplies them elementwise.
• Packet Reducer (Reduce within Packet): reduces along contracted axes mapped to Packet via a parallel tree, one tree per lane.
• Time Reducer (Reduce across Time): accumulates per-cycle results in the shared accumulator.
• Lane Folder (Fold Lane): emits the buffer to the output stream, absorbing Lane into either OutPacket or OutTime depending on the mode. For reductions across slices or chips, the Vector Engine handles the reduction downstream.

The Outer stage caps Lane ≤ 8 and Packet ≤ 64 B (on RNGD); see Packet Reducer and Time Reducer for more details.

For an end-to-end latency budget that stacks all four stages plus the Inter-Slice Reducer (e.g., 65,536 → 1 scalar in ~296 cycles), see Kernel Examples: Chip/Cluster Reduce.

Example: Batched MatMul

Quick Start walks through dot product, GEMV, and GEMM. Batched matmul extends GEMM with a leading batch axis V: \(VMK, KN \rightarrow VMN\). For each of V independent (M × K) inputs and a shared (K × N) weight, the kernel produces the (M × N) product.

The three variants below classify kernels by which axis sits in Time. The remaining axes are exploited as spatial parallelism. They share these axes:

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![V = 32, M = 32, N = 8, K = 32];   // V batch, M×N output, K contraction
}

(See 2D Convolution for another example, which uses a separate set of Stream Adapter machinery.)

K in Time

K (the contraction axis) sits in Time. M splits across Cluster and Slice, and V splits as well: V % 16 joins Slice while V / 16 = 2 joins Time alongside K (V × M = 1024 doesn’t fit the 512 spatial cells per chip on RNGD, so V’s outer chunk must iterate). Packet pads to 1 # 32 and the reduction proceeds sequentially across cycles instead of via the Packet Reducer’s spatial tree, so only 1 of 32 multipliers does useful work per cycle (1/32 MAC utilization for bf16). The result is a degenerate kernel, shown only as an educational baseline.

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![V = 32, M = 32, N = 8, K = 32];   // V batch, M×N output, K contraction

type Chip    = m![1];                   // single chip
type Cluster = m![M / 16];              // outer M split across clusters (M / 16 = 2)
type Slice   = m![M % 16, V % 16];      // inner M × inner V = 16 × 16 = 256 slices per cluster
type Lane    = m![N];                   // N (output channels) partitions the 8 hardware lanes

/// Batched matmul with K placed in Time.
fn bmatmul_k_in_time<'l, const T: Tu>(
    // Streaming operand: V outer + K in Time, with a one-element Packet m![1].
    input: CollectTensor<'l, T, bf16, Chip, Cluster, Slice, m![V / 16, K], m![1 # 16]>,
    // TRF operand: N in Lane, K in Element. Stored into TRF by a prior .to_trf() call.
    trf: &TrfTensor<bf16, Chip, Cluster, Slice, Lane, m![K]>,
    // Output: one (M × N) f32 matrix per (slice, V-outer) pair.
) -> ContractTensor<'l, T, f32, Chip, Cluster, Slice, m![V / 16], m![N]> {
    input
         // Outer: Lane = m![N] (inferred from trf), OutTime = m![V / 16, K], OutPacket = m![1 # 32].
         // input: 1 K-element broadcast across all N lanes.
         // trf:   1 K-element per lane, advancing one K-step per cycle.
         .contract_outer::<m![V / 16, K], m![1 # 16], _, _, _>(trf)
         // Packet Reducer: OutPacket = m![1]. Nothing to reduce.
         .contract_packet::<m![1]>()
         // Time Reducer: OutTime = m![V / 16]. K iterates over Time and accumulates; V outer survives.
         .contract_time::<m![V / 16]>()
         // Lane Folder: Lane folds into OutPacket. Interleaved mode emits 8 lanes per cycle.
         .contract_lane::<m![V / 16], m![N]>(LaneMode::Interleaved)
}

let mut ctx = Context::acquire();

let a: CollectTensor<'_, _, bf16, Chip, Cluster, Slice, m![V / 16, K], m![1 # 16]> = CollectTensor::new(&mut ctx.main, Tensor::zero());
let b: TrfTensor<bf16, Chip, Cluster, Slice, Lane, m![K]> = unsafe { TrfTensor::from_addr(TrfAddress::Full) };
let _o = bmatmul_k_in_time(a, &b);
}

To avoid this pathological case, keep K in Packet (parallel reduction via the Packet Reducer’s tree) and spread the surviving axes (V, M, N) across Cluster, Slice, and Lane to maximize spatial parallelism. The two strategies below apply this principle, each with one axis per class for simplicity; real kernels may split a single axis across multiple classes when sizes demand.

M in Time

V (batch) distributes across Cluster and Slice (one batch element per slice). M in Time, K in Packet. This strategy is applicable when (1) the slice count covers the batch, (2) N fits in Lane, and (3) K fits in a single Packet. When K is larger than Packet, split K across Packet (spatial) and Time (temporal). It maximizes MAC utilization across lanes.

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![V = 32, M = 32, N = 8, K = 32];   // V batch, M×N output, K contraction

type Chip    = m![1];                   // single chip
type Cluster = m![V / 16];              // outer V split across clusters (V / 16 = 2)
type Slice   = m![V % 16 # 256];        // inner V split across slices (V % 16 = 16 per cluster)
type Lane    = m![N];                   // N (output channels) partitions the 8 hardware lanes (N = 8 fills the cap)

/// Batched matmul: V slices × (M × K) · (K × N) → V × M × N.
fn bmatmul_m_in_time<'l, const T: Tu>(
    // Streaming operand: M in Time, K in Packet.
    // Element type can be i4, i8, f8, or bf16; integers widen to i32 output, floats to f32.
    input: CollectTensor<'l, T, bf16, Chip, Cluster, Slice, m![M, K / 16], m![K % 16]>,
    // TRF operand: N in Lane (one output channel per lane), K in Element.
    // Stored into TRF by a prior .to_trf() call in the sub context.
    trf: &TrfTensor<bf16, Chip, Cluster, Slice, Lane, m![K]>,
    // Output: one (M × N) f32 matrix per slice.
) -> ContractTensor<'l, T, f32, Chip, Cluster, Slice, m![M], m![N]> {
    input
         // Outer: broadcast input and trf, multiply elementwise.
         // Lane = m![N] (inferred from trf), OutTime = m![M], OutPacket = m![K].
         // input: K elements broadcast across all N lanes.
         // trf:   K elements per lane, broadcast across all M cycles.
         .contract_outer::<m![M], m![K], _, _, _>(trf)
         // Packet Reducer: OutPacket = m![1]. Sum K spatially via the reduction tree.
         .contract_packet::<m![1]>()
         // Time Reducer: OutTime = m![M]. Nothing to reduce.
         .contract_time::<m![M]>()
         // Lane Folder: Lane folds into OutPacket. Interleaved mode emits 8 lanes per cycle.
         .contract_lane::<m![M], m![N]>(LaneMode::Interleaved)
}

let mut ctx = Context::acquire();

let a: CollectTensor<'_, _, bf16, Chip, Cluster, Slice, m![M, K / 16], m![K % 16]> = CollectTensor::new(&mut ctx.main, Tensor::zero());
let b: TrfTensor<bf16, Chip, Cluster, Slice, Lane, m![K]> = unsafe { TrfTensor::from_addr(TrfAddress::Full) };
let _o = bmatmul_m_in_time(a, &b);
}

V in Time

V (batch) in Time, K in Packet. M splits across Cluster and Slice. This strategy is applicable when (1) the slice count covers M (M / 16 in Cluster, M % 16 in Slice), (2) N fits in Lane, and (3) K fits in a single Packet. Useful when batch is the dominant axis (e.g., batched inference).

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![V = 32, M = 32, N = 8, K = 32];   // V batch, M×N output, K contraction

type Chip    = m![1];                   // single chip
type Cluster = m![M / 16];              // outer M split across clusters (M / 16 = 2)
type Slice   = m![M % 16 # 256];        // inner M split across slices (M % 16 = 16 per cluster)
type Lane    = m![N];                   // N (output channels) partitions the 8 hardware lanes

/// Batched matmul with V (batch) placed in Time.
fn bmatmul_v_in_time<'l, const T: Tu>(
    // Streaming operand: V in Time, K in Packet.
    input: CollectTensor<'l, T, bf16, Chip, Cluster, Slice, m![V, K / 16], m![K % 16]>,
    // TRF operand: N in Lane, K in Element. Stored into TRF by a prior .to_trf() call.
    trf: &TrfTensor<bf16, Chip, Cluster, Slice, Lane, m![K]>,
    // Output: one (V × N) f32 matrix per slice.
) -> ContractTensor<'l, T, f32, Chip, Cluster, Slice, m![V], m![N]> {
    input
         // Outer: Lane = m![N] (inferred from trf), OutTime = m![V], OutPacket = m![K].
         // input: K elements broadcast across all N lanes.
         // trf:   K elements per lane, broadcast across all V cycles.
         .contract_outer::<m![V], m![K], _, _, _>(trf)
         // Packet Reducer: OutPacket = m![1]. Sum K spatially via the reduction tree.
         .contract_packet::<m![1]>()
         // Time Reducer: OutTime = m![V]. Nothing to reduce.
         .contract_time::<m![V]>()
         // Lane Folder: Lane folds into OutPacket. Interleaved mode emits 8 lanes per cycle.
         .contract_lane::<m![V], m![N]>(LaneMode::Interleaved)
}

let mut ctx = Context::acquire();

let a: CollectTensor<'_, _, bf16, Chip, Cluster, Slice, m![V, K / 16], m![K % 16]> = CollectTensor::new(&mut ctx.main, Tensor::zero());
let b: TrfTensor<bf16, Chip, Cluster, Slice, Lane, m![K]> = unsafe { TrfTensor::from_addr(TrfAddress::Full) };
let _o = bmatmul_v_in_time(a, &b);
}

Outer

The Outer stage broadcasts the two operands into a matching shape and multiplies them elementwise.

“Outer” comes from the outer product of linear algebra. For vectors u (length n) and v (length m), u v^T is the n × m matrix where (u v^T)[i, j] = u[i] × v[j]. That matrix is produced by broadcasting u along the column axis (length m), broadcasting v along the row axis (length n), and multiplying elementwise. The Outer stage’s three sub-stages are the hardware embodiment of this exact semantics, run in series:

• The Stream Adapter handles (and broadcasts) the streaming operand from the Collect Engine.
• The TRF Sequencer handles (and broadcasts) the TRF operand from TRF SRAM.
• The Multiplier widens the operand types (i4 / i8 to i32, f8 / bf16 to f32) and multiplies the two aligned operands elementwise.

The output is a single multiplied tensor in the joint mapping [Chip, Cluster, Slice, Lane, Time, Packet], ready for the Packet Reducer to reduce-add.

Interface

.contract_outer(&trf) on CollectTensor invokes the Outer stage.

impl<
    'l,
    const T: Tu,
    P: CanApplyContractOuter,
    D: Scalar + ContractionCast,
    Chip: M,
    Cluster: M,
    Slice: M,
    Time: M,
    Packet: M,
    B: Backend,
> TuTensor<'l, T, P, D, Chip, Cluster, Slice, Time, Packet, B>
{
    /// Runs the Outer stage: stashes the two un-broadcast operands (widened to the accumulator type)
    /// and the layouts [`super::lane::contract_lane`] needs to fuse them into a [`LazyContraction`]. No
    /// materializing alternative, no per-backend branch -- every backend fuses the same way.
    #[primitive(TuTensor::contract_outer)]
    pub fn contract_outer<OutTime: M, OutPacket: M, Lane: M, TrfElement: M, TrfD>(
        self,
        trf_tensor: &TrfTensor<TrfD, Chip, Cluster, Slice, Lane, TrfElement, B>,
    ) -> ContractOuterTensor<'l, T, <D as ContractionCast>::Output, D, Chip, Cluster, Slice, Lane, OutTime, OutPacket, B>
    where
        D: Cast<<D as ContractionCast>::Output>,
        // The weight (TRF) type must form a valid contraction-engine operand pair with the
        // stream type `D`: same type, or a mixed integer precision within a
        // family (i4/i5 x i4/i5, i8/i9 x i8/i9). Both operands widen to the
        // stream's accumulator for the multiply.
        TrfD: Scalar + ContractionWeight<D> + Cast<<D as ContractionCast>::Output>,
    {
        type Out<D> = <D as ContractionCast>::Output;

        // Skipping the broadcast transpose does not skip its validity contract -- a malformed
        // contraction would otherwise slip past these asserts and panic far downstream instead.
        stream_adapter::verify_stream_adapter::<D, Lane, Time, Packet, OutTime, OutPacket>();
        trf_sequencer::verify_trf_sequencer::<TrfD, Lane, TrfElement, OutTime, OutPacket>();

        // The operands keep their own compact layouts: lhs is `[Chip, Cluster, Slice, Time, Packet]`
        // (`self.inner`), rhs is `[Chip, Cluster, Slice, Lane, TrfElement]` (`trf_tensor`). A
        // bare-buffer backend reads its strides from these; `MathStorage` ignores them (its axes live
        // in the storage).
        let lhs_map = <m![{ Chip }, { Cluster }, { Slice }, { Time }, { Packet }]>::to_value();
        let rhs_map = <m![{ Chip }, { Cluster }, { Slice }, { Lane }, { TrfElement }]>::to_value();
        let pre_reduce = <m![{ Chip }, { Cluster }, { Slice }, { Lane }, { OutTime }, { OutPacket }]>::to_value();

        // Widen each operand to the `Out<D>` accumulator up front, at the operand's own (compact) size,
        // not pre_reduce's -- the fold at `contract_lane` then runs entirely in `Out<D>`. Parity-identical
        // to the old per-cell widen (same `Cast::cast` per element), it just never allocates the
        // pre_reduce-shaped broadcast this stage used to build.
        //
        // `map_bounded`, NOT the plain `map`: `D` (the stream side) and `TrfD` (the TRF side) may each
        // legitimately be a non-`MaterializableScalar` staging type (`i5`/`i9`, produced by
        // `fetch_zero_point_sub` -- "an i5/i9 may still be a contraction weight resident in the TRF" per
        // its own doc). Such a type's storage-native length recovery over-reports (its `BITS` names a
        // real-hardware wire width, disconnected from its host in-memory size), so a plain `map`'s
        // internal whole-buffer walk reads/writes past the buffer once the count crosses the true
        // element count -- this is the ONE place either operand is read back as a whole tensor before
        // `contract_lane`'s fused `Backend::contraction` (its only other lifetime is written-once by
        // `fetch_zero_point_sub`, never read back that way), so it is also the one place this matters.
        let contraction = LazyContraction {
            lhs: self.inner.map_bounded(|v| -> Out<D> { v.cast() }).inner,
            rhs: trf_tensor.inner.map_bounded(|v| -> Out<D> { v.cast() }).inner,
            lhs_map,
            rhs_map,
            pre_reduce,
        };
        ContractOuterTensor::new(self.ctx, contraction)
    }
}

After their respective adapters (Stream Adapter for the streaming path, TRF Sequencer for the TRF path), both paths feed Lane / Time / Packet of matching shape, and the Multiplier multiplies them elementwise at aligned positions.

The streaming operand’s Time / Packet map to the output’s OutTime / OutPacket: OutPacket absorbs the innermost size-1-or-2 factor of Time via Packing. OutTime retains the remaining factors of Time, with broadcast factors added at its innermost positions via Broadcast. Broadcast factors come from the TRF operand’s Lane / Element (where the streaming operand replicates against the TRF mapping) and from any purely-output axes that appear in OutTime / OutPacket but in neither the input nor the TRF (e.g. einsum AB, BC -> ABCD where D is broadcast).

The TrfTensor has shape [Chip, Cluster, Slice, Lane, Element], with Chip / Cluster / Slice / Lane spatially parallel: Chip / Cluster / Slice pass through to the output, and Lane partitions per-lane data across 1–8 hardware lanes. Element (the per-lane layout set by .to_trf()) is reshaped by the TRF Sequencer to fill OutTime / OutPacket.

Stream Adapter

The Stream Adapter transforms the streaming Time / Packet into the computation shape (Lane / OutTime / OutPacket) via two operations: Packing and Broadcast. The compiler derives the three free variables (PackSize, LaneBroadcast, TimeBroadcast) from the user-supplied OutTime / OutPacket, the TRF operand, and any purely-output broadcast axes, giving the mapping:

Lane      = LaneBroadcast
OutTime   = [Time / PackSize, TimeBroadcast]
OutPacket = [Time % PackSize, Packet] # (64 / D::SIZE)

Packing

The Collect Engine produces 32 B flits, and the Outer stage emits packets of PackSize × 32 B (32 or 64 on RNGD). Packing combines PackSize ∈ {1, 2} consecutive flits into one packet:

PackTime   = [Time / PackSize]
PackPacket = [Time % PackSize, Packet] # (PackSize × 32 / D::SIZE)

PackSize is set by matching OutPacket against the input Packet: PackSize = 2 if OutPacket absorbs the innermost size-2 factor of Time, otherwise PackSize = 1. Equivalently, PackSize = OutPacket::SIZE * D::SIZE / 32, so the user picks OutPacket (32 B or 64 B) and Packing’s collect-flit count follows.

Hardware always operates on 64 B packets internally; when PackSize = 1, the unused 32 B half holds zeros that do not propagate into the logical OutPacket type. Downstream stages (Packet Reducer, Lane Folder) therefore see only the PackSize × 32 B payload, avoiding dummy cycles, see the Lane Folder Sequential note.

Broadcast

After packing, the Stream Adapter broadcasts the data spatially via LaneBroadcast (the TRF’s Lane mapping, ∈ {1, 2, 4, 8}) and temporally via TimeBroadcast. TimeBroadcast covers factors of TRF Element not in the input Time, and also any purely-output axes in OutTime that appear in neither the input nor the TRF: the same broadcast machinery replicates the packet across both. Each destination receives the same OutPacket:

Lane      = LaneBroadcast
OutTime   = [PackTime, TimeBroadcast]
OutPacket = PackPacket

TimeBroadcast factors occupy the innermost positions of OutTime: the same OutPacket is re-sent across those factors before iterating any outer PackTime factor.

Examples

The example below exercises both operations: Packing absorbs the innermost size-2 factor L of Time into Packet (PackSize = 2), Lane Broadcast distributes the resulting packet to N = 8 lanes, and Time Broadcast tiles the streaming data across a TRF-only B = 5 axis.

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![M = 32, N = 8, K = 16, L = 2, B = 5];

fn stream_adapter_example<'l, const T: Tu>(
    input: CollectTensor<'l, { T }, bf16, m![1], m![1 # 2], m![1 # 256], m![M, L], m![K]>,
    trf: &TrfTensor<bf16, m![1], m![1 # 2], m![1 # 256], m![N], m![B, L, K]>,
) -> ContractOuterTensor<'l, { T }, f32, bf16, m![1], m![1 # 2], m![1 # 256], m![N], m![M, B], m![L, K]> {
    // Packing (PackSize = 2):
    //   L = 2 (innermost Time) absorbed into Packet.
    //   PackTime = [M = 32], PackPacket = [L = 2, K = 16] = 32 bf16 = 64B.
    // Lane Broadcast: same packet to all N = 8 lanes.
    // Time Broadcast: B = 5 (TRF-only) added at innermost OutTime.
    //   OutTime = [M, B = 5], OutPacket = [L = 2, K = 16].
    input.contract_outer::<m![M, B], m![L, K], _, _, _>(trf)
}

let mut ctx = Context::acquire();

let a: CollectTensor<'_, _, bf16, m![1], m![1 # 2], m![1 # 256], m![M, L], m![K]> = CollectTensor::new(&mut ctx.main, Tensor::zero());
let b: TrfTensor<bf16, m![1], m![1 # 2], m![1 # 256], m![N], m![B, L, K]> = unsafe { TrfTensor::from_addr(TrfAddress::Full) };
let _o = stream_adapter_example(a, &b);
}

Constraints

• OutPacket::SIZE * Storage::SIZE ∈ {32, 64} bytes (on RNGD), where Storage is the pre-widen operand dtype (e.g. bf16 = 2 B, not the widened f32 accumulator the result tensor carries): 32 for PackSize = 1, 64 for PackSize = 2. The user picks this size and Packing’s collect-flit count follows.
• PackSize ∈ {1, 2} (see Packing).
• Lane::SIZE ∈ {1, 2, 4, 8}.

Performance

PackSize sets MAC utilization. PackSize = 2 fills the full 64 B and uses all MACs. PackSize = 1 fills only 32 B, so the zero-padded half always multiplies by zero and effective throughput halves.

PackSize = 2 takes 2 cycles per Packet (two 32 B flits combine into one 64 B Packet), but this is not a pipeline bottleneck: upstream supplies one 32 B flit every cycle at the full fetch rate, so the Stream Adapter consumes flits as fast as they arrive and emits a Packet every 2 cycles to match downstream consumption.

Time Broadcasting amortizes fetches. Broadcast factors reuse the same streaming packet across cycles without re-fetching, which eliminates bandwidth cost for those factors.

Fetch bandwidth (up to 32 B/cycle per fetch) bounds the Stream Adapter overall. Interleave fetch patterns across slices to maximize utilization.

TRF Sequencer

The TRF Sequencer reads a TrfTensor and reshapes its Element into OutTime / OutPacket for the Packet Reducer. See Register Files for the TRF storage layout (lanes, banks, rows, double-buffering, cache).

The mapping:

OutTime   = (sequencing over [Element / ReadSize] with broadcasts)
OutPacket = [PacketBroadcast, Element % ReadSize]

OutPacket is filled each cycle by one TRF read: one full OutPacket (640 bits per lane across both banks, or 320 bits when only one bank is read) is generated every cycle. See To Contraction Engine for the per-slice totals across 8 lanes (in bytes). The read pulls the innermost contiguous portion of Element and replicates it to fill the 64 B OutPacket. The compiler picks the largest ReadSize such that Element % ReadSize == OutPacket % ReadSize and ReadSize * D::SIZE ≤ 64 bytes: a wider ReadSize spans both TRF banks per lane, a narrower one uses just one bank.

OutTime is filled across cycles by sequencing Element / ReadSize (plus optional broadcasts). The TRF Sequencer uses the same nested-loop configuration as all other sequencers.

Examples

In this example, ReadSize covers all of Element in one 64 B read, so Element / ReadSize is trivial and the sequencer iterates only broadcasts:

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![M = 32, N = 8, K = 32];

fn trf_sequencer_full_read<'l, const T: Tu>(
    input: CollectTensor<'l, T, bf16, m![1], m![1 # 2], m![1 # 256], m![M, K / 16], m![K % 16]>,
    trf: &TrfTensor<bf16, m![1], m![1 # 2], m![1 # 256], m![N], m![K]>,
) -> ContractOuterTensor<'l, T, f32, bf16, m![1], m![1 # 2], m![1 # 256], m![N], m![M], m![K]> {
    // Element         = K
    // ReadSize        = 32
    // PacketBroadcast = 1
    // OutTime         = M      (sequencing over [K / 32] (= 1), broadcast M)
    // OutPacket       = K      (= [1, K % 32])
    input.contract_outer::<m![M], m![K], _, _, _>(trf)
}

let mut ctx = Context::acquire();

let a: CollectTensor<'_, _, bf16, m![1], m![1 # 2], m![1 # 256], m![M, K / 16], m![K % 16]> = CollectTensor::new(&mut ctx.main, Tensor::zero());
let b: TrfTensor<bf16, m![1], m![1 # 2], m![1 # 256], m![N], m![K]> = unsafe { TrfTensor::from_addr(TrfAddress::Full) };
let _o = trf_sequencer_full_read(a, &b);
}

In this example, ReadSize covers only part of Element, so Element / ReadSize is non-trivial and the sequencer iterates the outer Element factor alongside a broadcast:

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![M = 32, N = 8, K = 16, L = 2, O = 2];

fn trf_sequencer_partial_read<'l, const T: Tu>(
    input: CollectTensor<'l, T, bf16, m![1], m![1 # 2], m![1 # 256], m![O, M, L], m![K]>,
    trf: &TrfTensor<bf16, m![1], m![1 # 2], m![1 # 256], m![N], m![O, K]>,
) -> ContractOuterTensor<'l, T, f32, bf16, m![1], m![1 # 2], m![1 # 256], m![N], m![O, M], m![L, K]> {
    // Element         = [O, K]
    // ReadSize        = 16
    // PacketBroadcast = L
    // OutTime         = [O, M]    (sequencing over [O, K] / 16 (= O), broadcast M)
    // OutPacket       = [L, K]    (= [L, [O, K] % 16])
    input.contract_outer::<m![O, M], m![L, K], _, _, _>(trf)
}

let mut ctx = Context::acquire();

let a: CollectTensor<'_, _, bf16, m![1], m![1 # 2], m![1 # 256], m![O, M, L], m![K]> = CollectTensor::new(&mut ctx.main, Tensor::zero());
let b: TrfTensor<bf16, m![1], m![1 # 2], m![1 # 256], m![N], m![O, K]> = unsafe { TrfTensor::from_addr(TrfAddress::Full) };
let _o = trf_sequencer_partial_read(a, &b);
}

Constraints

• Hardware dimensions: Chip::SIZE, Cluster::SIZE, and Slice::SIZE must match the hardware configuration (see Sequencer).
• Address alignment: when Element % ReadSize covers all 64 B, the read spans both TRF banks per lane, so the sequencer’s base address and all strides must align to 64 B.

Architecture

Across cycles, the sequencer iterates the outer factors of Element (i.e. Element / ReadSize) using the same nested-loop configuration as all other sequencers, so a single TrfTensor walks Element / ReadSize cycles before exhausting its content. PacketBroadcast factors replicate the same row within a single cycle, filling the 64 B OutPacket past the natural ReadSize without consuming additional TRF read bandwidth.

Performance

Throughput is one full OutPacket per lane per cycle: 640 bits per lane when both banks are read, 320 bits per lane when only one bank is read. See Register Files: To Contraction Engine for the per-slice byte totals. The TRF read cache and bank alternation (see Register Files: Double Buffering) keep the concurrent sub-context store unblocked across broadcast reuse and narrow reads.

Multiplier

The Multiplier consumes the two aligned operands from the Stream Adapter and TRF Sequencer, widens each input element to the contraction output type (i4/i8 -> i32, f8/bf16 -> f32) to keep the downstream accumulator from overflowing, and multiplies them elementwise. Its output, a single tensor in the joint mapping [Chip, Cluster, Slice, Lane, Time, Packet], becomes the input to the Packet Reducer. Each Time cycle, every Lane produces a full packet of products in parallel.

Packet Reducer

The Packet Reducer reduce-adds the innermost contracted axes within a single Packet, with one reduction tree per lane.

Interface

.contract_packet() invokes the Packet Reducer. Each lane receives a 32 B or 64 B Packet of i4, i8, f8, or bf16 elements, inherited from the Outer stage’s OutPacket. Formally, it computes \(\text{output}[i] = \sum_{j} \text{input}[i, j]\), where i ranges over the surviving (output) axes and j ranges over the contracted axes inside Packet.

impl<
    'l,
    const T: Tu,
    D: Scalar,
    Storage: ContractionCast<Output = D>,
    Chip: M,
    Cluster: M,
    Slice: M,
    Lane: M,
    Time: M,
    Packet: M,
    B: Backend,
> ContractOuterTensor<'l, T, D, Storage, Chip, Cluster, Slice, Lane, Time, Packet, B>
{
    /// Spatial reduction within `Packet`: validates the reduce-add along the contracted axes inside
    /// `Packet` that the fused fold at `contract_lane` will perform. `D` is the widened accumulator the
    /// deferred carrier stays keyed on; the DPE input packet is still sized in `Storage` bytes.
    #[primitive(ContractOuterTensor::contract_packet)]
    pub fn contract_packet<OutPacket: M>(
        self,
    ) -> ContractPacketTensor<'l, T, D, Chip, Cluster, Slice, Lane, Time, OutPacket, B> {
        verify_contract_packet::<Storage, Packet, OutPacket>();
        // Carry the deferred operands forward unreduced: the fused contraction at `contract_lane`
        // performs this Packet reduction too. This stage only re-types the carrier to `OutPacket`.
        ContractPacketTensor::new(self.ctx, self.inner)
    }
}

The kernel below uses all 8 lanes in parallel: tree depth 5 sums over the 32 bf16 elements of B, producing one f32 per A position.

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 32, B = 32, C = 8];

fn matmul<'l, const T: Tu>(
    input: CollectTensor<'l, T, bf16, m![1], m![1 # 2], m![1 # 256], m![A, B / 16], m![B % 16]>,
    trf: &TrfTensor<bf16, m![1], m![1 # 2], m![1 # 256], m![C], m![B]>,
) -> ContractTensor<'l, T, f32, m![1], m![1 # 2], m![1 # 256], m![A], m![C]> {
    //
    // Spatial reduction: tree depth 5 reduces 32 bf16 elements along B → f32
    // Output (Interleaved): Time = [A], Packet = [C]
    input.contract_outer::<m![A], m![B], _, _, _>(&trf)
         .contract_packet::<m![1]>()
         .contract_time::<m![A]>()
         .contract_lane::<m![A], m![C]>(LaneMode::Interleaved)
}

let mut ctx = Context::acquire();

let a: CollectTensor<'_, _, bf16, m![1], m![1 # 2], m![1 # 256], m![A, B / 16], m![B % 16]> = CollectTensor::new(&mut ctx.main, Tensor::zero());
let b: TrfTensor<bf16, m![1], m![1 # 2], m![1 # 256], m![C], m![B]> = unsafe { TrfTensor::from_addr(TrfAddress::Full) };
let _o = matmul(a, &b);
}

Architecture

ReducePacket = Packet / 2^d        for 0 ≤ d ≤ log2(Packet::SIZE)
OutPacket    = ReducePacket        if ReducePacket::SIZE ≤ 32
               ReducePacket = 32   otherwise

The Packet Reducer first runs an independent reduction tree per lane on its input Packet. At depth 0, the tree leaves hold the input Packet’s elements, and each subsequent depth sums pairs, halving the element count. The maximum tree depth is log2(Packet::SIZE), so 7 for i4 (Packet::SIZE = 128), 6 for i8 / f8 (64), 5 for bf16 (32). Given the user’s OutPacket, the compiler derives the tree depth d, and the tree consumes the innermost 2^d elements to produce ReducePacket.

The Packet Reducer then trims ReducePacket to OutPacket, whose size is capped at 32 elements because the downstream Time Reducer’s per-lane accumulator only has 32 columns. When ReducePacket::SIZE > 32, the outer dummy is trimmed and only the innermost 32 elements survive. For example, i4 arrives as a 128-element Packet, so d ∈ {0, 1} produce a 128- or 64-element ReducePacket, both trimmed to OutPacket::SIZE = 32.

Performance

Latency depends on tree depth: 7 cycles for i4, 6 for i8/f8, 5 for bf16. Wider element types reduce depth because fewer elements fit in each Packet. The adder tree is fully pipelined, so depth adds first-output latency but does not reduce steady-state throughput: one Packet enters and one reduced output emerges every cycle once the pipeline is filled.

When Lane < 8, the inactive lanes’ reduction trees are idle, so per-cycle throughput drops proportionally to Lane::SIZE / 8.

Time Reducer

The Time Reducer accumulates the Packet Reducer’s [Lane, Packet] output across Time into OutTime, with temporal accumulators.

Interface

.contract_time::<OutTime>() invokes the Time Reducer. OutTime names the Time dimensions that survive (the rest are summed away).

impl<'l, const T: Tu, D: Scalar, Chip: M, Cluster: M, Slice: M, Lane: M, Time: M, Packet: M, B: Backend>
    ContractPacketTensor<'l, T, D, Chip, Cluster, Slice, Lane, Time, Packet, B>
{
    /// Accumulates per-cycle contractions over the `Time` dimension via the shared
    /// accumulator buffer, shrinking input `Time` to `OutTime`. The axes present in
    /// `Time` but absent from `OutTime` are reduce-added.
    #[primitive(ContractPacketTensor::contract_time)]
    pub fn contract_time<OutTime: M>(
        self,
    ) -> ContractTimeTensor<'l, T, D, Chip, Cluster, Slice, Lane, OutTime, Packet, B> {
        verify_contract_time(Time::to_value(), OutTime::to_value());
        // Carry the deferred operands forward unreduced: the fused contraction at `contract_lane`
        // performs this Time reduction too. This stage only re-types the carrier to `OutTime`.
        ContractTimeTensor::new(self.ctx, self.inner, Time::to_value())
    }
}

For example, the kernel below reduces a 2D tensor along B (surviving only A).

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 2048, B = 32];

/// Reduces along B; A survives.
fn reduce_b<'l, const T: Tu>(
    // Streaming operand: Slice = m![A / 8] (256 outer A chunks across slices).
    // Time = m![B / 16, A % 8]; Packet = m![B % 16].
    // B splits across Packet (B % 16) and Time (B / 16): each cycle produces a partial sum.
    input: CollectTensor<'l, T, bf16, m![1], m![1 # 2], m![A / 8], m![B / 16, A % 8], m![B % 16]>,
    // TRF operand: single-lane weight per slice.
    trf: &TrfTensor<bf16, m![1], m![1 # 2], m![A / 8], m![1], m![B]>,
    // Output: one f32 per (slice, A % 8) cell.
) -> ContractTensor<'l, T, f32, m![1], m![1 # 2], m![A / 8], m![A % 8], m![1 # 8]> {
    input
         // Outer: Lane = m![1], OutTime = m![B / 16, A % 8], OutPacket = m![B % 16].
         .contract_outer::<m![B / 16, A % 8], m![B % 16], _, _, _>(trf)
         // Packet Reducer: OutPacket = m![1]. Collapses B % 16 spatially.
         .contract_packet::<m![1]>()
         // Time Reducer: OutTime = m![A % 8]. Accumulator receives
         // Time::SIZE = (B / 16) × (A % 8) = 2 × 8 = 16 flits; B / 16 outer
         // chunks accumulate into 8 slots indexed by A % 8.
         .contract_time::<m![A % 8]>()
         // Lane Folder: Lane folds into OutPacket. Sequential mode (Lane = m![1]).
         .contract_lane::<m![A % 8], m![1 # 8]>(LaneMode::Sequential)
}

let mut ctx = Context::acquire();

let a: CollectTensor<'_, _, bf16, m![1], m![1 # 2], m![A / 8], m![B / 16, A % 8], m![B % 16]> = CollectTensor::new(&mut ctx.main, Tensor::zero());
let b: TrfTensor<bf16, m![1], m![1 # 2], m![A / 8], m![1], m![B]> = unsafe { TrfTensor::from_addr(TrfAddress::Full) };
let _o = reduce_b(a, &b);
}

Architecture

The Time Reducer receives the Packet Reducer’s per-cycle [Lane, Packet] output. The hardware caps Lane::SIZE ≤ 8 (spatially parallel lanes) and Packet::SIZE ≤ 32 upstream.

Each cycle the Time Reducer folds the [Lane, Packet] spatial grid across Time into OutTime. OutTime must be a subset of Time with the relative order of surviving dimensions preserved (enforced by verify_contract_time). Dimensions in Time absent from OutTime are summed away, and the outermost such dimension iterates over flits.

Let InnerTime denote the inner non-reduce dimensions of Time (the dimensions inner to the outermost reduce dimension that survive in OutTime). In reduce_b above, Time = m![B / 4, A % 8] and OutTime = m![A % 8], so B / 4 is the outermost reduce dimension (iterates over Time::SIZE = 2 × 8 = 16 flits) and InnerTime = m![A % 8].

Accumulation requires InnerTime::SIZE slots of [Lane, Packet], one per InnerTime tuple value. Flits with the same tuple accumulate into the same slot. For reduce_b, 8 slots accumulate across the B / 4 = 2 iterations, and after flit 15 the buffer contains the final reduced results and hands them off to the Lane Folder:

Time = m![B / 4, A % 8]
          ~~~~~  ~~~~~
        outer R  inner non-R (A % 8)

Flit sequence (B / 4 has values 0,1; A % 8 has values 0..7):

  flit #0:  B/4=0, A%8=0  ──→ ┌─────────────────┐
  flit #8:  B/4=1, A%8=0  ──→ │ slot 0 (A%8=0)  │  accumulates B for A%8=0
                              └─────────────────┘

  flit #1:  B/4=0, A%8=1  ──→ ┌─────────────────┐
  flit #9:  B/4=1, A%8=1  ──→ │ slot 1 (A%8=1)  │  accumulates B for A%8=1
                              └─────────────────┘
   ⋮

  flit #7:  B/4=0, A%8=7  ──→ ┌─────────────────┐
  flit #15: B/4=1, A%8=7  ──→ │ slot 7 (A%8=7)  │  accumulates B for A%8=7
                              └─────────────────┘

  8 non-reduce positions → 8 slots used

Constraints

The mapping fits the buffer when InnerTime::SIZE does not exceed the slot capacity (the number of slots the buffer holds).

The slot capacity follows from the buffer’s 1,024 cells and the downstream Lane Folder’s LaneMode. Each slot is a [Lane, Packet] chunk whose shape the LaneMode decides, so the number of slots is 1,024 divided by the cells per chunk:

For reduce_b under the downstream .contract_lane(LaneMode::Sequential) with Lane::SIZE = 1, the slot capacity is 32 and InnerTime::SIZE = 8 fits comfortably. If InnerTime::SIZE exceeded the slot capacity, you would restructure Time (e.g., split B further) or switch LaneMode to trade throughput for slot headroom.

Performance

Throughput is one packet per cycle on the input side. The effective output rate is 1 / N of the input after reducing N inputs into one output.

Latency for a Time-dimension reduction of size N is approximately N cycles.

Lane Folder

The Lane Folder is the Contraction Engine’s final stage. It eliminates the Lane dimension by relocating its 8 values into either OutPacket (Interleaved) or OutTime (Sequential). No values are summed: the stage folds Lane into another axis rather than reducing it.

Interface

.contract_lane(mode) invokes the Lane Folder. The stage drains the upstream Time Reducer’s buffer through an 8-element-wide output bus one cycle at a time, and LaneMode selects what each cycle’s flit carries.

impl<
    'l,
    const T: Tu,
    D: ContractionCast + MaterializableScalar,
    Chip: M,
    Cluster: M,
    Slice: M,
    Lane: M,
    Time: M,
    Packet: M,
    B: Backend,
> ContractTimeTensor<'l, T, D, Chip, Cluster, Slice, Lane, Time, Packet, B>
{
    /// Folds the `Lane` dimension into the output stream.
    /// `LaneMode::Interleaved` relocates `Lane` into `OutPacket`;
    /// `LaneMode::Sequential` relocates `Lane` into `OutTime`.
    #[primitive(ContractTimeTensor::contract_lane)]
    pub fn contract_lane<OutTime: M, OutPacket: M>(
        self,
        mode: LaneMode,
    ) -> ContractTensor<'l, T, D, Chip, Cluster, Slice, OutTime, OutPacket, B> {
        verify_contract_lane(
            Lane::to_value(),
            Time::to_value(),
            Packet::to_value(),
            OutTime::to_value(),
            OutPacket::to_value(),
            self.pre_reduce_time,
            mode,
        );
        // Finalize the carried operands with ONE fused contraction onto this stage's input mapping
        // `[Chip, Cluster, Slice, Lane, Time, Packet]` (Packet/Time were never actually reduced by the
        // earlier stages, only relabeled to their post-stage extents). The Lane fold relayout
        // (`transpose(false)`) then runs on the result, exactly as it always has.
        //
        // `out` is rebuilt here from this stage's own type params, independently of the `pre_reduce`
        // stashed by `contract_outer`; `Backend::contraction` reduces `pre_reduce` onto `out` via
        // `pre_reduce.carve(out)`, the same mapping-algebra carve `reduce` uses elsewhere, which is the
        // authority on whether `out` is a valid restriction of `pre_reduce` -- NOT a manual re-check here.
        // A naive per-symbol `.axes()` comparison is unsound for that: a contracted symbol can split
        // across a spatial slot this fold never touches (e.g. `Cluster`, carrying a `K`-fragment that
        // survives to `out` unreduced) and the slots that actually get contracted (`Time`/`Packet`,
        // carrying the rest of `K`); `pre_reduce`'s canonical `K` term then legitimately has a different
        // shape (wider modulo) than `out`'s, even though `out` is a correct restriction of `pre_reduce`.
        let contraction = self.inner;
        let out = <m![{ Chip }, { Cluster }, { Slice }, { Lane }, { Time }, { Packet }]>::to_value();
        let reduced: Tensor<D, m![{ Chip }, { Cluster }, { Slice }, { Lane }, { Time }, { Packet }], B> =
            Tensor::from_inner(B::contraction(
                &contraction.lhs,
                &contraction.rhs,
                &contraction.lhs_map,
                &contraction.rhs_map,
                &contraction.pre_reduce,
                &out,
            ));
        ContractTensor::new(self.ctx, reduced.transpose(false))
    }
}

The minimal examples below take a ContractTimeTensor (the output of the upstream Time Reducer) and call only .contract_lane(...), so each example shows the Lane Folder in isolation. The input Packet carries the size that survived the Packet Reducer, one of {1, 2, 4, 8, 16, 32} elements per lane.

Interleaved

The Lane dimension folds into OutPacket: each cycle reads one column position across all 8 lanes (one value per lane, 8 values per flit), with Lane materialized as the innermost OutPacket.

OutTime   = [Time, Packet]
OutPacket = [Lane # 8]

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![N = 8, M = 4, P = 16];

/// Lane folds into OutPacket.
fn lane_interleaved<'l, const T: Tu>(
    // Input from upstream Time Reducer: Lane = m![N], Time = m![M], Packet = m![P].
    input: ContractTimeTensor<'l, T, f32, m![1], m![1 # 2], m![1 # 256], m![N], m![M], m![P]>,
    // Output: OutTime = m![M, P] = [Time, Packet], OutPacket = m![N] = [Lane].
) -> ContractTensor<'l, T, f32, m![1], m![1 # 2], m![1 # 256], m![M, P], m![N]> {
    input.contract_lane::<m![M, P], m![N]>(LaneMode::Interleaved)
}

let mut ctx = Context::acquire();

let a: CollectTensor<'_, _, bf16, m![1], m![1 # 2], m![1 # 256], m![M], m![P]> = CollectTensor::new(&mut ctx.main, Tensor::zero());
let b: TrfTensor<bf16, m![1], m![1 # 2], m![1 # 256], m![N], m![P]> = unsafe { TrfTensor::from_addr(TrfAddress::Full) };

let i: ContractTimeTensor<'_, _, f32, m![1], m![1 # 2], m![1 # 256], m![N], m![M], m![P]> = a 
    .contract_outer::<m![M], m![P], m![N], m![P], _>(&b)
    .contract_packet::<m![P]>()
    .contract_time::<m![M]>();

let _o = lane_interleaved(i);
}

Sequential

The Lane dimension folds into OutTime: each cycle reads 8 column positions from one lane’s Packet (8 values per flit), with Lane iterating across successive cycles. Since each cycle is 8 elements wide, Packet is first padded up to a multiple of 8 (the bus width), then split into PadPacket / 8 cycles per lane and PadPacket % 8 elements per cycle.

PadPacket = Packet # align_up(Packet::SIZE, 8)   (pad Packet up to the next multiple of 8)
OutTime   = [Time, Lane, PadPacket / 8]
OutPacket = [PadPacket % 8]

For Packet::SIZE < 32, [PadPacket / 8]::SIZE = ceil(Packet::SIZE / 8) is the number of cycles per packet (e.g., 1 cycle for Packet::SIZE = 4, 2 cycles for Packet::SIZE = 16).

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![N = 8, M = 4, P = 16];

/// Lane folds into OutTime.
fn lane_sequential<'l, const T: Tu>(
    // Input from upstream Time Reducer: Lane = m![N], Time = m![M], Packet = m![P].
    input: ContractTimeTensor<'l, T, f32, m![1], m![1 # 2], m![1 # 256], m![N], m![M], m![P]>,
    // Output: OutTime = m![M, N, P / 8] = [Time, Lane, Packet / 8], OutPacket = m![P % 8] = [Packet % 8].
) -> ContractTensor<'l, T, f32, m![1], m![1 # 2], m![1 # 256], m![M, N, P / 8], m![P % 8]> {
    input.contract_lane::<m![M, N, P / 8], m![P % 8]>(LaneMode::Sequential)
}

let mut ctx = Context::acquire();

let a: CollectTensor<'_, _, bf16, m![1], m![1 # 2], m![1 # 256], m![M], m![P]> = CollectTensor::new(&mut ctx.main, Tensor::zero());
let b: TrfTensor<bf16, m![1], m![1 # 2], m![1 # 256], m![N], m![P]> = unsafe { TrfTensor::from_addr(TrfAddress::Full) };

let i: ContractTimeTensor<'_, _, f32, m![1], m![1 # 2], m![1 # 256], m![N], m![M], m![P]> = a 
    .contract_outer::<m![M], m![P], m![N], m![P], _>(&b)
    .contract_packet::<m![P]>()
    .contract_time::<m![M]>();

let _o = lane_sequential(i);
}

Constraints

The Lane Folder has no constraints of its own. The LaneMode selected here determines the slot capacity bound that the upstream Time Reducer enforces (see Time Reducer Constraints).

Performance

In Interleaved mode, throughput drops by Lane::SIZE / 8 when Lane < 8 (inactive lanes leave bus positions empty).

In Sequential mode, when Packet::SIZE < 8 (e.g., Packet::SIZE = 4 after the Packet Reducer collapses half of an bf16 packet), each cycle carries exactly Packet::SIZE elements rather than the full 8-element bus width: the output is narrower per cycle, but there is no padding and no wasted bus slots.

Latency is negligible: the Lane Folder reshapes per-lane outputs and does not add cycles beyond the buffer’s drain time.

2D Convolution

2D convolution is the einsum $(H + Fh)$(W + Fw)K, FhFwKC -> HWC with spatial output axes H, W, output channel C, and contraction axes Fh, Fw, K. The $(W + Fw) sliding maps to Stream Adapter shift-reuse (documented below).

The 4 variants in Variants differ in how the Stream Adapter shifts the sliding window (the conv-specific machinery). Choice of which axis to put in Time follows the same trade-off as in the matmul mapping discussion and is not repeated.

Variants

Filter-Stride 1

For stride-1 convolution, the Fetch Engine handles $(H+Fh) sliding before data reaches the Stream Adapter. The Stream Adapter then processes the $(W+Fw) dimension via shift-reuse to produce Fw, W sliding in the computation. The example below uses shift-stride 1 with two shifts.

// Configuration: input_type = bf16, trf_type = bf16
//        Input mapping: [ H: [H=30, Fh=3, K=32, $(W=30 + Fw=3)=32] ] (1)
//          TRF mapping: [ Lane: [C=8] | H: [K=32, C=24, Fh=3, Fw=3] ] (1)
//  Contraction mapping: [ H: [H=30, C/8=3, Fh=3, K=32, Fw=3] | Lane: [C=8] | T: [W=30+2#] ] (1)
// Accumulation mapping: [ H: [H=30, C=32] | T: [W=30+2#] ] (1)

Filter-Stride 2

For stride-2 convolution, a shift-stride of 2 (one shift per output position) extracts strided windows. The input expression $(W:2=15 + Fw=4)=32 factors into Fw/2=2, (W=15, Fw=2) by extracting a size-2 axis with stride :2 as an outer product: $(W:2=15 + (Fw/2:2=2, Fw=2))=32 becomes Fw/2=2, $(W:2=15, Fw=2).

// Configuration: input_type = bf16, trf_type = bf16
//        Input mapping: [ H: [H=15, Fh=4, K=32, $(W:2=15 + Fw=4)=32] ] (1)
//          TRF mapping: [ Lane: [C=8] | H: [K=32, C=24, Fh=4, Fw=4] ] (1)
//  Contraction mapping: [ H: [H=15, C/8=3, Fh=4, K=32, Fw/2=2] | Lane: [C=8] | T: [W=15+1#, Fw=2] ] (1)
// Accumulation mapping: [ H: [H=15, C=32] | T: [W=15+1#] ] (1)

The previous example underutilizes MACs because the shift buffer is not full. Setting feed_flits to 3 (from the default of 2) fills more flits in the shift buffer and achieves full MAC utilization. The transformation $(W:2=16 + Fw=4)=34 then produces Fw/2=2, (W=16, Fw=2).

// Configuration: feed_flits = 3, input_type = bf16, trf_type = bf16
//        Input mapping: [ H: [H=16, Fh=4, K=32, $(W:2=16 + Fw=4)=34] ] (1)
//          TRF mapping: [ Lane: [C=8] | H: [K=32, C=24, Fh=4, Fw=4] ] (1)
//  Contraction mapping: [ H: [H=16, C/8=3, Fh=4, K=32, Fw/2=2] | Lane: [C=8] | T: [W=16, Fw=2] ] (1)
// Accumulation mapping: [ H: [H=16, C=32] | T: [W=16] ] (1)

Dilation 2

For dilation-2 convolution, the filter samples input positions separated by a stride of 2. A shift-stride of 2 with 2 shifts extracts these dilated filter positions.

The transformation $(W=27 + Fw:2=3)=32 produces Fw=3, W=27, extracting a size-3 axis with stride :2 from a linear combination as an outer product: $(W=27 + Fw:2=3)=32 becomes Fw=3, $(W=27).

// Configuration: input_type = bf16, trf_type = bf16
//        Input mapping: [ H: [H=27, Fh=3, K=32, $(W=27 + Fw:2=3)=32] ] (1)
//          TRF mapping: [ Lane: [C=8] | H: [K=32, C=24, Fh=3, Fw=3] ] (1)
//  Contraction mapping: [ H: [H=27, C/8=3, Fh=3, K=32, Fw=3] | Lane: [C=8] | T: [W=27+5#] ] (1)
// Accumulation mapping: [ H: [H=27, C=32] | T: [W=27+5#] ] (1)

Filter-Stride 2, Dilation 2

Combining stride-2 and dilation-2 requires shift operations similar to dilation 2 alone.

The transformation $(W:2=14 + Fw:2=3)=31 + 1# produces Fw=3, W=14, 1+1#, extracting a size-3 axis with stride :2 from a linear combination as an outer product: $(W:2=14 + Fw:2=3)=31 becomes Fw=3, $(W:2=14).

The TRF must carry zeros in dummy slots so that 1+1# contracted with 1+1z yields 1, rather than the arbitrary 1# that would result from contracting 1+1# with 1+1#. The notation 1z is like 1# (dummy padding) but filled with zeros instead of arbitrary values.

// Configuration: input_type = bf16, trf_type = bf16
//        Input mapping: [ H: [H=14, Fh=3, K=32, $(W:2=14 + Fw:2=3)=31+1#] ] (1)
//          TRF mapping: [ Lane: [C=8] | H: [K=32, C=24, Fh=3, Fw=3, 1+1z] ] (1)
//  Contraction mapping: [ H: [H=14, C/8=3, Fh=3, K=32, Fw=3] | Lane: [C=8] | T: [W=14+2#, 1+1z] ] (1)
// Accumulation mapping: [ H: [H=14, C=32] | T: [W=14+2#] ] (1)

Stream Adapter Machinery for Convolutions

Convolution workloads need sliding-window data reuse to avoid refetching the same input elements. The Stream Adapter provides this reuse through three extensions: 3-flit collection, transpose, and shift-and-reuse.

Skip this section if you only work with einsum (matrix multiplication) workloads.

Flit Buffer: feed_flits 3

feed_flits: 3 extends the Flit Buffer beyond the default 2-flit capacity to fill all 96 bytes with three consecutive 32 B flits. The third flit gives the Stream Shift Unit enough buffered data to shift the window without refetching, at the cost of one extra flit’s worth of buffering per Packet.

See Shift for how the Stream Shift Unit consumes the extra flit.

Transpose

The Packet Reducer reduces adjacent pairs starting from the innermost axis, so the contracted axis must be innermost. Transpose reorders axes within a 32 B flit when the incoming data has a different axis order.

Supported Transposes

The supported transposes depend on data type (total volume is always 32 B).

These types are the ones the Contraction Engine can compute. Types like i32 and f32 cannot use the Packet Reducer’s reduction tree.

Example 1: Transpose within flit

axes![A = 3, B = 2, C = 2, D = 8];

// Input: time -> [1], num_flits -> [3_a], flit -> [2_b × 2_c × 8_d], i8
//
// Possible transpose outputs:
//   1. [2][16] → [16][2]: flit -> [2_c × 8_d × 2_b]
//   2. [4][8]  → [8][4]:  flit -> [8_d × 2_b × 2_c]
//
// From the Stream Shift Unit onward, num_flits × flit = flits:
//   1. flits -> [3_a × 2_c × 8_d × 2_b]
//   2. flits -> [3_a × 8_d × 2_b × 2_c]

Example 2: Transpose in Contraction Mapping

axes![P = 64, A = 2, B = 2, C = 16];

// Configuration: feed_flits = 2, datatype = bf16, transpose_flit(32B) = true
// Switch Engine output: [ P: [P_1=64] | H: [a_1=2, c_1=16, b_1=2] ] (16)
// Contraction mapping:       [ P: [P_1=64] | H: [a_1=2] | Lane: [] | T: [b_1=2, c_1=16] ] (16)

• c_1=16, b_1=2 becomes b_1=2, c_1=16, moving b to the innermost position for Packet Reducer reduction.
• With axes properly ordered, the Stream Adapter applies shift-and-reuse for sliding window operations.

Shift (Stream Shift Unit)

The Stream Shift Unit performs shift-and-reuse for sliding-window operations like convolutions. Rather than fetching overlapping windows multiple times, it fetches data once and shifts it to produce multiple windows.

Three parameters control shifting.

• initial_shift: starting offset when data is first loaded.
• shift_stride: amount to shift per iteration along the shift dimension.
• pop_dim: dimension that triggers fetching new data.

initial_shift

The initial_shift parameter sets the starting offset when data enters the shift buffer. The buffer holds elements in order from low address to high. Negative initial_shift shifts data toward later positions (higher addresses, upper bits), padding the early positions with zeros. Positive initial_shift shifts toward earlier positions (lower addresses, lower bits), padding the late positions.

Valid ranges

The distinct-value counts (32, 16, 8) correspond to half the number of elements in a 32 B flit, matching the Stream Shift Unit’s buffer capacity. The ranges are asymmetric (one more positive than negative) because they encode a power-of-two count of distinct positions with zero not at the exact midpoint.

After applying initial_shift, the output slices to 64 B for the Packet Reducer, with zeros filling any padding.

Example: negative initial_shift

axes![A = 96, B = 3];

// Input: time -> [3_b], flits -> [a], i8
// initial_shift = -7
//
// After initial shift: time -> [3_b], Lane -> [7_pad + a.slice(57)]
//   - Left portion is zero-padded
//   - After shifting by 7, only the lower 64 B are sliced and output
//   - For feed_flits = 1 or 2, portions beyond the actual fetched region are zero-padded

Example: positive initial_shift

axes![A = 96, B = 3];

// Input: time -> [3_b], flit -> [a], i8
// initial_shift = 8
//
// After initial shift: time -> [3_b], Lane -> [a.offset(8).slice(64)]
//   - 8-element shift + 64 B slicing
//   - For feed_flits ≠ 3, the upper 8 bytes are zero-padded

Per-index initial_shift

The initial_shift_dim parameter selects a different shift per index.

axes![A = 96, B = 3];

// Input: time -> [3_b], flits -> [a], i8
// seq_limits: [3, 1, 1, 1, 1, 1, 1, 1] (flits not shown in index)
// initial_shift_dim = 0 (b-axis)
// initial_shift_elements = [-7, 8, 0]
//
// at b = 0: Lane -> [7_pad + a.slice(57)]
// at b = 1: Lane -> [a.offset(8).slice(64)]
// at b = 2: Lane -> [a.slice(64)]

The initial_shift applies once when flits pop, and not during reuse/shift.

Example: negative initial_shift (contraction mapping)

axes![P = 64, A = 2, C = 32];

// Configuration: feed_flits = 2, datatype = bf16
//   init_shift = -1, init_shift_range = (-3, 4)
// Switch Engine output: [ P: [P_1=64] | H: [a_1=2, [[c_1=(1,31)]+1]=32] ] (16)
// Contraction mapping:       [ P: [P_1=64] | H: [a_1=2] | Lane: [] | T: [c_1=32] ] (16)

• With init_shift: -1, data shifts left by 1 element, adding 1 zero-padding element at the start.
• [[c_1=(1,31)]+1]: first element is padding, 31 elements are original data.

Example: positive initial_shift (contraction mapping)

axes![P = 64, A = 2, C = 32];

// Configuration: feed_flits = 2, datatype = bf16
//   init_shift = 1, init_shift_range = (-3, 4)
// Switch Engine output: [ P: [P_1=64] | H: [a_1=2, [1+[c_1=31]]=32] ] (16)
// Contraction mapping:       [ P: [P_1=64] | H: [a_1=2] | Lane: [] | T: [[[c_1=31]+1]=32] ] (16)

• With init_shift: 1, data shifts right by 1 element, dropping the first element and adding 1 zero-padding at the end.
• Switch Engine mapping [1+[c_1=31]]: first element is included.
• Contraction mapping [[c_1=31]+1]: 31 data elements followed by 1 padding element.

Example: per-index initial_shift using indirect vectors

axes![P = 64, A = 2, C = 32];

// Configuration: feed_flits = 2, datatype = bf16
//   init_shift_tag = a_1, init_shifts = [-1, 1], init_shift_range = (-3, 4)
//   indirect_vecs: [I0 = (c_1=32)[1, -1]]
// Switch Engine output: [ P: [P_1=64] | H: [a_1=2 @ I0_1, c_1=32] ] (16)
// Contraction mapping:       [ P: [P_1=64] | H: [a_1=2] | Lane: [] | T: [[[c_1=31]+1]=32] ] (16)

• With init_shift_tag: a_1 and init_shifts: [-1, 1], the shift varies per index.
• At a_1=0: shift by -1 (left). At a_1=1: shift by 1 (right).
• The indirect vector [1, -1] controls element selection from c_1=32 for each a_1 index.

Example: initial_shift with interleaved sliding window

axes![P = 64, A = 2, B = 31, C = 3];

// Configuration: feed_flits = 2, datatype = bf16
//   init_shift = -1, init_shift_range = (-3, 4), shift_stride = 1
// Switch Engine output: [ P: [P_1=64] | H: [a_1=2, $[(b_1=(1,30):1)+(c_1=3:1)]=32] ] (16)
// Contraction mapping:       [ P: [P_1=64] | H: [a_1=2, c_1=3] | Lane: [] | T: [[[b_1=31]+1]=32] ] (16)

• $[(b_1=(1,30):1)+(c_1=3:1)] is an interleaved pattern: b_1 starts at offset 1 with stride 1, and c_1 has 3 iterations with stride 1.
• Initial shift of -1 adds 1 padding element at the start, producing [[b_1=31]+1].

shift_stride and pop_dim

The pop_dim parameter marks when to fetch new data: when the index at pop_dim increments, fresh data loads and initial_shift reapplies. The shift_dim is the dimension along which reuse occurs: each iteration along shift_dim applies an additional shift of shift_stride elements.

Indices below pop_dim that are not shift_dim produce tiled (broadcast) outputs, giving the dimension ordering tile → shift_dim → pop_dim (inner to outer).

Valid shift_stride ranges

Example: shift_stride with pop_dim

axes![A = 96, B = 3];

// initial_shift = -1, shift_dim = 1, shift_stride = 3, pop_dim = 2
// seq_limits: [2, 3, 3, 1, 1, 1, 1, 1]
// Input: time -> [b = 3], flits -> [a], i8
//
// flits #0 (indexer: [0, 0, 0]): Apply initial shift.
//   [1_pad + a], slice to 64 → Lane -> [(1_pad + a) % 63]
// flits #1 (indexer: [1, 0, 0]): Same as #0 (dim0 is not shift_dim → tiling)
// flits #2 (indexer: [0, 1, 0]): shift_dim=1, apply shift_stride=3 from #0 state.
//   [a @ 2], slice to 64 → Lane -> [(a @ 2) % 64]
// flits #3 (indexer: [1, 1, 0]): Same as #2 (dim0 is not shift_dim → tiling)
// flits #4 (indexer: [0, 2, 0]): shift_dim=1, apply shift_stride=3 from #2 state.
//   [a @ 5], slice to 64 → Lane -> [(a @ 5) % 64]
// flits #5 (indexer: [1, 2, 0]): Same as #4 (dim0 is not shift_dim → tiling)
// flits #6 (indexer: [0, 0, 1]): pop_dim=2, fetch new flits and apply initial shift.
//   [1_pad + a], slice to 64 → Lane -> [(1_pad + a) % 63]
//
// Output mapping: time -> [b × (1_pad + a) / f=3:3 × Broadcast=2], Lane -> [(1_pad + a) / w=64:1]

Shift Examples

Shift with stride 1

axes![P = 64, A = 2, B = 31, C = 3];

// Configuration: feed_flits = 2, datatype = bf16
//   init_shift = -1, init_shift_range = (-3, 4), shift_stride = 1
// Switch Engine output: [ P: [P_1=64] | H: [a_1=2, $[(b_1=(1,30):1)+(c_1=3:1)]=32] ] (16)
// Contraction mapping:       [ P: [P_1=64] | H: [a_1=2, c_1=3] | Lane: [] | T: [[[b_1=31]+1]=32] ] (16)

Shift with stride 2

axes![P = 64, A = 2, B = 16, C = 4];

// Configuration: feed_flits = 2, datatype = bf16
//   init_shift = -2, init_shift_range = (-3, 4), shift_stride = 2
// Switch Engine output: [ P: [P_1=64] | H: [a_1=2, $[(b_1=(1,15):2)+(c_1=4:1)]=32] ] (16)
// Contraction mapping:       [ P: [P_1=64] | H: [a_1=2, c_2=2] | Lane: [] | T: [b_1=16, c_1=2] ] (16)

Data reuse without shift (tiling)

axes![P = 64, A = 2, C = 32];

// Configuration: feed_flits = 2, datatype = bf16
// Switch Engine output: [ P: [P_1=64] | H: [a_1=2, c_1=32] ] (16)
// Contraction mapping:       [ P: [P_1=64] | H: [a_1=2, #t_1=5] | Lane: [] | T: [c_1=32] ] (16)

• #t_1=5 broadcasts the same data 5 times without shifting.

pop_dim with shift dimension

axes![P = 64, A = 2, C = 32];

// Configuration: feed_flits = 2, datatype = bf16
// Switch Engine output: [ P: [P_1=64] | H: [a_1=2, c_1=32] ] (16)
// Contraction mapping:       [ P: [P_1=64] | H: [a_1=2, #s_1=3, #t_1=5] | Lane: [] | T: [c_1=32] ] (16)

• #s_1=3: shift dimension, where data shifts 3 times (shift-and-reuse).
• #t_1=5: pop_dim, broadcasting (tiling) the result 5 times.
• New data fetches only when moving beyond the outermost dimension.

pop_dim with sliding window and tiling

axes![P = 64, A = 2, B = 31, C = 3];

// Configuration: feed_flits = 2, datatype = bf16
//   init_shift = -1, init_shift_range = (-3, 4), shift_stride = 1
// Switch Engine output: [ P: [P_1=64] | H: [a_1=2, $[(b_1=(1,30):1)+(c_1=3:1)]=32] ] (16)
// Contraction mapping:       [ P: [P_1=64] | H: [a_1=2, c_1=3, #t_1=5] | Lane: [] | T: [[[b_1=31]+1]=32] ] (16)

• c_1=3 acts as the shift dimension with stride 1, generating 3 shifted windows per data buffer.
• #t_1=5 tiles (broadcasts) each of the 3 windows 5 times.
• New data fetches only after all c_1 and #t_1 iterations complete.

Constraints

The hardware limits each advanced operation, and the table below summarizes the limits with their physical causes.

Design Rationale

• 96-byte Flit Buffer: single-cycle access for the downstream Packet Reducer requires register file storage rather than standard SRAM. Register files consume significantly more area per bit, making large buffers prohibitively expensive. 96 bytes (3 flits) balances useful convolution patterns with silicon cost.

• Single-flit Transpose: restricting transpose to a single flit keeps the operation fast for the common case (ensuring the contraction axis is innermost). Extending it across multiple flits would require either a much larger permutation network or multi-cycle buffering.

• Shift buffer limits: the Stream Shift Unit implements sliding windows by physically shifting data through a register chain. The current limits (15 for i8, 7 for bf16) support common convolution filter sizes (3×3, 5×5, 7×7) while keeping hardware cost reasonable.

• TRF Sequencer alignment: the Outer stage’s elementwise multiplication and the Packet Reducer’s tree form a fixed-function multiply-accumulate array that expects precisely aligned input streams. It cannot buffer, reorder, or adapt to mismatched mappings, so misaligned mappings cause incorrect computation, not graceful degradation.

Performance

• Transpose latency: transpose within a flit adds 1-2 cycles of latency. This overhead is unavoidable when the contraction axis is not innermost.
• Shift configuration: incorrect shift configuration causes incorrect results, not just performance degradation. Other considerations:
  • Initial shift configuration overhead (setup cycles).
  • Stride and pop_dim parameters affect buffer management.
• Data reuse benefits: properly configured shift operations dramatically reduce memory bandwidth for convolutions by reusing input data across multiple output positions. Without shift, each output requires separate input fetches.

Vector Engine

The Vector Engine performs elementwise computation and reduction. Examples include activations (GELU, SiLU), normalizations (softmax, layer norm), binary operations, and intra- and inter-slice reductions.

The engine accepts only 32-bit types, i32 and f32. An upstream Contraction Engine widens types automatically (bf16 products accumulate in f32, i8 products in i32). When that engine is bypassed, the Fetch Engine must widen the input via its type-cast adapter.

Interface

In a single Tensor Unit invocation, the Vector Engine portion is the method chain from vector_init() to vector_final(). The engine has two sub-pieces: the Intra-Slice Chain (elementwise / binary / per-slice reduce stages) and the Inter-Slice Reducer (reduces across the 256 slices in a cluster). Between vector_init() and vector_final(), the chain alone, the reducer alone, or both can run. When both run, the order is IntraFirst (chain then reducer) or InterFirst (reducer then chain).

The intra-slice chain is entered via vector_intra_slice_tag(), or vector_intra_slice_unzip() when the input carries a 2-way grouping axis to split into two parallel streams (see Pair Mode). Either entry point fires right after vector_init() or on the inter-slice reducer’s output. The inter-slice reducer is entered via vector_inter_slice_reduce(), either right after vector_init() or from a compatible intra-slice stage. For stage-by-stage API coverage, see Intra-Slice Chain and Inter-Slice Reducer.

The signatures below cover the vector_init()-side entry methods only. The same method names (vector_intra_slice_tag, vector_inter_slice_reduce) also exist on chain and reducer tensors for the chain↔reducer transitions; those are documented in the child pages.

impl<'l, const T: Tu, P: CanApplyVectorInit, D: VeScalar, Chip: M, Cluster: M, Slice: M, Time: M, Packet: M>
    TuTensor<'l, T, P, D, Chip, Cluster, Slice, Time, Packet>
{
    /// Initializes Vector Engine processing for this tensor.
    #[primitive(TuTensor::vector_init)]
    pub fn vector_init(self) -> VectorInitTensor<'l, T, D, Chip, Cluster, Slice, Time, Packet> {
        VectorInitTensor::new(self.ctx, self.inner)
    }
}

impl<'l, const T: Tu, D: VeScalar, Chip: M, Cluster: M, Slice: M, Time: M, Packet: M>
    VectorInitTensor<'l, T, D, Chip, Cluster, Slice, Time, Packet>
{
    /// Enters VE intra-slice pipeline (single stream).
    #[primitive(VectorInitTensor::vector_intra_slice_tag)]
    pub fn vector_intra_slice_tag(
        self,
        branch: TagMode,
    ) -> VectorBranchTensor<'l, T, D, Chip, Cluster, Slice, Time, Packet, D, Fresh, { VeOrder::IntraFirst }> {
        VectorBranchTensor::new(self.ctx, self.inner, branch)
    }

    /// Enters VE intra-slice pipeline (two-group / unzip).
    #[primitive(VectorInitTensor::vector_intra_slice_unzip)]
    pub fn vector_intra_slice_unzip<I: AxisName, TileTime: M, SplitTime: M>(
        self,
    ) -> VectorTensorPair<'l, T, D, stage::Tag, Chip, Cluster, Slice, SplitTime, Packet> {
        VectorTensorPair::new::<I, Time, TileTime>(self.ctx, self.inner)
    }
}

impl<'l, const T: Tu, Chip: M, Cluster: M, Slice: M, Time: M, Packet: M>
    VectorInitTensor<'l, T, i32, Chip, Cluster, Slice, Time, Packet>
{
    /// Performs inter-slice reduce for i32 as the first VE operation.
    #[primitive(VectorInitTensor::vector_inter_slice_reduce)]
    pub fn vector_inter_slice_reduce<OutSlice: M, OutTime: M>(
        self,
        op: InterSliceReduceOpI32,
    ) -> VectorInterSliceReduceTensor<'l, T, i32, Chip, Cluster, OutSlice, OutTime, Packet, { VeOrder::InterFirst }>
    {
        let reduced = self.inner.reduce(op.reduce_fn(), op.identity(), true);
        create_inter_slice_reduce_tensor(self.ctx, reduced)
    }
}

Examples

A few representative examples follow to give a feel for what a Vector Engine call looks like, with the full API tour deferred to the child pages Intra-Slice Chain and Inter-Slice Reducer.

ReLU Activation

This pass applies ReLU elementwise, computing \(output[b, k, m, n] = \max(input[b, k, m, n], 0)\).

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![B = 2, K = 256, M = 16, N = 16];

// ReLU activation after batched matrix multiplication.
// Chain-only pass, so the reducer is skipped and the path trivially resolves to IntraFirst.
// Both clusters (B = 2) and all 256 slices (K) carry real data, no padding.
fn relu<'l, const T: Tu>(
    input: ContractTensor<'l, T, f32, m![1], m![B], m![K], m![M, N / 8], m![N % 8]>,
) -> VectorFinalTensor<'l, T, f32, m![1], m![B], m![K], m![M, N / 8], m![N % 8]> {
    input
        .vector_init()
        .vector_intra_slice_tag(TagMode::Zero)
        // max(x, 0), the ReLU itself
        .vector_clip(ClipBinaryOpF32::Max, 0.0f32)
        .vector_final()
}

let mut ctx = Context::acquire();

let c: ContractTensor<'_, _, f32, m![1], m![B], m![K], m![M, N / 8], m![N % 8]> = ContractTensor::new(&mut ctx.main, Tensor::zero());
let _o = relu(c);
}

ReLU Then Reduce

This pass applies ReLU per slice and then reduces across R, giving \(output[a, b] = \sum_{r \in R} \max(input[a, b, r], 0)\).

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 512, B = 2, R = 4];

// Chain applies ReLU, then reducer sums across slices.
// IntraFirst shape (chain runs first, then reducer).
// Both clusters (B = 2), all 256 slices (A / 8 * R), and full Way8 packet (A % 8) carry real data.
fn relu_then_reduce<'l, const T: Tu>(
    input: CollectTensor<'l, T, i32, m![1], m![B], m![A / 8, R], m![1], m![A % 8]>,
) -> VectorFinalTensor<'l, T, i32, m![1], m![B], m![A / 8, 1 # 4], m![1], m![A % 8]> {
    input
        .vector_init()
        .vector_intra_slice_tag(TagMode::Zero)
        // max(x, 0), the ReLU
        .vector_clip(ClipBinaryOpI32::Max, 0)
        // sum across R slices
        .vector_inter_slice_reduce::<m![A / 8, 1 # 4], m![1]>(InterSliceReduceOpI32::AddSat)
        .vector_final()
}

let mut ctx = Context::acquire();

let c: CollectTensor<'_, _, i32, m![1], m![B], m![A / 8, R], m![1], m![A % 8]> = CollectTensor::new(&mut ctx.main, Tensor::zero());
let _o = relu_then_reduce(c);
}

Reduce Then Bias

This pass reduces across R and then adds a constant bias, giving \(output[a, b] = \left(\sum_{r \in R} input[a, b, r]\right) + 100\).

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 512, B = 2, R = 4];

// Reducer sums across slices, then chain adds a bias to the reduced result.
// InterFirst shape (reducer runs first, then chain).
// Both clusters (B = 2), all 256 slices, and full Way8 packet carry real data.
fn reduce_then_add<'l, const T: Tu>(
    input: CollectTensor<'l, T, i32, m![1], m![B], m![A / 8, R], m![1], m![A % 8]>,
) -> VectorFinalTensor<'l, T, i32, m![1], m![B], m![A / 8, 1 # 4], m![1], m![A % 8]> {
    input
        .vector_init()
        // sum across R slices
        .vector_inter_slice_reduce::<m![A / 8, 1 # 4], m![1]>(InterSliceReduceOpI32::AddSat)
        .vector_intra_slice_tag(TagMode::Zero)
        // add bias 100
        .vector_fxp(FxpBinaryOp::AddFxp, 100)
        .vector_final()
}

let mut ctx = Context::acquire();

let c: CollectTensor<'_, _, i32, m![1], m![B], m![A / 8, R], m![1], m![A % 8]> = CollectTensor::new(&mut ctx.main, Tensor::zero());
let _o = reduce_then_add(c);
}

Intra- and Inter-Slice Reduction

This pass reduces R entirely by combining the intra-slice reducer (over R’s Time and Packet portions) and the inter-slice reducer (over R’s Slice portion). The einsum form is BR -> B, with saturating addition.

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![B = 2, R = 8192];

// R splits across Slice (R / 32 = 256), Time (R % 32 / 4 = 8), and Packet (R % 4, padded to 8 in Way8).
// Chain runs intra-slice reduce over R's Time and Packet portions, then the reducer collapses the Slice portion.
// IntraFirst shape (chain runs first, then reducer).
fn full_sum<'l, const T: Tu>(
    input: CollectTensor<'l, T, i32, m![1], m![B], m![R / 32], m![R % 32 / 4], m![R % 4 # 8]>,
) -> VectorFinalTensor<'l, T, i32, m![1], m![B], m![1 # 256], m![1], m![1 # 8]> {
    input
        .vector_init()
        .vector_intra_slice_tag(TagMode::Zero)
        // Way8 → Way4 (back 4 packet positions were padding)
        .vector_narrow_trim::<m![R % 4]>()
        // sum over R's Time and Packet portions
        .vector_intra_slice_reduce::<R, m![1], m![1 # 4]>(IntraSliceReduceOpI32::AddSat)
        // Way4 → Way8
        .vector_widen_pad::<m![1 # 8]>()
        // sum over R's Slice portion across all 256 slices
        .vector_inter_slice_reduce::<m![1 # 256], m![1]>(InterSliceReduceOpI32::AddSat)
        .vector_final()
}

let mut ctx = Context::acquire();

let c: CollectTensor<'_, _, i32, m![1], m![B], m![R / 32], m![R % 32 / 4], m![R % 4 # 8]> = CollectTensor::new(&mut ctx.main, Tensor::zero());
let _o = full_sum(c);
}

Pair Add

This pass unzips two interleaved groups along I and adds them pair-wise. The einsum form is ABI -> AB.

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 2048, B = 2, I = 2];

// Pair-mode entry via unzip, then a zip op fuses the two streams with an add.
// Both clusters (B = 2), all 256 slices (A / 8), and full Way8 packet (A % 8) carry real data.
fn pair_add<'l, const T: Tu>(
    input: CollectTensor<'l, T, i32, m![1], m![B], m![A / 8], m![I], m![A % 8]>,
) -> VectorFinalTensor<'l, T, i32, m![1], m![B], m![A / 8], m![1], m![A % 8]> {
    input
        .vector_init()
        // split into group 0 and group 1 along I
        .vector_intra_slice_unzip::<I, m![1 # 2], m![1]>()
        // group0 + group1
        .vector_clip_zip(ClipBinaryOpI32::AddFxp)
        .vector_final()
}

let mut ctx = Context::acquire();

let c: CollectTensor<'_, _, i32, m![1], m![B], m![A / 8], m![I], m![A % 8]> = CollectTensor::new(&mut ctx.main, Tensor::zero());
let _o = pair_add(c);
}

Intra-Slice Chain

The Intra-Slice Chain performs elementwise, binary, and intra-slice reduce operations on each slice’s data independently. It handles post-contraction processing such as activation and normalization. For example, computing \(\operatorname{sigmoid}(XW + b)\) runs \(XW\) on the Contraction Engine, and the addition plus sigmoid activation in the Intra-Slice Chain.

Interface

The example below applies a fixed-point bias, runs sigmoid on the float path (narrowing to 4-way and widening back), and finishes with a ReLU clip. It computes \(output[a, b] = \max(\operatorname{sigmoid}(input[a, b] + 100), 0)\).

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 512, B = 2];

fn staged_pipeline<'l, const T: Tu>(
    input: CollectTensor<'l, T, i32, m![1], m![B], m![A / 2], m![1], m![A % 2 # 8]>,
) -> VectorFinalTensor<'l, T, i32, m![1], m![B], m![A / 2], m![1], m![A % 2 # 8]> {
    input
    .vector_init()
    .vector_intra_slice_tag(TagMode::Zero)
    // input + 100
    .vector_fxp(FxpBinaryOp::AddFxp, 100)
    // i32 → f32 (fixed-point, int_width = 31)
    .vector_fxp_to_fp(31)
    // Way8 → Way4 for the float path
    .vector_narrow_trim::<m![A % 2 # 4]>()
    // sigmoid(input + 100)
    .vector_fp_unary(FpUnaryOp::Sigmoid)
    // Way4 → Way8
    .vector_widen_pad::<m![A % 2 # 8]>()
    // f32 → i32
    .vector_fp_to_fxp(31)
    // max(sigmoid(input + 100), 0)
    .vector_clip(ClipBinaryOpI32::Max, 0)
    .vector_final()
}

let mut ctx = Context::acquire();

let c: CollectTensor<'_, _, i32, m![1], m![B], m![A / 2], m![1], m![A % 2 # 8]> = CollectTensor::new(&mut ctx.main, Tensor::zero());
let _o = staged_pipeline(c);
}

Pipeline

The chain starts (as shown above) with vector_intra_slice_tag() (right after vector_init() or on the inter-slice reducer’s output), or with vector_intra_slice_unzip() directly after vector_init() for Pair Mode (see below).

    #[primitive(VectorInitTensor::vector_intra_slice_tag)]
    pub fn vector_intra_slice_tag(
        self,
        branch: TagMode,
    ) -> VectorBranchTensor<'l, T, D, Chip, Cluster, Slice, Time, Packet, D, Fresh, { VeOrder::IntraFirst }> {

    #[primitive(VectorInitTensor::vector_intra_slice_unzip)]
    pub fn vector_intra_slice_unzip<I: AxisName, TileTime: M, SplitTime: M>(
        self,
    ) -> VectorTensorPair<'l, T, D, stage::Tag, Chip, Cluster, Slice, SplitTime, Packet> {

After entry, the chain steps through the pipeline stages below in a fixed order; software chains the relevant ones and skips the rest, as the example skips Logic, FpDiv, and Filter. Each row lists the stage’s position in the chain (#), its name (Stage), the API method that triggers it (Method), the way it runs in (Way, either 8 or 4 elements per cycle), and whether it accepts an operand (Operand). The type system enforces every stage transition at compile time, so methods become callable only after the preceding chain reaches a compatible state. Per-stage detail is in Stages below.

Stages run either 8-way (8 elements per cycle) or 4-way (4 elements per cycle). The floating-point cluster runs 4-way to amortize its half-throughput ALUs against the rest of the chain. A chain that uses the float path therefore enters 8-way, calls Narrow (vector_narrow_split or vector_narrow_trim) before the float stages, and calls Widen (vector_widen_concat or vector_widen_pad) afterward to return to 8-way. The example does exactly this: vector_narrow_trim then vector_fp_unary(Sigmoid) then vector_widen_pad.

The chain exits via vector_final() (as in the example) or vector_inter_slice_reduce() (handing off to the Inter-Slice Reducer). Both exits require 8-way, so any active 4-way stage must pass through Widen first.

Operands

Binary and ternary ops have two slot kinds: a stream (the running tensor, i.e., the self of the method chain, fixed by the chain) and one or two operands (the extra inputs). Each operand comes from one of three sources.

Ternary ops (FmaF) take a pair (operand0, operand1).

The same op method picks up different sources by the argument type:

.vector_fxp(FxpBinaryOp::AddFxp, 100)                // operand from constant
.vector_fxp(FxpBinaryOp::MulInt, VeRhs::vrf(&vrf))   // operand from VRF tensor
.vector_clip(ClipBinaryOpI32::Max, Stash)            // operand from stash (set earlier)

The Stash source comes from vector_stash(), which snapshots the running tensor so a later binary or ternary op can read it back as the Stash operand. The typical use is a residual or skip-connection like max(f(x), x), where the original x must survive across intermediate stages. Call vector_stash() at any Stashable stage (Branch, Logic, Fxp, Narrow, Fp, FpDiv, Clip); the snapshot stays live until the Tensor Unit invocation ends and feeds any later binary or ternary call that takes Stash. The slot is single-use (a second vector_stash() is a compile-time error) and typed, so an f32 stash only feeds f32 ops. The stash is also read-once: it feeds exactly one later op (reading it moves the slot past Occupied, so a second Stash read is a compile-time error). A value that must be read more than once is not a stash - put it in a read-many VRF (VeRhs::vrf). The mapping follows the running tensor, so a stash taken before Narrow is still usable after Widen. Stash is unavailable in Pair Mode, and the IntraFirst transition to the inter-slice reducer drops it, so anything stashed before vector_inter_slice_reduce() is gone afterward.

An argument mode then picks which slots hold the stream versus the operands so the same op can compute, e.g., stream + operand or operand - stream. For example, BinaryArgMode::Mode10 swaps the slots so SubFxp computes operand - stream:

.vector_fxp_with_mode(FxpBinaryOp::SubFxp, BinaryArgMode::Mode10, 7)  // computes 7 - stream

BinaryArgMode picks which two of a binary op’s slots are stream vs operand (unary ops have no mode and always run as op(stream)):

TernaryArgMode does the same for ternary ops:

Pair Mode

Pair mode runs the chain on a tensor whose elements split into two interleaved groups, so an op can relate the two groups (e.g., pair-wise add, asymmetric scale). Entry is vector_intra_slice_unzip() directly after vector_init(), applied to a collected tensor that carries a 2-way grouping axis. Starting the chain with vector_intra_slice_unzip() precludes the Filter stage downstream. Under the hood, vector_intra_slice_unzip() uses TagMode::AxisToggle to derive each element’s GroupId from the 2-way grouping axis.

The flow has four steps:

1. vector_intra_slice_unzip() splits the input into two parallel streams (group 0 and group 1).
2. The chain runs through stages with both groups in lock-step (the paired phase).
3. A _zip op fuses the two streams back into one (the merged phase).
4. The merged stream continues to vector_final() like a normal chain.

Stages during the paired phase fall into two flavors:

• Common stages (vector_fxp_to_fp, vector_narrow_split, vector_widen_concat, vector_fp_to_fxp) act on both groups uniformly. vector_narrow_trim and vector_widen_pad are not available on pairs; use the _split / _concat variants instead.
• Per-group ops take one argument per group:
  • Binary and ternary (vector_fxp, vector_fp_binary, vector_fp_ternary, vector_clip, etc.) accept () on a side to skip it, or different operands on each side.
  • Unary (vector_fp_unary) is the exception: it takes flags (op, group0_apply, group1_apply) and false skips that group.

Pair mode reinterprets BinaryArgMode depending on the op: per-group ops (vector_fxp_with_mode, vector_fp_binary_with_mode, etc.) apply the mode inside each group independently (0 is that group’s stream, 1 is that group’s operand), while _zip ops (vector_fxp_zip_with_mode, etc.) take the two slots as the two grouped streams (0 is Group 0’s stream, 1 is Group 1’s stream):

Pair-mode constraints:

• stash() and filter() are unavailable throughout pair mode (both paired and merged phases).
• Before _zip (the paired phase), the chain cannot transition to the inter-slice reducer, since vector_inter_slice_reduce() is not available on per-group tensors. After _zip (the merged phase), the result is Commitable again and can call vector_inter_slice_reduce() if the current stage supports the transition.
• ALU usage is shared across the two groups: an ALU used in either group counts as consumed for both.

Stages

Within a stage, each ALU runs at most once per Tensor Unit invocation. This matters mainly in Logic, Fxp, Fp, and Clip, where multiple operators share a stage-local ALU pool. For example, tanh(sqrt(x)) cannot fit in a single Tensor Unit invocation because both tanh and sqrt consume the FpFpu ALU.

Tag

The Tag stage is the chain’s entry point and assigns each 32-bit element in a flit a 4-bit Tag (0-15), which later stages use to apply conditional operations. Bit 3 (the MSB) is GroupId, used by Filter and pair mode to split elements into Group 0 / Group 1. Bits 0..2 are general-purpose flag bits filled by comparison results.

The TagMode selects how the 4 bits are computed for each element:

For example, with i32 data and Comparison([Less{0}, Equal{5}, Greater{100}, True]), an element x = 7 yields bits 0/0/0/1 (LSB first), so its tag is 0b1000 = 8.

Once tags are assigned, later binary and ternary ops can condition an operand on the GroupId MSB so different tag groups see different values. BinaryOperandTag::always(operand) applies to all groups, BinaryOperandTag::group(operand, GroupId::Zero) applies only to group 0. TernaryOperandTag is the ternary form.

Logic Cluster

The Logic Cluster performs bitwise operations on i32 or f32 (bit-level). It runs 8-way.

The stage exposes five ALU classes (LogicAnd, LogicOr, LogicXor, LogicLshift, LogicRshift), each runnable once per Tensor Unit invocation. Operators sharing the same class cannot fuse into one invocation.

i32 operations:

f32 operations:

Fxp Cluster

The Fxp Cluster performs integer and fixed-point arithmetic on i32. It runs 8-way.

The stage exposes four ALU classes (FxpAdd, FxpLshift, FxpMul, FxpRshift), each runnable once per Tensor Unit invocation. Operators sharing the same class cannot fuse into one invocation.

The single-ALU rule rejects, for example, two ops that both target FxpAdd:

// PANICS: "FxpAdd is already in use"
input
    .vector_init()
    .vector_intra_slice_tag(TagMode::Zero)
    .vector_fxp(FxpBinaryOp::AddFxp, 10)    // uses FxpAdd
    .vector_fxp(FxpBinaryOp::MulInt, 2)     // uses FxpMul ✓
    .vector_fxp(FxpBinaryOp::SubFxp, 5)     // uses FxpAdd again ✗
    .vector_final()

FxpToFp Conversion

The FxpToFp Conversion stage converts i32 to f32. The int_width parameter specifies the integer bit width for the conversion; int_width = 31 is the standard i32 ↔ f32 conversion.

Narrow

The Narrow stage switches 8-way to 4-way. An 8-way packet carries 8 active elements (Packet = m![... # 8]) and a 4-way packet carries 4 (Packet = m![... # 4]). Narrowing halves throughput on the float and reduce path, so the same logical tensor shape takes twice as many packets or Tensor Unit invocations.

Shape semantics:

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 512, B = 2, S = 64];

fn vector_narrow_split_semantics<'l, const T: Tu>(
    input: VectorBranchTensor<'l, T, i32, m![1], m![B], m![S / 4 # 256], m![S % 4], m![A % 8], i32, Fresh, { stage::VeOrder::IntraFirst }>,
) -> VectorNarrowTensor<'l, T, i32, m![1], m![B], m![S / 4 # 256], m![S % 4, A / 4 % 2], m![A % 4], i32, Fresh, { stage::VeOrder::IntraFirst }>
{
    input.vector_narrow_split::<m![S % 4, A / 4 % 2], m![A % 4]>()
    // shape semantics: [T], [P] -> [T, P / 2], [P % 4]
}

fn vector_narrow_trim_semantics<'l, const T: Tu>(
    input: VectorBranchTensor<'l, T, f32, m![1], m![B], m![A / 2], m![1], m![A % 2 # 8], f32, Fresh, { stage::VeOrder::IntraFirst }>,
) -> VectorNarrowTensor<'l, T, f32, m![1], m![B], m![A / 2], m![1], m![A % 2 # 4], f32, Fresh, { stage::VeOrder::IntraFirst }>
{
    input.vector_narrow_trim::<m![A % 2 # 4]>()
    // shape semantics: [T], [P] -> [T], [P = 4]
}

let mut ctx = Context::acquire();

let i: VectorBranchTensor<'_, _, i32, m![1], m![B], m![S / 4 # 256], m![S % 4], m![A % 8], i32, Fresh, { stage::VeOrder::IntraFirst }> = VectorBranchTensor::new(&mut ctx.main, Tensor::zero(), TagMode::Zero);
let _o = vector_narrow_split_semantics(i);

let i: VectorBranchTensor<'_, _, f32, m![1], m![B], m![A / 2], m![1], m![A % 2 # 8], f32, Fresh, { stage::VeOrder::IntraFirst }> = VectorBranchTensor::new(&mut ctx.main, Tensor::zero(), TagMode::Zero);
let _o = vector_narrow_trim_semantics(i);
}

Float Cluster

The Float Cluster provides unary, binary, and ternary floating-point operations on f32. It runs 4-way, so the input must already have passed through Narrow.

It exposes five independent ALUs (FpFma, FpFpu, FpExp, FpMul0, FpMul1), each runnable once per Tensor Unit invocation. This stage is where ALU planning matters most.

Unary ops:

Binary ops:

Ternary ops:

For example, to compute exp(sqrt(((x + 1) * 2) * 3)):

• x1 = x + 1 via FpFma (FpBinaryOp::AddF)
• x2 = x1 * 2 via FpMul0 (FpBinaryOp::MulF(FpMulAlu::Mul0))
• x3 = x2 * 3 via FpMul1 (FpBinaryOp::MulF(FpMulAlu::Mul1))
• x4 = sqrt(x3) via FpFpu (FpUnaryOp::Sqrt)
• x5 = exp(x4) via FpExp (FpUnaryOp::Exp)

IntraSliceReduce

The IntraSliceReduce stage reduces axes within a single slice. It runs 4-way. The stage uses a dedicated accumulator-tree ALU, so no user-selectable ALU is exposed.

See Intra-Slice Reduce for details.

FpDiv

The FpDiv stage performs floating-point division. It runs 4-way. The stage uses a dedicated floating-point divider, so no user-selectable ALU is exposed.

Widen

The Widen stage transitions from 4-way back to 8-way. Later stages (FpToFxp, Clip, Filter, Output) then see 8-element packets again.

Shape semantics:

#![allow(unused)]
fn main() {
#![feature(adt_const_params)]
extern crate furiosa_opt_std;
use furiosa_opt_std::prelude::*;
axes![A = 512, B = 2, S = 64, R = 8];

fn vector_widen_concat_semantics<'l, const T: Tu>(
    input: VectorIntraSliceReduceTensor<'l, T, i32, m![1], m![B], m![S / 4 # 256], m![A / 4 % 2], m![A % 4], i32, Fresh, { stage::VeOrder::IntraFirst }>,
) -> VectorWidenTensor<'l, T, i32, m![1], m![B], m![S / 4 # 256], m![1], m![A % 8], i32, Fresh, { stage::VeOrder::IntraFirst }>
{
    input.vector_widen_concat::<m![1], m![A % 8]>()
    // shape semantics: [T, P / 2], [P % 4] -> [T], [P]
}

fn vector_widen_pad_semantics<'l, const T: Tu>(
    input: VectorFpTensor<'l, T, f32, m![1], m![B], m![A / 2], m![1], m![A % 2 # 4], f32, Fresh, { stage::VeOrder::IntraFirst }>,
) -> VectorWidenTensor<'l, T, f32, m![1], m![B], m![A / 2], m![1], m![A % 2 # 8], f32, Fresh, { stage::VeOrder::IntraFirst }>
{
    input.vector_widen_pad::<m![A % 2 # 8]>()
    // shape semantics: [T], [P] -> [T], [P # 8]
}

let mut ctx = Context::acquire();

let i: VectorBranchTensor<'_, _, i32, m![1], m![B], m![S / 4 # 256], m![R, A / 4 % 2], m![A % 4 # 8], i32, Fresh, { stage::VeOrder::IntraFirst }> = VectorBranchTensor::new(&mut ctx.main, Tensor::zero(), TagMode::Zero);
let i = i
    .vector_narrow_trim::<m![A % 4]>()
    .vector_intra_slice_reduce::<R, m![A / 4 % 2], m![A % 4]>(IntraSliceReduceOpI32::AddSat);

let _o = vector_widen_concat_semantics(i);

let i: VectorBranchTensor<'_, _, f32, m![1], m![B], m![A / 2], m![1], m![A % 2 # 8], f32, Fresh, { stage::VeOrder::IntraFirst }> = VectorBranchTensor::new(&mut ctx.main, Tensor::zero(), TagMode::Zero);
let i = i.vector_narrow_trim::<m![A % 2 # 4]>().vector_fp_unary(FpUnaryOp::Exp);
let _o = vector_widen_pad_semantics(i);
}

FpToFxp Conversion

The FpToFxp Conversion stage converts f32 back to i32. The int_width parameter specifies the integer bit width.

Clip Cluster

The Clip Cluster performs clamping and comparison operations. It runs 8-way.

The stage exposes three ALU classes (ClipAdd, ClipMax, ClipMin), each runnable once per Tensor Unit invocation.

i32 operations: