Design

AI Hardware Updated Sep 2026

Tensor engine fundamentals

Engineering notebook — August 20, 2026

Overview

A tensor engine accelerates matrix multiply (GEMM) and convolution, the core operations in deep learning inference and training. The canonical form discussed here is an output-stationary systolic array computing C = A × B, where A is M×K and B is K×N.

Architecture: the output-stationary systolic array

Core idea

A grid of identical processing elements (PEs) — say 4×4 — where each PE at position (row, col) is responsible for computing exactly one output element: C[row][col]. The PE is "stationary" in the sense that it never moves data anywhere; it just accumulates.

           B flows down ↓
            col0  col1  col2  col3
A flows →   ┌─────┬─────┬─────┬─────┐
row 0       │PE00 │PE01 │PE02 │PE03 │
            ├─────┼─────┼─────┼─────┤
row 1       │PE10 │PE11 │PE12 │PE13 │
            ├─────┼─────┼─────┼─────┤
row 2       │PE20 │PE21 │PE22 │PE23 │
            ├─────┼─────┼─────┼─────┤
row 3       │PE30 │PE31 │PE32 │PE33 │
            └─────┴─────┴─────┴─────┘

Rows of A enter from the left, shifting right one PE per cycle.
Columns of B enter from the top, shifting down one PE per cycle.
Every cycle, each PE computes: acc += a_in * b_in, then passes a_in to its right neighbor and b_in to its neighbor below.
After K cycles, PE[row][col].acc holds the complete dot product = C[row][col].

What each PE does (every cycle)

acc  ← acc + (a_in × b_in)
a_out ← a_in      // forward right
b_out ← b_in      // forward down

Nothing more. The intelligence of the design is in the data routing, not the PE itself.

The skew problem and diagonal wavefront

Why skewing is necessary

Data cannot be fed naively — the right element of A and the right element of B must arrive at the correct PE on the correct cycle. This requires skewed (staggered) inputs:

Row i of A is delayed by i cycles before entering column 0.
Column j of B is delayed by j cycles before entering row 0.

This creates a diagonal wavefront: on each cycle, the active multiply-accumulate operations sweep diagonally across the PE array.

Skew input structure

B column inputs (top edge):
  col 0: b00 → no delay
  col 1: b01 → 1 cycle delay register
  col 2: b02 → 2 cycle delay registers
  col 3: b03 → 3 cycle delay registers

A row inputs (left edge):
  row 0: a00 → no delay
  row 1: a10 → 1 cycle delay register
  row 2: a20 → 2 cycle delay registers
  row 3: a30 → 3 cycle delay registers

Once inside the array, forwarding takes care of the rest automatically — no further control logic needed inside the PE grid during the compute phase.

The three operational phases

Every matrix multiply through this array passes through three phases:

Phase	Duration	What's happening
Fill	~N cycles	Data propagates in; not all PEs have valid operands yet. No output.
Compute	K cycles	All PEs are actively MACing every cycle. Steady state.
Drain	~N cycles	All input consumed; accumulators read out, often by shifting through the array.

For an N×N array with K-length inner dimension, total cycles ≈ K + 2N. The 2N overhead is amortized over large K, but matters for small matrices.

Building blocks (Verilog perspective)

Key parameters

parameter DATA_W  = 8;   // input precision (e.g. INT8)
parameter ACC_W   = 32;  // accumulator width (INT8 × INT8 → INT32)
parameter N       = 4;   // array dimension (N×N PEs)
parameter K       = 16;  // inner dimension (dot product length)

Single PE (conceptual)

module pe #(parameter DATA_W=8, ACC_W=32) (
  input  clk, rst_n, valid_in,
  input  signed [DATA_W-1:0] a_in, b_in,
  output reg signed [DATA_W-1:0] a_out, b_out,
  output reg signed [ACC_W-1:0] acc
);
  always @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      acc   <= 0;
      a_out <= 0;
      b_out <= 0;
    end else if (valid_in) begin
      acc   <= acc + a_in * b_in;  // MAC
      a_out <= a_in;               // pass right
      b_out <= b_in;               // pass down
    end
  end
endmodule

Skew logic (left-edge and top-edge delay chains)

// Left-edge: delay row i by i cycles before feeding PE[i][0]
genvar i;
generate
  for (i = 0; i < N; i = i + 1) begin : row_skew
    shift_reg #(.WIDTH(DATA_W), .DEPTH(i)) sr_a (
      .clk(clk), .din(a_row[i]), .dout(a_skewed[i])
    );
  end
endgenerate

// Top-edge: delay column j by j cycles before feeding PE[0][j]
generate
  for (i = 0; i < N; i = i + 1) begin : col_skew
    shift_reg #(.WIDTH(DATA_W), .DEPTH(i)) sr_b (
      .clk(clk), .din(b_col[i]), .dout(b_skewed[i])
    );
  end
endgenerate

Architecture decisions and trade-offs

Decision	Options	Notes
Dataflow	Output-stationary, weight-stationary, input-stationary	Output-stationary simplest; weight-stationary minimizes weight reloads
PE array	Systolic grid vs. SIMD lanes	Systolic = high utilization at fixed shape; SIMD = more flexible
Precision	INT8, FP16, BF16, FP32	INT8 with INT32 accumulation is the sweet spot for inference
Tiling	Software vs. hardware	Real matrices far exceed array size; tiling logic is essential
Memory	SRAM buffers + DMA	Double-buffering hides load latency behind compute

What's not in this basic version

The design described here is the minimal kernel — one tile, no wrapping logic. Real engines add:

Tiling/blocking — break large matrices into tiles that fit the array, sequence them
Double buffering — overlap loading tile N+1 while computing tile N
Quantization/dequantization — INT8 in, float out (or vice versa)
AXI/DMA interface — move data between DDR and on-chip SRAM
Control FSM — sequence fill → compute → drain across multiple tiles
Activation functions — ReLU, GELU applied post-accumulation

Next steps

[ ] Write and simulate full PE module in Verilog
[ ] Build shift-register skew logic for N=4
[ ] Wire up 4×4 array with parametric generate blocks
[ ] Write self-checking testbench against NumPy golden model
[ ] Add tiling FSM for matrices larger than 4×4
[ ] Explore weight-stationary alternative to eliminate skew registers