Catalog: every cajeta-llm kernel

Generated from the sources, census/oracle-map.tsv and the committed census tables of dev.cajeta.llm on 2026-10-26 by tools/catalog/gen-catalog.py. One entry per kernel registered in the test binary on any backend: the kernel’s name, linked to its source line, the llama.cpp counterpart the parity harness compares it against (NONE where llama.cpp has no twin and the roofline judges it), the census class on cuda, cpu and hip (CHECKED: a test credited its values against a host or oracle answer; RAN-UNCHECKED and SKIP: excused against a named plan item; DECLINED: the compiler produced no device code there; blank: not registered on that backend), and under it the kernel’s role from its own documentation. The families and routes are described in Language models on the device; the adaptation machinery in One kernel, any device.

274 library kernels, plus the probe, external and fixture kernels the binary also registers (listed last).

Census marks, one per backend (hover a mark for its class): ✓ CHECKED, ○ RAN-UNCHECKED, – SKIP, ✗ DECLINED, blank: not registered on that backend. Each kernel’s name links to its source line; the line under it is its role.

Wave mat-vecs: decode, one row per wave (llama.cpp mmvq) (56)

kernel	llama.cpp counterpart	cuda	cpu	hip
`f16F32WaveMatVecKernel`	mmvf.cu:mul_mat_vec_f	✓	✓	✓
y[yOff + i] = W[i]·x for one row per wave, W stored F16 row-major: a lane takes eight halves (16 bytes) per step, the wave 256. (f16 weight; not in the reference model)
`iq2sIq3xxsQkvWaveMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
The decode q, k and v projections in one launch for the recipe that quantizes q and k as IQ2_S and v as IQ3_XXS: the first (rowsQ+1)/2 waves take q row pairs, the next (rowsKv+1)/2 k row pairs, the last rowsKv one v row each, every dot exactly as the per-type wave kernel takes it, the biases in the epilogue. (fused q/k/v three-weight form; llama.cpp runs three MUL_MAT; judged against three mmvq launches)
`iq2sQ8WaveGateUpGluKernel`	NONE	✓	✓	✓
The shared expert's gate and up projections of two rows in one wave with the GLU of Prim.gluF32 in the epilogue: both dots exactly as #iq2sQ8WaveMatVecKernel takes them, so the launch is bit-identical to the three it replaces. (fused gate+up+GLU: llama.cpp runs two MUL_MAT and a GLU; judged by roofline)
`iq2sQ8WaveMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
The IQ2_S twin: raw sign bytes, and the grid index's high two bits from the block's qh. (format not in the reference model; judged at the q4_K shapes)
`iq2sQ8WaveMatVecLdsKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
#iq2sQ8WaveMatVecKernel with the codebook staged in LDS by the workgroup instead of read through L1 (spec 12.1's arms; the two must agree bit for bit). (LDS-staged variant of the wave mat-vec)
`iq2xsQ8WaveMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
The IQ2_XS twin of #iq2xxsQ8WaveMatVecKernel: the sign index rides the high seven bits of each qs entry and the sub-block scales are a byte pair per 64. (format not in the reference model; judged at the q4_K shapes)
`iq2xxsQ8WaveMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
format not in the reference model; judged at the q4_K shapes
`iq3sQ8WaveMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
The IQ3_S twin. (format not in the reference model; judged at the q4_K shapes)
`iq3xxsQ8IdGateUpGluKernel`	NONE	✓	–	✓
The IQ3_XXS twin of #q4kQ8IdGateUpGluKernel: gate and up rows of one selection slot in one wave, both dots exactly as #iq3xxsQ8IdMatVecKernel takes them, the GLU of Prim.gluF32 in the epilogue — bit-identical to the three launches it replaces. (fused MoE gate+up+GLU id form; judged by roofline)
`iq3xxsQ8IdMatVecKernel`	mmvq.cu:mul_mat_vec_q (MUL_MAT_ID)	✓	✓	✓
6.4.3 cause 3 — the IQ3_XXS id twin of #iq3xxsQ8WaveMatVecKernel: one wave per (selection slot, row) against the bank's whole slab, the dot loop unchanged. (MoE id form; the reference model is dense)
`iq3xxsQ8WaveGateUpGluKernel`	NONE	✓	–	✓
The IQ3_XXS form of #iq2sQ8WaveGateUpGluKernel: one row a wave, both dots exactly as #iq3xxsQ8WaveMatVecKernel takes them, Prim.gluF32's activation in the epilogue. (fused gate+up+GLU: llama.cpp runs two MUL_MAT and a GLU; judged by roofline)
`iq3xxsQ8WaveMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
The IQ3_XXS twin. (format not in the reference model; judged at the q4_K shapes)
`iq3xxsQkvWaveMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
The IQ3_XXS form of #qkvWaveMatVecKernel: waves 0..rowsQ-1 own q rows, the next rowsKv k rows, the next rowsKv v rows, each dot exactly as #iq3xxsQ8WaveMatVecKernel takes it, the biases in the epilogue; one body per weight parameter because the lowering knows buffer parameters only. (fused q/k/v three-weight form; llama.cpp runs three MUL_MAT; judged against three mmvq launches)
`iq4nlF32WaveMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
The IQ4_NL member of the f32-wave family — Q4_0's wave geometry with the nibble decoded through #iq4Kv and Vector.lut4, exactly as the one-item kernel decodes it. (format not in the reference model; judged at the q4_K shapes)
`iq4nlQ8IdDownCombineKernel`	NONE	✓	✓	✓
The IQ4_NL twin of #q4kQ8IdDownCombineKernel, the combine in moeCombineAddKernel's order: each expert's row total is reduced as the id mat-vec reduces it, then hs += sum_k w_k * tot_k — so the fused launch is bit-identical to the chain it replaces. (fused MoE down mat-vec + weighted combine; llama.cpp runs MUL_MAT_ID then a weighted reduction; judged by roofline)
`iq4nlQ8IdMatVecKernel`	mmvq.cu:mul_mat_vec_q (MUL_MAT_ID)	✓	✓	✓
The IQ4_NL id twin of #iq4nlQ8WaveMatVecKernel. (MoE id form; the reference model is dense)
`iq4nlQ8WaveMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
WAVE-COOPERATIVE IQ4_NL x q8_K. (format not in the reference model; judged at the q4_K shapes)
`q2kQ8WaveMatVecColsKernel`	mmvq.cu:mul_mat_vec_q (ncols_dst 2-8)	✓
#q2kQ8WaveMatVecKernel for up to #WAVE_COLS_MAX columns. (q2kQ8WaveMatVecKernel over up to eight columns; format not in the reference model, judged at the q4_K shapes (15.2.15.3))
`q2kQ8WaveMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
WAVE-COOPERATIVE Q2_K x q8_K (10.12.45.C). (format not in the reference model; judged at the q4_K shapes)
`q3kQ8WaveMatVecColsKernel`	mmvq.cu:mul_mat_vec_q (ncols_dst 2-8)	✓
#q3kQ8WaveMatVecKernel for up to #WAVE_COLS_MAX columns. (q3kQ8WaveMatVecKernel over up to eight columns; format not in the reference model, judged at the q4_K shapes (15.2.15.3))
`q3kQ8WaveMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
WAVE-COOPERATIVE Q3_K x q8_K (10.12.45.C). (format not in the reference model; judged at the q4_K shapes)
`q40F32WaveMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
The Q4_0 member of the f32-wave family — see the Q8_0 twin. (format not in the reference model; judged at the q4_K shapes)
`q40Q8WaveMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
WAVE-COOPERATIVE Q4_0 x q8_K. (format not in the reference model; judged at the q4_K shapes)
`q4kQ8IdDownCombineKernel`	NONE	✓	✓	✓
Unit 51: the down projection for ALL used experts of one output row in one wave, the combine in the epilogue: lane slot (lane / 4) walks the (expert, block) pairs kb = slot, slot+8, ... (fused MoE down mat-vec + weighted combine; llama.cpp runs MUL_MAT_ID then a weighted reduction; judged by roofline)
`q4kQ8IdGateUpGluKernel`	NONE	✓	✓	✓
Unit 48: gate AND up in one indirected launch, the GLU in its epilogue. (fused MoE gate+up+GLU id form; judged by roofline)
`q4kQ8IdMatVecKernel`	mmvq.cu:mul_mat_vec_q (MUL_MAT_ID)	✓	✓	✓
── Unit 40 (spec 15.21.3): expert-indirected mat-vec ────────────── The llama.cpp mul_mat_vec_id shape: ONE launch covers every selected expert of a bank. (MoE id form; the reference model is dense)
`q4kQ8WaveG2MatVecColsKernel`	mmvq.cu:mul_mat_vec_q (ncols_dst 2-8)
Q4_K x q8_K for up to four columns, llama.cpp's mat-vec geometry: a 64-thread group takes two rows, 16 lanes take one 256-weight block (each lane 16 consecutive weights: half a sub-block, one 16-byte load), so a group covers four blocks a trip and both waves stream the same two rows; each wave reduces its partials and the two meet in shared memory. (llama.cpp's mat-vec geometry, two rows a 64-thread group, 16 lanes a block; Cols pin 4 only (15.2.12.3.9.6.4))
`q4kQ8WaveMatVecColsKernel`	mmvq.cu:mul_mat_vec_q (ncols_dst 2-8)	✓	✓	✓
Q4_K x q8_K for up to #WAVE_COLS_MAX activation columns in one pass over the weights: y[yOff + c*yStride + row] (+)= W[row] .
`q4kQ8WaveMatVecColsSegPfKernel`	mmvq.cu:mul_mat_vec_q (ncols_dst 2-8)	✓
Up to three Q4_K weights in one launch: waves [0, rows0) run w0 into y0, the next rows1 run w1 into y1, the next rows2 run w2 into y2, each with the prefetching row body; the wave reduces follow once, so every output equals the separate launches bit for bit (15.2.12.3.7). (q4kQ8WavePfMatVecColsKernel over up to three weights in one grid (q\|k\|v, gate\|up))
`q4kQ8WaveMatVecColsSegPlainKernel`	mmvq.cu:mul_mat_vec_q (ncols_dst 2-8)	✓
Up to three Q4_K weights in one launch: waves [0, rows0) run w0 into y0, the next rows1 run w1 into y1, the next rows2 run w2 into y2, each with the plain row body; the wave reduces follow once, so every output equals the separate launches bit for bit (15.2.12.3.7). (q4kQ8WaveMatVecColsKernel over up to three weights in one grid (q\|k\|v, gate\|up); llama.cpp launches mmvq once per weight)
`q4kQ8WaveMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
WAVE-COOPERATIVE Q4_K x q8_K — llama.cpp's mmvq decomposition (plan 11.1).
`q4kQ8WavePfMatVecColsKernel`	mmvq.cu:mul_mat_vec_q (ncols_dst 2-8)	✓
#q4kQ8WaveMatVecColsKernel with block b + 1's weight loads issued before block b's math, for a compiler that keeps few loads in flight (cajeta-llm 15.2.12.3.1). (q4kQ8WaveMatVecColsKernel with the next block's weights prefetched, where a probe finds it faster (15.2.12.3.1))
`q4kQ8WavePfMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓
#q4kQ8WaveMatVecKernel with block b + 1's loads issued before block b's math, for a compiler that keeps few loads in flight (cajeta-llm 15.2.12.2.1). (q4kQ8WaveMatVecKernel with the next block prefetched, where a probe finds it faster (15.2.12.2.1))
`q4kQ8WaveR2C4MatVecColsKernel`	mmvq.cu:mul_mat_vec_q (ncols_dst 2-8)
#q4kQ8WaveR2MatVecColsKernel for four columns or fewer: without the accumulators for columns 5..8 it fits 13 waves a SIMD on gfx1151 instead of 12, so a 2048-wave launch is two full rounds and no tail (15.2.12.3.9.6). (q4kQ8WaveR2MatVecColsKernel at four columns or fewer, without the accumulators for columns 5..8 (15.2.12.3.9.6))
`q4kQ8WaveR2MatVecColsKernel`	mmvq.cu:mul_mat_vec_q (ncols_dst 2-8)	✓
#q4kQ8WavePfMatVecColsKernel over two rows a wave, each column's activation loaded once for both; row 2w + 1 past the end is clamped and not stored (15.2.12.3.4.1). (q4kQ8WaveMatVecColsKernel over two rows a wave, chosen where a probe finds it fastest (15.2.12.3.4.1))
`q4kQ8WaveR4MatVecColsKernel`	mmvq.cu:mul_mat_vec_q (ncols_dst 2-8)
#q4kQ8WaveR2C4MatVecColsKernel over four rows a wave: each column's activation is loaded once for all four (15.2.12.3.9.6.5). (q4kQ8WaveR2C4MatVecColsKernel over four rows a wave; Cols pin 5 only (15.2.12.3.9.6.5))
`q50F32WaveMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
The Q5_0 member of the f32-wave family — see the Q8_0 twin for the geometry and the Q5_0 f32 one-item kernel for why the 5th bit rides a scalar fold over the activations. (format not in the reference model; judged at the q4_K shapes)
`q50Q8WaveMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
WAVE-COOPERATIVE Q5_0 x q8_K. (format not in the reference model; judged at the q4_K shapes)
`q5kQ8IdMatVecKernel`	mmvq.cu:mul_mat_vec_q (MUL_MAT_ID)	✓
#q4kQ8IdMatVecKernel for Q5_K expert banks (cajeta-llm 15.2.15.4): the expert-indirected shape over 176-byte blocks, the fifth bit from qh at +16, the quants from +48, and the Q4_K header decode; the dot is the one-column Q5_K wave kernel's, operation for operation, so the id launch is bit-identical to a wave launch per selected expert. (MoE id form for Q5_K expert banks; the reference model is dense (15.2.15.4))
`q5kQ8WaveMatVecColsKernel`	mmvq.cu:mul_mat_vec_q (ncols_dst 2-8)	✓
#q5kQ8WaveMatVecKernel for up to #WAVE_COLS_MAX columns: the Q5_K decode (the fifth bit from qh), then the Q4_K column accumulate, whose arithmetic is the one-column Q5_K kernel's. (q5kQ8WaveMatVecKernel over up to eight columns; format not in the reference model, judged at the q4_K shapes (15.2.15.3))
`q5kQ8WaveMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
WAVE-COOPERATIVE Q5_K x q8_K (10.12.39). (format not in the reference model; judged at the q4_K shapes)
`q6kQ8IdDownCombineKernel`	NONE	✓	✓	✓
Unit 51: the down projection for ALL used experts of one output row in one wave, the combine in the epilogue: lane slot (lane / 4) walks the (expert, block) pairs kb = slot, slot+8, ... (fused MoE down mat-vec + weighted combine; llama.cpp runs MUL_MAT_ID then a weighted reduction; judged by roofline)
`q6kQ8IdMatVecKernel`	mmvq.cu:mul_mat_vec_q (MUL_MAT_ID)	–	–	–
MoE id form; the reference model is dense
`q6kQ8WaveMatVecColsKernel`	mmvq.cu:mul_mat_vec_q (ncols_dst 2-8)	✓	✓	✓
Q6_K (row-split resident layout) x q8_K for up to #WAVE_COLS_MAX columns, the twin of #q4kQ8WaveMatVecColsKernel.
`q6kQ8WaveMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
Q6_K wave mat-vec, ZERO-REDUNDANCY lane mapping.
`q6kQ8WavePfMatVecColsKernel`	mmvq.cu:mul_mat_vec_q (ncols_dst 2-8)	✓
#q6kQ8WaveMatVecColsKernel with block b + 1's weight loads issued before block b's math, for a compiler that keeps few loads in flight (cajeta-llm 15.2.12.3.1). (q6kQ8WaveMatVecColsKernel with the next block's weights prefetched, where a probe finds it faster (15.2.12.3.1))
`q6kQ8WavePfMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓
#q6kQ8WaveMatVecKernel with block b + 1's loads issued before block b's math, for a compiler that keeps few loads in flight (cajeta-llm 15.2.12.2.5). (q6kQ8WaveMatVecKernel with the next block prefetched, where a probe finds it faster (15.2.12.2.5))
`q6kQ8WaveR2MatVecColsKernel`	mmvq.cu:mul_mat_vec_q (ncols_dst 2-8)	✓
#q6kQ8WavePfMatVecColsKernel over two rows a wave, each column's activation loaded once for both; row 2w + 1 past the end is clamped and not stored (15.2.12.3.4.1). (q6kQ8WaveMatVecColsKernel over two rows a wave, chosen where a probe finds it fastest (15.2.12.3.4.1))
`q80F32WaveMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
Q8_0 x F32 activations, WAVE decomposition — the kernel the 72B's ffn_down actually needed. (format not in the reference model; judged at the q4_K shapes)
`q80Q8WaveMatVecColsKernel`	mmvq.cu:mul_mat_vec_q (ncols_dst 2-8)	✓
#q80Q8WaveMatVecKernel for up to #WAVE_COLS_MAX columns. (q80Q8WaveMatVecKernel over up to eight columns; format not in the reference model, judged at the q4_K shapes (15.2.15.3))
`q80Q8WaveMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
WAVE-COOPERATIVE Q8_0 x q8_K (10.12.39). (format not in the reference model; judged at the q4_K shapes)
`qkvWaveMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
Unit 49: the decode q, k and v projections in ONE launch. (fused q/k/v three-weight form; llama.cpp runs three MUL_MAT; judged against three mmvq launches)
`symQ8IdDownCombineKernel`	NONE	✓	✓	✓
Unit 9.6: #q4kQ8IdDownCombineKernel's shape over the widened symmetric slab — every used expert of one output row in one wave, a lane walking the (expert, 32-block) pairs, the weighted combine onto the residual in the epilogue and the next layer's norm + pack in the tail (that tail is the Q4_K kernel's, verbatim). (fused MoE down mat-vec + weighted combine; llama.cpp runs MUL_MAT_ID then a weighted reduction; judged by roofline)
`symQ8IdMatVecKernel`	mmvq.cu:mul_mat_vec_q (MUL_MAT_ID)	✓	✓	✓
Unit 9.6: the symmetric id mat-vec — #q80Q8WaveMatVecKernel's dot over the bank's WIDENED slab (int8 in the GEMM's 16x16 tiles at the padded width, f16 scales in the compact image) indirected through sel, so a Q8_0/Q5_0/Q4_0 expert bank takes one launch per bank like the k-quants do. (MoE id form; the reference model is dense)
`tq10Q8WaveMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
format not in the reference model; judged at the q4_K shapes
`tq20Q8WaveMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
format not in the reference model; judged at the q4_K shapes

Batch GEMMs on the integer dot product (3)

kernel	llama.cpp counterpart	cuda	cpu	hip
`q4kQ8Batch4MatMulKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
BATCHED Q4_K, FOUR TOKENS PER LANE (plan 10.12.4). (batched decode n in 2..8: llama.cpp takes mmvq up to MMVQ_MAX_BATCH_SIZE 8)
`q4kQ8BatchMatMulKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
BATCHED Q4_K x Q8_K: y[rows x outDim] = X[rows x cols] . (batched decode n in 2..8: llama.cpp takes mmvq up to MMVQ_MAX_BATCH_SIZE 8)
`q6kQ8BatchMatMulKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
BATCHED Q6_K x Q8_K, the twin of #q4kQ8BatchMatMulKernel and needed for the same reason it is: Q4_K_M is a MIXTURE. (batched decode n in 2..8: llama.cpp takes mmvq up to MMVQ_MAX_BATCH_SIZE 8)

Cooperative-matrix GEMMs and tiles: prefill and expert GEMMs (llama.cpp mmq, mul_mat_id) (107)

kernel	llama.cpp counterpart	cuda	cpu	hip
`attnFlashPrefillF16Kernel`	fattn.cu:flash_attn_ext (the mma-f16 path)
The online flash prefill at llama.cpp's precision: the tile, block walk and softmax of #attnFlashPrefillOnlineKernel, with the query, keys, probabilities and values all plain f16 into f32 accumulators, as llama.cpp's mma flash attention takes them. (the online prefill tile at llama.cpp's precision: query, probabilities and values plain f16 into f32 accumulators, no residual pairs, the query held in registers, O rescaled in its fragments (15.2.15.3.7))
`attnFlashPrefillOnlineKernel`	fattn.cu:flash_attn_ext (the mma-f16 path)	✓	✓	✗
The single-pass flash prefill (xpu-kernel-adaptor 1.8.3): the query tile of #attnFlashPrefillTileKernel (64 (query, head) pairs on one K/V head, hd 128), but ONE pass over the keys with the online softmax, in 64-key blocks. (the single-pass online-softmax prefill tile (1.8.3): one Q.K^T with the query residual pair, P in f16, four barriers a 64-key block; llama.cpp's mma flash attention)
`f16F16CoopX3Kernel`	mmf.cu:mul_mat_f	✓	✓	✓
F16 coop GEMM, 32x64 warp tile: the staging is a copy of the row's halves, the rest is the TQ2_0 X3 body. (f16 x f16 GEMM)
`IdTile_128_128_64_32_128_q4kId`	mmq.cu:mul_mat_q (MUL_MAT_ID)	✓	✓	✓
IdTile<128, 128, 64, 32, 128>: a member of the tile family, one body written over its shape (MoE id tile; the reference model is dense)
`IdTile_64_128_64_16_128_q4kId`	mmq.cu:mul_mat_q (MUL_MAT_ID)	✓	✓	✓
IdTile<64, 128, 64, 16, 128>: a member of the tile family, one body written over its shape (MoE id tile; the reference model is dense)
`iq2sF16CoopN256Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
6.4.2's doubled token tile for IQ2S — the X3 body with a 256-token tile, so each staged slab is dequantized once per 256 tokens. (format not in the reference model; judged at the q4_K prefill shapes)
`iq2sF16CoopN64Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
6.4.3's HALVED token tile for IQ2S — an MoE expert gets about 34 of 512 tokens and the 128 tile pays for all 128, so a ragged batch takes 64 and reads the slab twice instead. (format not in the reference model; judged at the q4_K prefill shapes)
`iq2sF16CoopX3Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
The IQ2_S coop GEMM: raw sign bytes and a ten-bit index, the codebook staged in LDS at entry. (format not in the reference model; judged at the q4_K prefill shapes)
`iq2xsF16CoopN256Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
6.4.2's doubled token tile for IQ2XS — the X3 body with a 256-token tile, so each staged slab is dequantized once per 256 tokens. (format not in the reference model; judged at the q4_K prefill shapes)
`iq2xsF16CoopN64Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
6.4.3's HALVED token tile for IQ2XS — an MoE expert gets about 34 of 512 tokens and the 128 tile pays for all 128, so a ragged batch takes 64 and reads the slab twice instead. (format not in the reference model; judged at the q4_K prefill shapes)
`iq2xsF16CoopX3Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
The IQ2_XS coop GEMM: nine-bit grid index and seven-bit sign index per group, a scale byte pair per 64. (format not in the reference model; judged at the q4_K prefill shapes)
`iq2xxsF16CoopN256Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
6.4.2's doubled token tile for IQ2XXS — the X3 body with a 256-token tile, so each staged slab is dequantized once per 256 tokens. (format not in the reference model; judged at the q4_K prefill shapes)
`iq2xxsF16CoopN64Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
6.4.3's HALVED token tile for IQ2XXS — an MoE expert gets about 34 of 512 tokens and the 128 tile pays for all 128, so a ragged batch takes 64 and reads the slab twice instead. (format not in the reference model; judged at the q4_K prefill shapes)
`iq2xxsF16CoopX3Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
IQ2_XXS coop GEMM, 32x64 warp tile: the workgroup stages the codebook in LDS at entry, then decodes a sub-block per lane per step. (format not in the reference model; judged at the q4_K prefill shapes)
`iq3sF16CoopN256Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
6.4.2: the IQ3_S coop GEMM with the token tile DOUBLED to 256, so each 128-row slab is read and dequantized once per 256 tokens instead of once per 128. (format not in the reference model; judged at the q4_K prefill shapes)
`iq3sF16CoopN64Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
6.4.3's HALVED token tile for IQ3S — an MoE expert gets about 34 of 512 tokens and the 128 tile pays for all 128, so a ragged batch takes 64 and reads the slab twice instead. (format not in the reference model; judged at the q4_K prefill shapes)
`iq3sF16CoopX3Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
The IQ3_S coop GEMM: two four-value grid words per group, their ninth index bit from qh. (format not in the reference model; judged at the q4_K prefill shapes)
`iq3xxsF16CoopIdN64Kernel`	mmq.cu:mul_mat_q (MUL_MAT_ID)	✓	✓	✓
6.4.4: the IQ3_XXS coop GEMM grouped over experts — the mul_mat_id shape for the coop family. (MoE id tile; the reference model is dense)
`iq3xxsF16CoopN256Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
6.4.2's doubled token tile for IQ3XXS — the X3 body with a 256-token tile, so each staged slab is dequantized once per 256 tokens. (format not in the reference model; judged at the q4_K prefill shapes)
`iq3xxsF16CoopN64Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
6.4.3's HALVED token tile for IQ3XXS — an MoE expert gets about 34 of 512 tokens and the 128 tile pays for all 128, so a ragged batch takes 64 and reads the slab twice instead. (format not in the reference model; judged at the q4_K prefill shapes)
`iq3xxsF16CoopX3Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
The IQ3_XXS coop GEMM: two four-value grid words per group of eight. (format not in the reference model; judged at the q4_K prefill shapes)
`iq4nlF16CoopIdN64Kernel`	mmq.cu:mul_mat_q (MUL_MAT_ID)	✓	✓	✓
The IQ4_NL twin of #iq3xxsF16CoopIdN64Kernel: the iq4nl N64 body, 32-element blocks (cols % 64 == 0 suffices), the same chunk-map prologue and guarded tail store. (MoE id tile; the reference model is dense)
`iq4nlF16CoopN64Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
6.4.3's HALVED token tile for IQ4NL — an MoE expert gets about 34 of 512 tokens and the 128 tile pays for all 128, so a ragged batch takes 64 and reads the slab twice instead. (format not in the reference model; judged at the q4_K prefill shapes)
`iq4nlF16CoopX1Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
7.5.2 — IQ4_NL coop GEMM, 64x32 warp tile. (format not in the reference model; judged at the q4_K prefill shapes)
`iq4nlF16CoopX3Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
7.5.2 — IQ4_NL coop GEMM, 32x64 warp tile. (format not in the reference model; judged at the q4_K prefill shapes)
`iq4xsF16CoopX1Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
7.5.2 — IQ4_XS coop GEMM, 64x32 warp tile. (format not in the reference model; judged at the q4_K prefill shapes)
`iq4xsF16CoopX3Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
7.5.2 — IQ4_XS coop GEMM, 32x64 warp tile. (format not in the reference model; judged at the q4_K prefill shapes)
`MwTile_128_128_64_32_128_q4kQ8`	mmq.cu:mul_mat_q	✓	✓	✓
MwTile<128, 128, 64, 32, 128>: a member of the tile family, one body written over its shape
`MwTile_64_64_32_32_128_q4kQ8`	mmq.cu:mul_mat_q	✓	✓	✓
MwTile<64, 64, 32, 32, 128>: a member of the tile family, one body written over its shape
`mxfp4MatVecKernelCoop`	mmq.cu:mul_mat_q	✓	✓	✓
Wave-cooperative MXFP4 mat-vec (Unit 32, unit 7). (format not in the reference model; judged at the q4_K prefill shapes)
`mxfp4MatVecKernelCoopQ8`	mmq.cu:mul_mat_q	✓	✓	✓
Wave-cooperative MXFP4 mat-vec over Q8-quantized activations (the llama.cpp mxfp4xq8 dot). (format not in the reference model; judged at the q4_K prefill shapes)
`mxfp4MatVecKernelCoopQ8Surface`	mmq.cu:mul_mat_q	✓	✓	✓
The Q8 coop MXFP4 mat-vec RE-EXPRESSED on the cajeta.xpu cooperative-tile surface (xpu-cooperative-tile spec §8.1, Phase A Unit 4 — witness A). (format not in the reference model; judged at the q4_K prefill shapes)
`q2kF16CoopX1Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
Q2_K coop GEMM, 64x32 warp tile. (format not in the reference model; judged at the q4_K prefill shapes)
`q2kF16CoopX3Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
Q2_K coop GEMM, 32x64 warp tile. (format not in the reference model; judged at the q4_K prefill shapes)
`q2kWmmaDeqMw8Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
10.12.45.D — the Q2_K T=8 deq GEMM: Q6's slice geometry (16 of 16) in a 2x4 wave tiling — four weight fragments and two token fragments feed eight mmas per 16-k step — carrying Q4's TWO-FACTOR drain. (format not in the reference model; judged at the q4_K prefill shapes; dequantizes to f16 then WMMA; same op as MMQ)
`Q2Tile_128_128_32_64_128_q2kQ8`	mmq.cu:mul_mat_q
Q2Tile<128, 128, 32, 64, 128>: a member of the tile family, one body written over its shape (Q6Tile's staging and mma with Q2_K's two-factor drain over the raw 84-byte blocks: per-16 integer scales folded per mma, the per-16 mins a rank-1 update against in-kernel activation sums; llama.cpp MMQ (15.2.15.3.4))
`Q2Tile_64_64_32_32_128_q2kQ8`	mmq.cu:mul_mat_q
Q2Tile<64, 64, 32, 32, 128>: a member of the tile family, one body written over its shape (Q6Tile's staging and mma with Q2_K's two-factor drain over the raw 84-byte blocks: per-16 integer scales folded per mma, the per-16 mins a rank-1 update against in-kernel activation sums; llama.cpp MMQ (15.2.15.3.4))
`q3kF16CoopX1Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
Q3_K coop GEMM, 64x32 warp tile. (format not in the reference model; judged at the q4_K prefill shapes)
`q3kF16CoopX3Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
Q3_K coop GEMM, 32x64 warp tile. (format not in the reference model; judged at the q4_K prefill shapes)
`q3kWmmaDeqMw8Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
10.12.45.D — the Q3_K T=8 deq GEMM: the Q6 Mw8 kernel's exact shape (16 slices of 16, a 2x4 wave tiling so four weight fragments and two token fragments feed eight mmas per 16-k step) with Q3's header: d at 108 and the 16 signed six-bit scales unpacked from 12 bytes at staging, split across the lane halves like the Q6 kernel's prefold. (format not in the reference model; judged at the q4_K prefill shapes; dequantizes to f16 then WMMA; same op as MMQ)
`Q3Tile_128_128_32_64_128_q3kQ8`	mmq.cu:mul_mat_q
Q3Tile<128, 128, 32, 64, 128>: a member of the tile family, one body written over its shape (the Q6_K tile with Q3_K's widen over the resident (split) payload: signed 3-bit values, sixteen-element scales folded per mma; llama.cpp MMQ (15.2.15.3.4))
`Q3Tile_64_64_32_32_128_q3kQ8`	mmq.cu:mul_mat_q
Q3Tile<64, 64, 32, 32, 128>: a member of the tile family, one body written over its shape (the Q6_K tile with Q3_K's widen over the resident (split) payload: signed 3-bit values, sixteen-element scales folded per mma; llama.cpp MMQ (15.2.15.3.4))
`q40F16CoopX1Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
Q4_0 coop GEMM, 64x32 warp tile. (format not in the reference model; judged at the q4_K prefill shapes)
`q40F16CoopX3Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
Q4_0 coop GEMM, 32x64 warp tile. (format not in the reference model; judged at the q4_K prefill shapes)
`q41F16CoopX1Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
7.5.1 — Q4_1 coop GEMM, 64x32 warp tile. (format not in the reference model; judged at the q4_K prefill shapes)
`q41F16CoopX3Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
7.5.1 — Q4_1 coop GEMM, 32x64 warp tile. (format not in the reference model; judged at the q4_K prefill shapes)
`q4kF16CoopKernel`	mmq.cu:mul_mat_q	✓	✓	✓
The f16-COOPMAT Q4_K GEMM (matrix cores; llama.cpp's mul_mm large config): BM=128 weight rows x BN=128 tokens per 128-thread WG (4 wave32 subgroups, one 64x64 quadrant each = 16 fragments of 16x16x16 f16 mma, f32 accumulate), BK=32 k-slice per stage.
`q4kF16CoopM64Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
The BM=64 x BN=64 "m" variant of #q4kF16CoopKernel — llama.cpp's shape selection: the large tile starves the machine when the launch is narrow (k/v projections at out=1024 put 32 workgroups on 20 WGPs), so narrow shapes take 4x the workgroups at the same thread count.
`q4kF16CoopN256GKernel`	mmq.cu:mul_mat_q	✓	✓	✓
The B tile SKIPS LDS: the staged activations are already f16 and row-major [token][k], which is exactly the column-major fragment the sb tile existed to produce — so B comes straight out of global memory.
`q4kF16CoopN256Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
The token tile DOUBLED to 256 (256 threads, 8 warps of 64x64, BK64).
`q4kF16CoopX1Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
Stage 9a — HALF THE ACCUMULATORS.
`q4kF16CoopX3Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
Stage 9b — X1's transpose: eight accumulators again, but as a 32x64 warp tile (2 A fragments, 4 B fragments) instead of 64x32.
`q4kQ8MmqKernel`	mmq.cu:mul_mat_q	✓	✓	✓
BATCHED Q4_K, LDS-TILED — llama.cpp's mul_mmq shape adapted to this DSL.
`q4kWmmaDeqEpiKernel`	mmq.cu:mul_mat_q	○	○	○
Q4_K epi GEMM over the tile-major widened int8 copy — the int8 prefill-weight mode's flagship. (epilogue fused into the tile; the counterpart op is the bare MUL_MAT; dequantizes to f16 then WMMA; same op as MMQ)
`q4kWmmaDeqKernel`	mmq.cu:mul_mat_q	○	✓	○
The deq GEMM (10.12.16): the staged kernel with the B feed moved to the widened TILE-MAJOR global copy. (dequantizes to f16 then WMMA; same op as MMQ)
`q4kWmmaDeqMw4Kernel`	mmq.cu:mul_mat_q	○	○	○
10.12.21 token blocking — the Mw kernel with FOUR token-tiles per workgroup (64 tokens x 128 rows). (dequantizes to f16 then WMMA; same op as MMQ)
`q4kWmmaDeqMw8Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
dequantizes to f16 then WMMA; same op as MMQ
`q4kWmmaDeqMwKernel`	mmq.cu:mul_mat_q	✓	–	✓
THE MULTI-WAVE deq GEMM (10.12.21 FINAL) — all three levers the six single-lever probes showed only pay TOGETHER: 1. (dequantizes to f16 then WMMA; same op as MMQ)
`q4kWmmaDeqN4Kernel`	mmq.cu:mul_mat_q	○	○	○
N=4 OUTPUT-TILED deq GEMM (10.12.21 second iteration) — the counter pass's answer. (dequantizes to f16 then WMMA; same op as MMQ)
`q4kWmmaDeqT2Kernel`	mmq.cu:mul_mat_q	○	○	○
REGISTER-TILED T=2 deq GEMM (10.12.21): one wave computes TWO 16-token tiles of the same output-tile column, so every B fragment is loaded ONCE and mma'd twice — B traffic and the per-j fold/epilogue overhead halve per token. (dequantizes to f16 then WMMA; same op as MMQ)
`q4kWmmaEpiKernel`	mmq.cu:mul_mat_q	○	○	○
Q4_K epi GEMM with the LDS-staged B widen — the packed mode. (epilogue fused into the tile; the counterpart op is the bare MUL_MAT)
`q4kWmmaIdEpiKernel`	mmq.cu:mul_mat_q (MUL_MAT_ID)	○	○	○
── Unit 42 (spec 15.21.4): the id-GEMM on the EPI seam. (MoE id tile; the reference model is dense; epilogue fused into the tile; the counterpart op is the bare MUL_MAT)
`q4kWmmaIdKernel`	mmq.cu:mul_mat_q (MUL_MAT_ID)	○	○	○
── Unit 41d (spec 15.21.4): the indirected GEMM — llama.cpp's mul_mat_id at batch width. (MoE id tile; the reference model is dense)
`q4kWmmaIdMwKernel`	mmq.cu:mul_mat_q (MUL_MAT_ID)	✓	✓	✓
MoE id tile; the reference model is dense
`q4kWmmaKernel`	mmq.cu:mul_mat_q	✓	✓	✓
y[rows x outDim] = X .
`q50F16CoopX1Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
Q5_0 coop GEMM, 64x32 warp tile. (format not in the reference model; judged at the q4_K prefill shapes)
`q50F16CoopX3Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
Q5_0 coop GEMM, 32x64 warp tile. (format not in the reference model; judged at the q4_K prefill shapes)
`q51F16CoopX1Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
7.5.1 — Q5_1 coop GEMM, 64x32 warp tile. (format not in the reference model; judged at the q4_K prefill shapes)
`q51F16CoopX3Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
7.5.1 — Q5_1 coop GEMM, 32x64 warp tile. (format not in the reference model; judged at the q4_K prefill shapes)
`Q5IdTile_128_128_32_64_128_q5kId`	mmq.cu:mul_mat_q (MUL_MAT_ID)	✓
Q5IdTile<128, 128, 32, 64, 128>: a member of the tile family, one body written over its shape (the Q5_K Id tile: the dense Q5_K tile's design over the expert chunk maps (15.2.15.4.2))
`Q5IdTile_64_64_32_32_128_q5kId`	mmq.cu:mul_mat_q (MUL_MAT_ID)	✓
Q5IdTile<64, 64, 32, 32, 128>: a member of the tile family, one body written over its shape (the Q5_K Id tile: the dense Q5_K tile's design over the expert chunk maps (15.2.15.4.2))
`q5kF16CoopX1Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
Q5_K coop GEMM, 64x32 warp tile. (format not in the reference model; judged at the q4_K prefill shapes)
`q5kF16CoopX3Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
Q5_K coop GEMM, 32x64 warp tile. (format not in the reference model; judged at the q4_K prefill shapes)
`q5kWmmaIdMwKernel`	mmq.cu:mul_mat_q (MUL_MAT_ID)	✓
#q4kWmmaIdMwKernel for Q5_K expert banks (cajeta-llm 15.2.15.4.2): the same multi-wave shape over 176-byte blocks — Q4_K's header and nibble layout (the nibbles at +48) plus the 32-byte fifth-bit plane at +16, so a B word is the nibble word OR the plane word's bit j spread to bit 4 of every byte; dsc and -dminmn fold as for Q4_K. (MoE id tile; the reference model is dense)
`Q5Tile_128_128_32_64_128_q5kQ8`	mmq.cu:mul_mat_q	✓
Q5Tile<128, 128, 32, 64, 128>: a member of the tile family, one body written over its shape (the Q4_K tile's Q5_K twin over the raw 176-byte blocks, the fifth-bit plane folded at staging (15.2.15.3.4))
`Q5Tile_64_64_32_32_128_q5kQ8`	mmq.cu:mul_mat_q	✓
Q5Tile<64, 64, 32, 32, 128>: a member of the tile family, one body written over its shape (the Q4_K tile's Q5_K twin over the raw 176-byte blocks, the fifth-bit plane folded at staging (15.2.15.3.4))
`Q6IdTile_128_128_32_64_128_q6kId`	mmq.cu:mul_mat_q (MUL_MAT_ID)
Q6IdTile<128, 128, 32, 64, 128>: a member of the tile family, one body written over its shape (the Q6_K Id tile: the dense Q6_K tile's design over the expert chunk maps (15.2.15.4.2))
`Q6IdTile_64_64_32_32_128_q6kId`	mmq.cu:mul_mat_q (MUL_MAT_ID)
Q6IdTile<64, 64, 32, 32, 128>: a member of the tile family, one body written over its shape (the Q6_K Id tile: the dense Q6_K tile's design over the expert chunk maps (15.2.15.4.2))
`q6kF16CoopKernel`	mmq.cu:mul_mat_q	✓	✓	✓
The Q6_K twin of #q4kF16CoopKernel: identical tile/compute structure, staging swapped for the 212-byte PADDED layout the mmq twin reads (53 dwords: ql 0..31, qh 32..47, 16 int8 scales 48..51, d f16 in the row's scale prefix).
`q6kF16CoopM64Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
The BM=64 x BN=64 variant of #q6kF16CoopKernel — see #q4kF16CoopM64Kernel for the shape rationale.
`q6kF16CoopN256GKernel`	mmq.cu:mul_mat_q	✓	✓	✓
The B tile SKIPS LDS: the staged activations are already f16 and row-major [token][k], which is exactly the column-major fragment the sb tile existed to produce — so B comes straight out of global memory.
`q6kF16CoopN256Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
The token tile DOUBLED to 256 (256 threads, 8 warps of 64x64, BK64).
`q6kF16CoopX1Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
The Q6_K twin of #q4kF16CoopX1Kernel: the same eight-accumulator 64x32 warp tile that took the q4 kernel off scratch memory, over Q6_K's 212-byte padded staging.
`q6kF16CoopX3Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
The Q6_K twin of #q4kF16CoopX3Kernel — 32x64 warp tile.
`q6kQ8MmqKernel`	mmq.cu:mul_mat_q	✓	✓	✓
BATCHED Q6_K, LDS-TILED — #q4kQ8MmqKernel's twin over the resident split layout (Unit 9).
`q6kWmmaDeqMw4Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
10.12.27 — the Q6_K token-blocked multi-wave deq GEMM: the Mw4 transform applied to q6kWmmaEpiKernel, which the profile put at ITS memory wall (49 GB packed / 323 ms = 152 GB/s, one 16-token tile per WG re-reading and RE-WIDENING the matrix 32x). (dequantizes to f16 then WMMA; same op as MMQ)
`q6kWmmaDeqMw8Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
10.12.39 — the Q6_K T=8 twin: the Q4 Mw8 retile ported. (dequantizes to f16 then WMMA; same op as MMQ)
`q6kWmmaEpiKernel`	mmq.cu:mul_mat_q	✓	✓	✓
Q6_K epi GEMM: signed widen, one scale per K-slice, no dmin. (epilogue fused into the tile; the counterpart op is the bare MUL_MAT)
`q6kWmmaIdEpiKernel`	mmq.cu:mul_mat_q (MUL_MAT_ID)	○	○	○
MoE id tile; the reference model is dense; epilogue fused into the tile; the counterpart op is the bare MUL_MAT
`q6kWmmaIdKernel`	mmq.cu:mul_mat_q (MUL_MAT_ID)	○	○	○
MoE id tile; the reference model is dense
`q6kWmmaIdMwKernel`	mmq.cu:mul_mat_q (MUL_MAT_ID)	○	○	○
MoE id tile; the reference model is dense
`q6kWmmaKernel`	mmq.cu:mul_mat_q	○	○	○
Q6_K x q8_K on the MATRIX CORES (plan 10.12.23) — the 10.12.14 structure with Q6_K's layout, routed for attn_v and the Q6_K ffn_down projections that 10.12.22 measured at 1290 ms on the dotAccum fallback (32 calls costing ~as much as all 192 WMMA calls).
`Q6Tile_128_128_32_64_128_q6kQ8`	mmq.cu:mul_mat_q	✓	–	✗
Q6Tile<128, 128, 32, 64, 128>: a member of the tile family, one body written over its shape (the Q4_K tile's Q6_K twin over the resident (split) payload: a sixteen-element scale granularity folds per mma; llama.cpp MMQ)
`Q6Tile_64_64_32_32_128_q6kQ8`	mmq.cu:mul_mat_q	✓
Q6Tile<64, 64, 32, 32, 128>: a member of the tile family, one body written over its shape (the Q4_K tile's Q6_K twin over the resident (split) payload: a sixteen-element scale granularity folds per mma; llama.cpp MMQ)
`q80F16CoopX1Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
Q8_0 coop GEMM, 64x32 warp tile. (format not in the reference model; judged at the q4_K prefill shapes)
`q80F16CoopX3Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
Q8_0 coop GEMM, 32x64 warp tile. (format not in the reference model; judged at the q4_K prefill shapes)
`q80WmmaDeqMw8Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
10.12.45.D — the Q8_0 T=8 deq GEMM: the Q4 Mw8 structure with the simplest possible drain — no mins, no rank-1 term, cf per j-slice is just the f16 d of Q8_0 block b*8 + j (34-byte stride). (format not in the reference model; judged at the q4_K prefill shapes; dequantizes to f16 then WMMA; same op as MMQ)
`Q8Tile_128_128_32_64_128_q80Q8`	mmq.cu:mul_mat_q
Q8Tile<128, 128, 32, 64, 128>: a member of the tile family, one body written over its shape (the Q6_K tile's staging and mma over Q8_0's resident (split) rows: int8 copied straight to the tile, the per-32 float d folded every block; llama.cpp MMQ (15.2.15.3.4))
`Q8Tile_64_64_32_32_128_q80Q8`	mmq.cu:mul_mat_q
Q8Tile<64, 64, 32, 32, 128>: a member of the tile family, one body written over its shape (the Q6_K tile's staging and mma over Q8_0's resident (split) rows: int8 copied straight to the tile, the per-32 float d folded every block; llama.cpp MMQ (15.2.15.3.4))
`QkTile_128_128_32_64_128_q4kQ8`	mmq.cu:mul_mat_q	✓	–	✗
QkTile<128, 128, 32, 64, 128>: a member of the tile family, one body written over its shape (the tensor-core prefill tile (1.8.3): packed Q4_K staged as int8 in shared memory once per workgroup beside the q8_K activation half-block, double-buffered, prefetched; llama.cpp MMQ)
`QkTile_64_64_32_32_128_q4kQ8`	mmq.cu:mul_mat_q	✓	–	✗
QkTile<64, 64, 32, 32, 128>: a member of the tile family, one body written over its shape (the tensor-core prefill tile (1.8.3): packed Q4_K staged as int8 in shared memory once per workgroup beside the q8_K activation half-block, double-buffered, prefetched; llama.cpp MMQ)
`symWmmaDeqIdMw8Kernel`	mmq.cu:mul_mat_q (MUL_MAT_ID)	✓	–	✓
10.3: the widened-feed id GEMM at the dense kernel's tile -- 128 tokens x 128 rows per workgroup, the 4x2 wave tiling, A staged through LDS, B straight from the tile-major int8 slab -- over the 128-row chunk map. (MoE id tile; the reference model is dense; dequantizes to f16 then WMMA; same op as MMQ)
`symWmmaDeqMw8IdKernel`	mmq.cu:mul_mat_q (MUL_MAT_ID)	✓	✓	✓
Unit 9.4: the id-GEMM for the SYMMETRIC formats (Q8_0, Q5_0, Q4_0), over a bank slab widened ONCE to int8 with its compact f16 scale image -- 7.4's dense representation, indirected. (MoE id tile; the reference model is dense; dequantizes to f16 then WMMA; same op as MMQ)
`tq10F16CoopX1Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
TQ1_0 coop GEMM, 64x32 warp tile: the staging decodes the chunk's thirty-two base-three trits per lane. (format not in the reference model; judged at the q4_K prefill shapes)
`tq10F16CoopX3Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
TQ1_0 coop GEMM, 32x64 warp tile. (format not in the reference model; judged at the q4_K prefill shapes)
`tq20F16CoopX1Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
TQ2_0 coop GEMM, 64x32 warp tile: the Q8_0 X1 tile with the staging over 256-weight blocks — k-step s is chunk s % 4 of block s / 4, and the two staging lanes of a row take the two bit pairs that chunk spans. (format not in the reference model; judged at the q4_K prefill shapes)
`tq20F16CoopX3Kernel`	mmq.cu:mul_mat_q	✓	✓	✓
TQ2_0 coop GEMM, 32x64 warp tile (the Q8_0 X3 tile). (format not in the reference model; judged at the q4_K prefill shapes)

Scalar mat-vecs: the host-order reference shape (16)

kernel	llama.cpp counterpart	cuda	cpu	hip
`iq4nlMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
7.5.2 — IQ4_NL mat-vec, one work item per row. (format not in the reference model; judged at the q4_K shapes)
`iq4xsMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
7.5.2 — IQ4_XS mat-vec. (format not in the reference model; judged at the q4_K shapes)
`mxfp4MatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
mxfp4 (gpt-oss); not in the reference model
`mxfp4MatVecKernelFast3`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
SIMD MXFP4 mat-vec, one work item per row (PORTABLE — correct on the CPU backend and every wave width). (mxfp4 (gpt-oss); not in the reference model)
`q40MatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
y[yOff + i] = W[i]·x for one row per work item, W packed Q4_0. (format not in the reference model; judged at the q4_K shapes)
`q41MatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
7.5.1 — y[yOff + i] = W[i]·x for one row per work item, W packed Q4_1. (format not in the reference model; judged at the q4_K shapes)
`q4kMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
y[yOff + i] = W[i]·x for one row per work item, W packed Q4_K.
`q4kMatVecKernelIL`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
q4kMatVecKernel over an INTERLEAVED weight (Quant.interleaveRows16).
`q4kQ8MatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
Q4_K weights against q8_K-QUANTIZED activations — the integer path.
`q4kQ8PackedMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
The integer path over ONE packed activation buffer (plan 10.1).
`q50MatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
y[yOff + i] = W[i]·x for one row per work item, W packed Q5_0. (format not in the reference model; judged at the q4_K shapes)
`q51MatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
7.5.1 — Q5_1: Q4_1's affine fold with Q5_0's fifth bit. (format not in the reference model; judged at the q4_K shapes)
`q5kMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
y[yOff + i] = W[i]·x for one row per work item, W packed Q5_K. (format not in the reference model; judged at the q4_K shapes)
`q6kMatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
y[yOff + i] = W[i]·x for one row per work item, W packed Q6_K.
`q8MatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
y[yOff + i] = W[i]·x for one row per work item, W packed Q8_0. (format not in the reference model; judged at the q4_K shapes)
`routerF32MatVecKernel`	mmvq.cu:mul_mat_vec_q	✓	✓	✓
The router mat-vec on the device (unit 40): logits[e] = W[e] . (format not in the reference model; judged at the q4_K shapes)

Attention, router, mixture-of-experts plumbing, relayout and element-wise kernels (92)

kernel	llama.cpp counterpart	cuda	cpu	hip
`addBiasRowsF32`	binbcast.cu:add	✓	✓	✓
y[r*n + i] += bias[i] — a projection bias broadcast over rows (spec 4.12). (bias add (with sigmoid for the gated form); llama.cpp fuses the unary separately)
`addF32`	binbcast.cu:add	✓	✓	✓
a += b elementwise on the device — the residual add.
`addSigmoidRowsF32`	binbcast.cu:add	✓	✓	✓
The shared expert's tail: ob[r][j] += sigma(z[r]) * dn[r][j], with gated == 0 standing for a model that binds no gate vector and therefore scales by one. (bias add (with sigmoid for the gated form); llama.cpp fuses the unary separately)
`argmaxRowKernel`	argmax.cu:argmax	✓	✓	✓
HostOps.argmaxRow on the device: tok[0] = the smallest index of the largest value in v[0, n).
`attendF16`	fattn.cu:flash_attn_ext	○	○	○
#attendF32 at f16 storage — accumulation stays f32 (5.7). (attention; the reference model is GQA-4 (32 heads over 8 kv heads))
`attendF32`	fattn.cu:flash_attn_ext	✓	✓	✓
Ragged-batch attention kernel (spec 6.13): every query row carries its batch item (seqOf) and absolute position (qPos); item KV lives at row base kvBase[item] of the PACKED k/v buffers. (attention; the reference model is GQA-4 (32 heads over 8 kv heads))
`attnCombineKernel`	fattn.cu:flash_attn_ext	✓	✓	✓
Softmax scores[h][0..count) in place, then out[h][d] = sum_t p[t] * V[from + t][kvHeadOf(h)][d]. (unfused score/combine pair; the counterpart is the fused op, so the pair is judged together)
`attnCombinePrefillKernel`	fattn.cu:flash_attn_ext	✓	✓	✓
Prefill softmax and V combine: one workgroup of 128 per (query row, head), over that row's causal range. (unfused score/combine pair; the counterpart is the fused op, so the pair is judged together)
`attnCombinePrefillV2Kernel`	fattn.cu:flash_attn_ext	✓	✓	✓
Prefill combine v2 (10.12.38): ONE WAVE per (query row, head) with lane l owning output dims [4l, 4l+3] — the b128 twin of the v2 score kernel. (unfused score/combine pair; the counterpart is the fused op, so the pair is judged together)
`attnFlashDecodeGqa1Kernel`	fattn.cu:flash_attn_ext	✓	✓	✓
Unit 9.6.6: flash decode for nH == nKv -- no KV sharing, hd 128. (attention; the reference model is GQA-4 (32 heads over 8 kv heads))
`attnFlashDecodeGqa4BatchKernel`	fattn.cu:flash_attn_ext	✓
#attnFlashDecodeGqa4Kernel for every row of a decode batch in one launch: workgroup r * nKv * maxSplits + w, row r's slot base, length and splits from rowTab[4r]. (attnFlashDecodeGqa4Kernel for every row of a decode batch in one launch (15.2.12.3.2.2))
`attnFlashDecodeGqa4Kernel`	fattn.cu:flash_attn_ext	✓	✓	✓
Split-S flash decode, GQA x4, hd == 128: wave (kvh, sp) runs the online-softmax recurrence over its position chunk for the four q-heads of kv group kvh, and writes one PARTIAL per head — the unscaled accumulator quad plus that chunk's (m, l) — for #attnFlashDecodeReduceKernel to combine. (attention; the reference model is GQA-4 (32 heads over 8 kv heads))
`attnFlashDecodeGqa8BatchKernel`	fattn.cu:flash_attn_ext	✓
#attnFlashDecodeGqa8Kernel for every row of a decode batch in one launch (cajeta-llm 15.2.15.4.1): that kernel's body (unit 57's vector loads, four positions a trip) with workgroup r * nKv * 2 * maxSplits + w, row r's slot base, length and splits from rowTab[4r]; two waves a kv head carrying four q-heads each, part indexed by q-head so the x4 reduce serves it. (attnFlashDecodeGqa8Kernel for every row of a decode batch in one launch (15.2.15.4.1))
`attnFlashDecodeGqa8Kernel`	fattn.cu:flash_attn_ext	✓	✓	✓
Flash decode attend, GQA x8 (unit 40): the ratio the 30B actually has (32 q heads / 4 kv heads), which the x4 gate silently dropped to the two-kernel scalar path — measured at ~108 us/layer of the decode token where this kernel's shape costs ~25. (attention; the reference model is GQA-4 (32 heads over 8 kv heads))
`attnFlashDecodeReduceKernel`	fattn.cu:flash_attn_ext	✓	✓	✓
Combine the split partials: one wave per head. (attention; the reference model is GQA-4 (32 heads over 8 kv heads))
`attnFlashPrefillGqa4Kernel`	fattn.cu:flash_attn_ext	✓	✓	✓
Flash prefill attend, GQA x4 (10.12.43): ONE WAVE serves all FOUR q-heads of a kv group. (attention; the reference model is GQA-4 (32 heads over 8 kv heads))
`attnFlashPrefillKernel`	fattn.cu:flash_attn_ext	✓	✓	✓
Flash prefill attend (10.12.43): the v2 score/combine PAIR fused into one online-softmax kernel — same wave-per-(row,head) shape, same b128 lane mapping (lane l owns dims [4l, 4l+3]), but the scores never touch memory. (attention; the reference model is GQA-4 (32 heads over 8 kv heads))
`attnFlashPrefillTileKernel`	fattn.cu:flash_attn_ext	✓	✓	✓
11.2: the query-tiled flash prefill -- 64 (query, head) pairs per workgroup on one K/V head, K/V through LDS in 32-key batches, f16 WMMA on (value, 2^10 residual) pairs read back from LDS; hd 128. (attention; the reference model is GQA-4 (32 heads over 8 kv heads))
`attnReducePackQ8BatchKernel`	NONE	✓
#attnReducePackQ8Kernel for every row of a decode batch in one launch: workgroup r * (nH / 2) + blk, row r's splits from rowTab[4r + 3], its packed row at r * packStride. (attnReducePackQ8Kernel for every row of a decode batch in one launch (15.2.12.3.2.2))
`attnReducePackQ8Kernel`	NONE	✓	✓	✓
Unit 50: the split reduce AND the Q8_K pack of the attention output in one launch; the reduce launch is gone and the attention output never lands in f32. (fused attention reduce + q8 pack of the output; judged by roofline)
`attnScoreKernel`	fattn.cu:flash_attn_ext	✓	✓	✓
scores[h][t] = scale * (q[h] . (unfused score/combine pair; the counterpart is the fused op, so the pair is judged together)
`attnScorePrefillKernel`	fattn.cu:flash_attn_ext	✓	✓	✓
Prefill scores: one wave per (query row, head, key position). (unfused score/combine pair; the counterpart is the fused op, so the pair is judged together)
`attnScorePrefillRowKernel`	fattn.cu:flash_attn_ext	✓	✓	✓
Prefill scores, reshaped (10.12.29, attempt 2): ONE single-wave workgroup per (query row, head), key positions looped INSIDE. (unfused score/combine pair; the counterpart is the fused op, so the pair is judged together)
`attnScorePrefillRowV2Kernel`	fattn.cu:flash_attn_ext	✓	✓	✓
Prefill scores v2 (10.12.38): the SAME wave-per-(row,head) shape as #attnScorePrefillRowKernel, with the operators the profile said to fix. (unfused score/combine pair; the counterpart is the fused op, so the pair is judged together)
`axpbyF32`	binbcast.cu:add	✓	✓	✓
dst = betadst + wsrc — the expert combine (unit 39).
`blockPadKernel`	NONE	✓	✓	✓
Generic block byte-pad, the retired Q6 pad without the hard-coded 210/212 — for Q3_K (110 -> 112), whose hmask/qs/scales/d already land on dword boundaries and only need the block STRIDE made a multiple of four. (cajeta's resident split layout; llama.cpp stores blocks whole; judged by roofline)
`blockRepack2Kernel`	NONE	✓	✓	✓
Block REPACK for the legacy 32-value formats. (cajeta's resident split layout; llama.cpp stores blocks whole; judged by roofline)
`concatI8Kernel`	concat.cu:concat	✓	✓	✓
10.12.32: device-side byte copy at an offset — builds the fused packed weight buffers (q\|k\|v, gate\|up) from the resident per-tensor copies at device bandwidth, once, at prewarm.
`copyF16Kernel`	cpy.cu:cpy_f16_f16	✓
dst[i] = src[i] for i < n, grid-stride over stride threads. (the KvPool growth copy (15.2.12.3.2.1))
`deqPadZeroKernel`	NONE	✓	✓	✓
7.4.3 — zero the k-tiles a padded widen does not write. (cajeta's resident split layout; llama.cpp stores blocks whole; judged by roofline)
`embedRowF16Kernel`	getrows.cu:get_rows	✓	✓	✓
The F16 row: 8-element blocks of halves (16 bytes). (the decode token's embedding row decoded on the device (12.2.11); llama.cpp get_rows per type)
`embedRowQ2kKernel`	getrows.cu:get_rows
The Q2_K row: { u8 scales[16]; u8 qs[64]; f16 d; f16 dmin }, 84 bytes a block. (token embedding row decoder for Q2_K tables (the 8B Q2_K files keep token_embd in it; 15.2.15.3.4))
`embedRowQ3kKernel`	getrows.cu:get_rows
The Q3_K row: { u8 hmask[32]; u8 qs[64]; u8 scales[12]; f16 d }, 110 bytes a block. (token embedding row decoder for Q3_K tables (the 8B Q3_K files keep token_embd in it; 15.2.15.3.4))
`embedRowQ4kKernel`	getrows.cu:get_rows	✓	✓	✓
hs[0..dim) = table[tok[0], :] for a Q4_K table of blocksPerRow 256-element blocks per row. (the decode token's embedding row decoded on the device (12.2.11); llama.cpp get_rows per type)
`embedRowQ5kKernel`	getrows.cu:get_rows
The Q5_K row: { f16 d; f16 dmin; u8 scales[12]; u8 qh[32]; u8 qs[128] }, Q4_K's header and nibbles (the nibbles 32 bytes later) plus a plane of fifth bits: element e of sub-block j carries bit j of qh[e % 32] (Quant.q5kBlock). (the Q5_K token embedding row decoded on the device, so Q5_K_M prompts batch across requests (2026-10-25); llama.cpp get_rows per type)
`embedRowQ6kKernel`	getrows.cu:get_rows	✓	✓	✓
The Q6_K row: { u8 ql[128]; u8 qh[64]; i8 scales[16]; f16 d }. (the decode token's embedding row decoded on the device (12.2.11); llama.cpp get_rows per type)
`embedRowQ80Kernel`	getrows.cu:get_rows	✓	✓	✓
The Q8_0 row: 32-element blocks of { f16 d; i8 q[32] }; lane t takes elements t, t + 256, ... (the decode token's embedding row decoded on the device (12.2.11); llama.cpp get_rows per type)
`f32CopyKernel`	cpy.cu:cpy_f32_f32	✓	✓	✓
10.12.45.B — device-to-device f32 copy.
`f32ToF16Kernel`	cpy.cu:cpy_f32_f16	✓	✓	✓
f32 -> f16 elementwise (activation conversion for the coop GEMM).
`gatherRowsF32Kernel`	getrows.cu:get_rows	✓	✓	✓
Unit 41: gather picked rows into group order — dst[i, :] = src[idx[i], :] over total = picks * cols elements.
`gluF32`	unary.cu:swiglu	✓	✓	✓
Fused gate activation × up: out[i] = act(g[i]) · u[i].
`gluFusedF32`	unary.cu:swiglu	✓	✓	✓
10.12.32: GLU over the FUSED gate\|up projection output — row r of src holds gate at [0, inter) and up at [inter, 2*inter).
`gluFusedPackF32`	unary.cu:glu + quantize.cu:quantize_q8_1	✓	✓	✓
The fused gate\|up GLU packed straight to q8_K (1.8.3): a workgroup of 32 lanes per 256-element block of the [rows x inter] result, eight elements a lane, reading gate at [r2inter + c) and up at + inter of the fused projection output and writing the block's quants, sums and scale as q8kPackKernel writes them (gluPackF32's tail), so the down projection's pack is this launch and the f32 GLU result is never written or read back (29 MB each way a layer at 512 tokens). (the fused gate\|up GLU packed straight to q8_K for the down projection (1.8.3); llama.cpp runs the GLU then quantizes the activation inside MUL_MAT)
`gluPackF32`	NONE	✓	✓	✓
The GLU and the Q8_K pack of its result in ONE launch: out takes the f32 row in place, as #gluF32 does, and packed the activation format the mat-vecs read. (fused GLU + q8 pack; judged by roofline)
`i8ReplicateRowsKernel`	cpy.cu:cpy_f32_f32	✓
dst[(r * reps + j)] = src[r] for rows of rowBytes bytes (cajeta-llm 15.2.15.4.1): each decode row's packed activation copied once per selected expert, so the id mat-vecs' per-selection activation base (selection kk reads row kk) serves a BATCH of rows with one launch a bank. (packed activation copied once per selected expert for a batched MoE decode (15.2.15.4.1))
`iqTableProbeKernel`	NONE	✓	✓	✓
Copy every resident table entry out, in the test's order. (codebook table read probe shipped in QuantKernel; judged by roofline)
`kvAppendKernel`	set-rows.cu:set_rows	✓	✓	✓
Write the already-rotated K row and the V row into row row of the per-layer device K/V planes. (KV cache write)
`kvAppendRangeKernel`	set-rows.cu:set_rows	✓	✓	✓
Copy count contiguous K/V rows into the device planes starting at fromRow. (KV cache write)
`moeCombineAddKernel`	moe-weighted-reduction.cu:moe_weighted_reduction	○	○	○
As #moeCombineKernel but ACCUMULATING into out — the residual add fused into the combine, one launch fewer per decode layer (unit 41e). (MoE expert combine; the reference model is dense)
`moeCombineAddRowsKernel`	moe-weighted-reduction.cu:moe_weighted_reduction	✓
#moeCombineAddKernel over nRows rows at once: row r's used expert outputs, weighted, accumulate onto its residual, in the one-row kernel's summation order (cajeta-llm 15.2.15.4.1). (moeCombineAddKernel over a batch of rows (15.2.15.4.1); the reference model is dense)
`moeCombineKernel`	moe-weighted-reduction.cu:moe_weighted_reduction	✓	✓	✓
out[i] = sum_k w[k] * rows[k*n + i] — the weighted expert combine over the down id-kernel's [used x n] output (unit 40's epilogue; deterministic, no atomics). (MoE expert combine; the reference model is dense)
`moeGroupKernel`	mmid.cu:mm_ids_helper	✓	✓	✓
Unit 45: the counting sort that groups a batch's picks by expert, ON THE DEVICE, in ONE workgroup — the host groupPicks reproduced exactly, including its order: picks land in their group in pick order, so the result is deterministic. (expert id grouping; llama.cpp sorts ids inside MUL_MAT_ID)
`moeGroupWaveKernel`	mmid.cu:mm_ids_helper
The grouping sort as a wave counting sort (cajeta-llm 15.2.15.4.2): the same output as #moeGroupKernel, order and maps included, without its per-expert walk of every pick (128 threads reading 4096 picks twice, 251 us a 30B layer against llama.cpp's mm_ids_helper at ~40). (expert id grouping as a wave counting sort (ballot ranks); same output as moeGroupKernel (15.2.15.4.2))
`moeScatterCombineKernel`	moe-weighted-reduction.cu:moe_weighted_reduction	✓	✓	✓
Unit 41: the weighted scatter-combine, deterministically — one workgroup per OUTPUT row walks its own picks, so no atomics and no ordering dependence: out[r, j] = sum_k gw[slot] * be[slot, j] with slot = rowSlot[r*used + k]. (MoE expert combine; the reference model is dense)
`moeTopKBatchKernel`	topk-moe.cu:topk_moe	✓	✓	✓
Unit 45: #moeTopKKernel over EVERY row of a batch — one wave per row, the same threshold-chained selection (the host routeFromLogits tie rule, lower index first), writing selAll[row*used + k] / wAll[...]. (MoE routing; the reference model is dense)
`moeTopKKernel`	topk-moe.cu:topk_moe	✓	✓	✓
Softmax + top-k + renorm on the device (unit 40), mirroring MoeFfn.routeInto EXACTLY: softmax over all experts (the weights are probabilities, so the full softmax is load-bearing), selection where ties keep the LOWER index, the f16-min divisor clamp on the renorm, then the optional scale. (MoE routing; the reference model is dense)
`mxfp4QuantActKernel`	quantize.cu:quantize_q8_1	✓	✓	✓
Quantize activations to per-32 Q8_0 — one GROUP per 32-element block: amax over the group, scale d = amax/127, quant = round(x/d). (activation quantization; llama.cpp does it inside MUL_MAT)
`q2kWidenKernel`	convert.cu:dequantize_block	✓	✓	✓
10.12.45.D — the Q2_K widen: the 2-bit value 0..3, unsigned, out to the tile-major layout; the per-16 scale AND min stay packed and ride the GEMM's cf/cg prefold. (widen to q8 for the batched path; llama.cpp dequantizes to f16 on the cuBLAS path instead)
`q3kWidenKernel`	convert.cu:dequantize_block	✓	✓	✓
10.12.45.D — the Q3_K widen: SIGNED (ql \| hbit<<2) - 4 out to the tile-major layout (the Q6 widen's signed-value precedent). (widen to q8 for the batched path; llama.cpp dequantizes to f16 on the cuBLAS path instead)
`q40WidenKernel`	convert.cu:dequantize_block	✓	✓	✓
The Q4_0 widen: nibble minus the implicit 8, which lands in [-8,7] — exactly representable, so the widened copy is lossless and the per-32 d rides the GEMM's cf prefold at an 18-byte stride. (widen to q8 for the batched path; llama.cpp dequantizes to f16 on the cuBLAS path instead)
`q4kWidenKernel`	convert.cu:dequantize_block	✓	✓	✓
10.12.16 — the ONE-TIME widen: packed Q4_K nibbles out to a TILE-MAJOR int8 copy holding the RAW nibble 0..15. (widen to q8 for the batched path; llama.cpp dequantizes to f16 on the cuBLAS path instead)
`q50WidenKernel`	convert.cu:dequantize_block	✓	✓	✓
The Q5_0 widen: the fifth bit comes out of the block's u32 qh — bit j for the low nibble of byte j, bit j+16 for the high — and the bias is 16, so the value lands in [-16,15]. (widen to q8 for the batched path; llama.cpp dequantizes to f16 on the cuBLAS path instead)
`q5kWidenKernel`	convert.cu:dequantize_block	✓	✓	✓
10.12.45.D — the Q5_K widen: (ql \| hbit<<4) out to the shared tile-major int8 layout. (widen to q8 for the batched path; llama.cpp dequantizes to f16 on the cuBLAS path instead)
`q6kScaleImageKernel`	NONE	○	–	○
10.3: the per-16 scale image of a Q6_K tensor for the widened wide-tile id GEMM -- d*sc as f32, sixteen per (row, 256-block). (cajeta's resident split layout; llama.cpp stores blocks whole; judged by roofline)
`q6kWidenKernel`	convert.cu:dequantize_block	✓	✓	✓
10.12.27 — the ONE-TIME Q6_K widen: signed q-32 values (they fit int8 exactly) out to the same TILE-MAJOR layout as the Q4 widen, so the Q6 deq GEMM shares the Q4 kernel's B addressing verbatim. (widen to q8 for the batched path; llama.cpp dequantizes to f16 on the cuBLAS path instead)
`q80WidenKernel`	convert.cu:dequantize_block	✓	✓	✓
10.12.45.D — the Q8_0 "widen": a pure tile-major re-layout — the values are ALREADY signed int8. (widen to q8 for the batched path; llama.cpp dequantizes to f16 on the cuBLAS path instead)
`q8kPackKernel`	quantize.cu:quantize_q8_1	✓	✓	✓
Quantize activations into Quant.q8kPack form ON THE DEVICE — the kernel that makes device residency possible (plan 10.2), and llama.cpp's quantize_row_q8_1_cuda in this engine's block layout. (activation quantization; llama.cpp does it inside MUL_MAT)
`qkNormRowsF32`	norm.cu:rms_norm_f32	○	○	○
The decode QK-norm PAIR in one launch (unit 41e): workgroups 0..nH-1 norm q's heads against qw, the rest norm k's heads against kw — the two rmsnormBatch launches this replaces were a fifth of the decode attention chain's small-kernel count.
`qkPrepBatchKernel`	NONE	✓
#qkPrepKernel for every row of a decode batch in one launch: workgroup r * (nH + nKv) + head, row r's slot base and position from rowTab[4r] and [4r + 1]. (qkPrepKernel for every row of a decode batch in one launch (15.2.12.3.2.2))
`qkPrepKernel`	NONE	✓	✓	✓
QK-norm, RoPE and the K/V append in ONE launch: a workgroup per head (q heads first, then k heads), 128 threads. (fused qk-norm + rope prep; judged by roofline)
`rmsnormF32`	norm.cu:rms_norm_f32	○	○	○
RMSNorm over rows of [rows, dim] (spec 5.1): normalize by root-mean-square over the last axis, learned scale, NO mean subtraction.
`rmsnormPackRowF32`	NONE	✓	✓	✓
Unit 48: RMSNorm and the Q8_K pack in ONE workgroup — the normed row lands in out (f32, for the router and the residual users) AND in packed (the mat-vec activation format), where the layer had a norm launch then a pack launch, twice. (fused rms-norm + q8 pack; judged by roofline)
`rmsnormRouterTopKKernel`	NONE	✓	✓	✓
Unit 59: the post-attention RMSNorm, its Q8_K pack AND the router + top-k in ONE launch. (fused rms-norm + router mat-vec + top-k; judged by roofline)
`rmsnormRowF32`	norm.cu:rms_norm_f32	✓	✓	✓
RMSNorm over ONE row already on the device (plan 10.2), as a WORKGROUP-COOPERATIVE reduction.
`rmsnormRowsF32`	norm.cu:rms_norm_f32	✓	✓	✓
One ROW per WORKGROUP, 256-lane cooperative reduction per row — the batched shape rocprofv3 demanded: the one-row-per-work-ITEM rmsnormF32 put 512 serial 4096-element reductions on 8 workgroups and measured 681 us per call, ~8.5x its memory floor.
`ropeAppendRowsKernel`	rope.cu:rope_neox + set-rows.cu:set_rows	✓	✓	✓
The prefill's RoPE and K/V append in ONE launch (1.8.3): a thread per (row, q head, rotation pair d < hd/2). (the prefill RoPE of q and k and the K/V append in one launch (1.8.3); llama.cpp ropes q and k then writes the cache rows)
`ropeDecodeKernel`	rope.cu:rope_neox	✓	✓	✓
Rotate t[nHeads x headDim] in place at absolute position pos.
`ropeF32`	rope.cu:rope_neox	✓	✓	✓
RoPE kernel over [rows, nHeads·headDim], positions per row.
`routeHistKernel`	mmid.cu:mm_ids_helper	✓	○	○
Unit 30: the routing histogram (spec 15.20). (expert id grouping; llama.cpp sorts ids inside MUL_MAT_ID)
`routerRowsF32Kernel`	mmvf.cu:mul_mat_vec_f	✓
The decode batch's router (cajeta-llm 15.2.15.4.1): one block per expert, as routerF32MatVecKernel, computing that expert's logit for EVERY row of the batch — W[e] is read once per row from L1 after the first, lanes stride the columns so the reads coalesce, and each row's sum is the one-row kernel's operation for operation (the same lane striding and Workgroup.reduce), so a batched row's logits are its solo logits bit for bit. (router f32 mat-vec over the decode batch's rows; the reference model is dense)
`routerTopKKernel`	topk-moe.cu:topk_moe	✓	✓	✓
Unit 50: the router mat-vec AND the top-k in ONE launch. (MoE routing; the reference model is dense)
`routerTopKWaveKernel`	topk-moe.cu:topk_moe	✓	✓	✓
Unit 53: the router + top-k as WAVES — one wave per expert row (lanes stride the 2048 columns four floats at a time, one wave reduction, no LDS, no barrier), the last wave to finish selects with all 128 logits held in four registers per lane: the softmax max and sum by wave reduction, then the same threshold-chained rounds as moeTopKKernel on registers. (MoE routing; the reference model is dense)
`scaleF32`	scale.cu:scale	✓	✓	✓

`sliceF32`	NONE	✓	✓	✓
10.12.35: copy n floats starting at srcOff — the last-row tail download uses it to bring home 16 KB instead of 8 MB. (a view in ggml, no kernel; judged by roofline)
`sliceI8Kernel`	NONE	✓	✓	✓
The read-side twin: copy n bytes starting at srcOff — the grid-sweep probe carves row-range slices with it. (a view in ggml, no kernel; judged by roofline)
`splitKernel`	NONE	–	–	–
The resident-layout split (codebook-quants spec §10): file block blockBase + b lands in its row's scale prefix and payload run of out. (cajeta's resident split layout; llama.cpp stores blocks whole; judged by roofline)
`splitPayloadKernel`	NONE	✓	✓	✓
The DWORD form of #splitKernel: one work item per payload dword rather than one wave per block. (cajeta's resident split layout; llama.cpp stores blocks whole; judged by roofline)
`splitRows3F32`	NONE	✓	✓	✓
10.12.32: scatter the fused q\|k\|v batch output into the three per-projection buffers — rope, append and attend stay untouched. (a view in ggml, no kernel; judged by roofline)
`splitScaleKernel`	NONE	✓	✓	✓
The scale half of the dword split: one work item per block, two bytes each, into the row's scale prefix. (cajeta's resident split layout; llama.cpp stores blocks whole; judged by roofline)
`splitScaleWordKernel`	NONE	✓	✓	✓
The DWORD form of #splitScaleKernel: one work item per dword of a row's scale prefix, so two blocks' f16 scales are assembled in registers and stored once through the word view. (cajeta's resident split layout; llama.cpp stores blocks whole; judged by roofline)
`symScaleImageKernel`	NONE	✓	✓	✓
7.4.2 — the COMPACT SCALE IMAGE: one f16 per 32-element block, colsPad/32 a row, zero in the pad. (cajeta's resident split layout; llama.cpp stores blocks whole; judged by roofline)
`tcSplitFixupKernel`	mmq.cu:mul_mat_q_stream_k_fixup	✓	✓	✓
The split-K fixup: y[t][c] is the sum over the splits of their partial tiles, one thread an element. (the split-K tiles' fixup: the partial tiles summed into y, one thread an element; llama.cpp's stream-K fixup)

Bench probes (22)

Kernels of the bench package and the in-tree probes, built into probe binaries to measure a mechanism (an LDS layout, a read pattern, a fragment load); never on a model’s route. The census classes them PROBE.

kernel
`coopLayoutProbeKernel`
Stage 5 — coopmat layout=1 (column-major) SEMANTICS, the two operations v0 sidestepped and llama.cpp's mul_mm depends on: a col-major B load (from a buffer AND from Shared) and a col-major accumulator store.
`dotCeilingKernel`
PURE dotAccum THROUGHPUT (plan 10.12.9) — no memory in the loop.
`f32MatVecKernel`
f32 weights, [out, in] row-major: the shape the DEVICE path already materializes today (Linear.bind disables packing when toDevice).
`flatReadKernel`
FULLY COALESCED: consecutive lanes read consecutive 16-byte chunks.
`ldsDotKernel`

`mmaF16Kernel`
f16 operands with an f32 accumulator — the pairing the skill names for AMD.
`mmaI8Kernel`
int8 operands with an int32 accumulator — what a quantized GEMM would actually want.
`q4kF16CoopC7Kernel`
Stage 7i — what caps the mma CEILING itself (65% of V7 at K=4096).
`q4kF16CoopCAKernel`
Stage 7h — WHERE THE TIME GOES, measured by varying only what each staging half WRITES, while the mma, the loop, the barriers and the LDS store count stay identical.
`q4kF16CoopCBKernel`

`q4kF16CoopCeilKernel`
Stage 7a — the mma CEILING: stage ONE tile, then run every k-iteration's mma against it.
`q4kF16CoopCLKernel`

`q4kF16CoopV1Kernel`
Stage 6 — the v1 f16-coopmat GEMM, llama.cpp's mul_mm large config verbatim (their 0.43 s prefill shape on this GPU): BM=128 weight rows x BN=128 tokens per workgroup of 128 threads (4 wave32 subgroups), BK=32 k-slice per stage.
`q4kF16CoopV7Kernel`
Stage 7i — V7 = V5 + BK64: two 32-slices staged per barrier pair (stride 72, LDS 36 KB, 4 mma k-steps), halving the barrier count.
`refusedBlockKernel`
21.1.6's forcing kernel on cuda and hip.
`rowReadKernel`
ROOFLINE: read a row and do almost nothing with it.
`skewReadKernel`
Read a row with 16-byte loads starting skew bytes into it.
`streamReadKernel`
The DRAM read ceiling: every lane streams 16-byte vectors, a wave 512 contiguous bytes per load, eight loads in flight per lane (4 KB per wave), the whole buffer covered — no compute, no reduction, the sum steered never to hit needle.
`tileMmaKernel`

`tileSplatKernel`

`touchLinesKernel`
Unit 55: the prefetch touch — one load per 64-byte line of a buffer, on a second stream, so a weight that a dependent launch will stream in a moment is already in the last-level cache when it does.
`wideReadKernel`
The same read with FOUR independent accumulators.

Standard-library kernels the binary registers (27)

Tensor kernels of the Cajeta standard library (cajeta.ml), linked into every program that uses it: element-wise arithmetic, reductions, matmuls, padding and cropping, bitonic sort and scan steps. The library does not launch them on a model’s route; the census classes them EXTERNAL and the standard library’s own suites hold them.

arithBf16, arithF16, arithF32, bincountI32, bitonicStepF32, copyRangeF32, cropBf16, cropF32, gatherF32, matmulBf16, matmulBf16Wide, matmulF16, matmulF32, matmulF64, matmulI8, narrowF32ToBf16, padBf16, padF16, padF32, reduceMaxF32, reduceMinF32, reduceSumBf16, reduceSumF16, reduceSumF32, reduceSumSqF32, scanStepF32, scatterF32

Fixtures and retired kernels (1)

kernel	where
`routerBatchF32Kernel`	removed 2026-10-24
the prefill's thread-per-expert router, replaced by `routerRowsF32Kernel`; the cpu and hip census tables still carry its row until their next leg

Source: docs/xpu/catalog-llm.md · 60 min read · 13294 words