docs / specification / xpu

The kernel reference interpreter

The reference interpreter runs a @Kernel’s source on the host and produces the answer every backend is compared against. It reads the same syntax tree the kernel lowering reads, but none of the lowering’s code. Its values carry their own width and signedness, and its arithmetic follows the host compiler. A backend that extends with the wrong sign, picks the wrong compare or drops a lane therefore disagrees with it.

#include "cajeta/xpu/reference/KernelInterpreter.h"
namespace ref = cajeta::xpu::reference;

ref::Launch l;
l.grid[0] = 4;
l.block[0] = 64;
ref::run(kernel, {ref::Arg::buffer(y), ref::Arg::buffer(x), ref::Arg::f32(2.5f)}, l);

A buffer argument is host memory that the interpreter reads and writes in place. Its element type is the parameter’s KernelBuffer<T>. A scalar argument is passed as its bits.

Execution model

• Workgroups run one after another, in x, then y, then z order.
• The survey resolves names that need the compiler’s type registries, such as enum constants and @Device helpers, before anything runs. Those registries are thread-local, so a work-item’s thread never looks them up.
• Within a workgroup, each work-item runs until it reaches a rendezvous or returns. A rendezvous is Barrier.workgroup, Barrier.wave or a wave collective. Once every live member of the rendezvous’s scope has arrived, the collective is computed and they all continue.
• A cooperative matrix is the wave’s object, so a tile operation that reads memory (load, fromWords, the epilogue verbs) is a wave collective: it reads once every lane of the wave has arrived, so what the lanes stored just before it, with no barrier between, is there, as on a wave that runs in lockstep. A kernel that uses a tile is a wave kernel and needs the width.
• Scheduling is deterministic, so a run is reproducible.
• A kernel with no rendezvous runs its work-items in sequence on one thread.
• Memory is sequentially consistent, so Barrier.workgroupMemory and Barrier.deviceMemory order nothing.
• Each Shared<T> declaration has one array per workgroup, which every work-item of that workgroup binds. A Swizzled<T, S> array is a Shared array: its XOR permutation of physical slots is an involution the program never observes, so logical index a is element a. A BlockPadded<T, Block, Pad> array is the same with the backend’s layout: element a lives at slot a + (a / Block) · Pad, the declared size counts slots, and the bound is on the slot, so a logical index that lands past the end is out of bounds.
• A work-item’s linear index is x + y·dimX + z·dimX·dimY.
• Lane and wave come from the linear index and the wave width: the lane is linear % W and the wave is linear / W. Lanes that have returned are inactive.
• The wave width is the launch’s waveWidth. When that is 0, the width comes from the kernel’s @Wave(width = N). A kernel that uses a wave operation with neither is refused, because its answer depends on the width.

Arithmetic

Arithmetic follows the host compiler’s rules (BinaryOpExpression).

• An unsuffixed integer literal is int32, or int64 if it is past int32 or has an L suffix. Next to a typed operand, a bare literal takes that operand’s type.
• Integer operands extend to the wider width, each by its own signedness. +, - and * wrap.
• /, % and the bitwise operators are unsigned when either operand is unsigned.
• A comparison is signed when either operand is signed.
• >> follows the signedness of the shifted operand alone. >>> is always logical.
• float32 work is done in float32, one rounding per operation, and nothing is contracted into a fused multiply-add.
• float16 and bfloat16 are computed in float32 and rounded back, to nearest even.
• An f suffix makes a literal float32. Any other float literal is float64.
• != on floats is true when either operand is NaN.
• A conversion from float to integer truncates toward zero.
• c ? a : b evaluates the condition and then ONE arm, as the kernel lowering branches; the untaken arm never reads.

Built-ins

These are every built-in the interpreter defines.

A tile is a wave-uniform value. Each work-item holds its own copy, and every copy is computed the same way from the same arguments.

What is refused, and what fails

Before any work-item runs, refusals(kernel) lists every construct the interpreter cannot define. Each entry names the construct and its line. If the list is not empty, run throws XPU-REF01 with the first entry, so the kernel is never run partially. Today that list includes:

• textures, images and samplers;
• value types;
• KernelThread.clock(), which has no reference value by nature;
• a helper with backend arms and no body (no portable arm).

The conformance corpus adds built-ins here as its kernels need them.

While running, undefined behaviour that the interpreter can see fails the run with XPU-REF02. The message names the operation and its line:

• an index out of bounds;
• a read of a Shared element that no work-item wrote;
• a shift by the width or more;
• integer division by zero, or MIN / -1;
• a float that converts to an integer out of range;
• a Barrier.workgroup that some work-items return without reaching;
• lanes of one wave that reach different wave operations;
• a shuffle from an inactive lane.

The conformance corpus

The corpus is real launches, recorded as a backend runs them, then replayed through the interpreter.

1. Run a program, such as cajeta-llm’s tests, with CAJETA_XPU_RECORD=<dir>. The runtime records the first CAJETA_XPU_RECORD_PER_KERNEL launches of each kernel (default 2). For each one it writes launch.json, giving the kernel, backend, grid, block, wave width and arguments, plus each allocation an argument points into, before (a<i>.in) and after (a<i>.out). An allocation is written once, however many arguments point into it, so aliasing survives the replay. The runtime skips a launch on a deferred stream, a Vulkan launch, and one whose allocations come to more than CAJETA_XPU_RECORD_MAX_BYTES (default 64 MiB). Each skip goes in <dir>/skipped.tsv with its reason.
2. Compile the same program with CAJETA_XPU_CONFORMANCE=<dir>, and optionally CAJETA_XPU_CONFORMANCE_HELD=<file>. The compiler replays every recording against its own kernels, writes <dir>/conformance.tsv and stops before generating code. It exits 1 when any launch fails.

Each launch gets one of these outcomes:

The held list has one tab-separated line per kernel and backend. <kernel> <backend> held <note> holds a known disagreement, and <kernel> <backend> ulps=N <why> states a float bound in units in the last place, and <kernel> <backend> rel=X <why> states one as a fraction of the buffer’s scale. The two can be joined as ulps=N,rel=X, and an element inside either passes. A backend of * matches every backend. On an integer buffer ulps=N is a step bound: the two elements’ distance as integers of the buffer’s width, for a quantized output whose float scale the backend reduced in another order.

The scale is the largest finite reference magnitude the launch wrote, over the whole buffer when it wrote nothing. An element the launch left as it was (a fill pattern, a row it did not reach) does not set it, and nor does a sentinel, a magnitude at or above a quarter of the type’s largest value (an attention kernel’s -3e38 for an empty partition). Both still have to match; they only do not make a relative bound vacuous.

With CAJETA_XPU_CONFORMANCE_DUMP set, a failed launch leaves the reference’s bytes of each allocation a parameter reads beside the backend’s, as a<i>.ref.

A relative bound is for results whose error scales with the buffer rather than with the element: a @FastMath kernel, which the backend may reassociate, contract and flush to zero, and a function such as sin near one of its zeros. There a count of ulps is huge for an error the buffer cannot see, and a ulp bound wide enough to admit it would also admit a real defect elsewhere. A float failure reports its first element, its worst distance in both measures, and how many elements fall outside the bound.

Comparing against it

Integers compare bit for bit unless a step bound is stated. Floats compare within a stated count of units in the last place, using ulpDistance, or within a stated fraction of the buffer’s scale, and each comparison states its bound and the reason for it.

The interpreter shares the front end with the code under test, so a front-end defect can agree with itself. cajeta-llm’s host oracles stay in that repository’s CI as the independent check on that path.

Testing the instrument

CAJETA_XPU_FAULT=dot-activation-sign restores the single-flag dot lowering that cajeta 3ba58bda fixed. XpuReferenceInterpreterTests sets it, checks that the cpu backend is caught disagreeing, and checks that the two agree once it is unset. The variable exists only for that test.

See also

• specs/archive/xpu-kernel-independence-spec.md §2.2 and §7.2.
• test/xpu/XpuReferenceInterpreterTests.cpp.

Source: docs/specification/xpu/ReferenceInterpreter.md · 14 min read · 3186 words