docs / benchmarks / xpu

Benchmarks index
  1. Overview
  1. Exemptions
  2. Methodology

aot

  1. Overview
  2. Components

codec

  1. Overview
  2. Base32
  3. Base64
  4. Deflate (dev.cajeta.codec)

codec / csv

  1. CSV

codec / json

  1. JSON

collection

  1. Overview
  2. ArrayList
  3. BPlusTree
  4. Cache
  5. Collectors
  6. HashMap
  7. HashSet
  8. Heap
  9. ImmutableList
  10. ImmutableMap
  11. ImmutableSet
  12. LinkedList
  13. RedBlackTree
  14. Sort

collection / ltm

  1. LtmBPlusTree

concurrent

  1. Overview
  2. AtomicInt32
  3. AtomicInt64
  4. Channel
  5. Fiber
  6. FiberLocal
  7. Lock
  8. Mutex
  9. RwLock
  10. Semaphore
  11. Tasks

gfx

  1. Overview
  2. Texture2D

hash

  1. Overview
  2. DefaultHasher
  3. Hash
  4. SipHash
  5. XXHash3

http

  1. Overview
  2. Client
  3. Server
  4. Server-sent events
  5. Start-up and footprint
  6. TLS
  7. WebSocket

io

  1. Overview
  2. Buffer

io / file

  1. Directory walk
  2. File
  3. FileInfo
  4. FileReader
  5. FileWriter
  6. Path
  7. Watcher

io / net

  1. BufferPool
  2. ByteBuffer
  3. Server
  4. TcpListener
  5. TcpStream
  6. UdpSocket

io / net / uri

  1. Uri
  2. UriBuilder

lang

  1. Overview
  2. Allocation and ownership
  3. Calls and dispatch
  4. Exceptions
  5. Guid
  6. Loops
  7. Math and SIMD
  8. Optional
  9. Pair
  10. Slice
  11. String
  12. StringBuilder
  13. Whole programs: the Benchmarks Game

lang / stream

  1. ArrayStream
  2. Stream

llm

  1. Overview
  2. Decode
  3. Kernel parity
  4. Load and first token
  5. Long context
  6. Memory
  7. Prefill

math

  1. Overview
  2. Ray
  3. Rotation
  4. Tensor
  5. Transform

math / fft

  1. Fft

math / linalg

  1. LinAlg

math / npio

  1. Npy

math / poly

  1. Poly

math / random

  1. Generator

math / stats

  1. Stats

ml

  1. Overview
  2. Classical models
  3. Neural models
  4. XGBoost

nucleo

  1. Overview
  2. Column
  3. Fuse
  4. Table
  5. Tape
  6. Transforms

primavera

  1. Overview
  2. Build time
  3. Dependency injection at scale
  4. REST endpoint
  5. Start-up and footprint
  6. Templated page

process

  1. Overview
  2. Command
  3. Process

reflect

  1. Overview
  2. Class

search

  1. Overview

search / distance

  1. Distance

search / fuzzy

  1. Matcher

search / ngram

  1. Index

time

  1. Overview
  2. Clock
  3. DateTimeFormatter
  4. Duration
  5. Instant
  6. LocalDate
  7. LocalDateTime
  8. LocalTime
  9. ZonedDateTime

toolchain

  1. Overview
  2. Binary size
  3. Build tool
  4. Compile time
  5. Debugger
  6. Notebook kernel start-up
  7. Start-up

toolchain / kernel-startup

  1. Overview
  2. Where cold-start time goes

wire

  1. Overview
  2. CompressStream
  3. Encoder
  4. SchemaEncoder

xpu

  1. Overview
  2. Device
  3. KernelBuffer
  4. KernelStream
  5. KernelTimer

xpu / mesh

  1. MeshSimplifier
Machine
no runs yet

Device

cajeta.xpu.Device — the active device and the kernels that run on it. BabelStream gives sustained bandwidth, mixbench the roofline, the GEMM rows the matrix cores through CooperativeMatrix, and the last row is the one no other suite has: one @Kernel source run on CUDA, HIP, Vulkan and the CPU with a consistency check. Device rows are measured in interleaved rounds because GPU clocks are bimodal.

Benchmarks

idworkloadagainstcheck
xpu/babelstream-copyBabelStream copy at 2^25 float64BabelStream CUDA and HIP reference buildschecksum
xpu/babelstream-mulmulsamechecksum
xpu/babelstream-addaddsamechecksum
xpu/babelstream-triadtriad, the canonical bandwidth numbersamechecksum
xpu/babelstream-dotdotsamewithin epsilon
xpu/device-saxpyy = a*x + y on device at 2^26 float32CUDA and HIP C++, PyTorchagainst host
xpu/wave-reducesum of 2^26 float32 with wave reductionsCUB and rocPRIM DeviceReduce, PyTorch sumwithin epsilon
xpu/scaninclusive scan of 2^26 int32CUB and rocPRIM DeviceScan, PyTorch cumsumlast element and spot checks
xpu/gemm-f644096 × 4096 GEMM at float64 (the first suite’s device matmul, kept under its true type)cuBLAS and rocBLAS, PyTorch matmulagainst host within epsilon
xpu/gemm-f324096 × 4096 GEMM through CooperativeMatrix at float32cuBLAS and rocBLAS, PyTorch matmulagainst host within epsilon
xpu/gemm-f16the same at float16 with float32 accumulatesamewithin epsilon
xpu/gemm-bf16the same at bfloat16samewithin epsilon
xpu/mandelbrot-devicemandelbrot at 16000 × 16000 on deviceCUDA and HIP C++within 1e-5 relative of the host (FMA contraction moves a handful of pixels)
xpu/mixbench-sweepthe mixbench operational-intensity sweep, points 0 to 256 as variants, on a fixed-point input so the check is exactmixbench CUDA and HIP reference buildsexact
xpu/launch-overhead1e4 trivial dispatches; the row is the time for all 1e4 (divide by 1e4 for a launch)CUDA and HIP empty kernels, Vulkan computecount
xpu/cross-backend-showcasethe saxpy, reduce and GEMM kernels from one source on every backend present; variants cuda, hip, vulkan, cpuCajeta’s own backendsbit-exact where exact, epsilon where not

Rows record the backend and driver; CUDA and Vulkan rows come from Phoenix, HIP and Vulkan rows from Proton. Compiler manifests on v0.46.3: bf16 is native matrix-core on NVPTX and the software tier on Vulkan, f32 the software tier on NVPTX and native on Vulkan, f16 native on both; no kernel spills. The CUDA competitor is one devserver program (cuBLAS, CUB, texture objects), which also builds with hipcc on a ROCm machine.

Results

One chart per benchmark, for the machine selected at the top of the page. Languages are the bars, best first; Cajeta’s bar is in color and the competitors’ are gray. A skipped competitor keeps its row with the reason in place of a bar. A benchmark with no row on the selected machine says so and shows nothing else.

xpu/babelstream-copy

Not yet measured on Proton.
xpu/babelstream-mul

Not yet measured on Proton.
xpu/babelstream-add

Not yet measured on Proton.
xpu/babelstream-triad

Not yet measured on Proton.
xpu/babelstream-dot

Not yet measured on Proton.
xpu/device-saxpy

Not yet measured on Proton.
xpu/wave-reduce

Not yet measured on Proton.
xpu/scan

Not yet measured on Proton.
xpu/gemm-f64

Not yet measured on Proton.
xpu/gemm-f32

Not yet measured on Proton.
xpu/gemm-f16

Not yet measured on Proton.
xpu/gemm-bf16

Not yet measured on Proton.
xpu/mandelbrot-device

Not yet measured on Proton.
xpu/mixbench-sweep

Not yet measured on Proton.
xpu/launch-overhead

Not yet measured on Proton.
xpu/cross-backend-showcase

Not yet measured on Proton.

Reproduce

cd bench
./bench.sh --bench babelstream-copy               # Cajeta
./bench.sh --bench babelstream-copy --competitors # every pinned competitor

Every row records the compiler and competitor versions, flags, machine, OS and driver; METHODOLOGY states the rules a number must satisfy before it is published.

See also

  • Reference: Device
  • Source: runtime/src/cajeta/xpu/Device.cajeta
  • Benchmark source: bench/src/main/cajeta/bench/xpu/

Source: bench/xpu/Device.md · 3 min read · 586 words