affine_qmm_t computes only batch element 0 of 3D inputs — rows of later batch elements are garbage (int4 and int8)
affine_qmm_t called with a 3D input computes the first batch element only and
returns garbage (deterministic, bit-identical across calls) for every later batch
element. 2D inputs of any size are correct; [1, S, K] is correct; [B, 1, K] with
B > 1 corrupts rows 1+; [2, 4, K] corrupts exactly rows 4–7 (the second batch
element) - i.e. the kernel treats the input as [shape[-2], K] and ignores the leading
batch dimension. Identical for bits=4/8 × bf16/fp16.
Hit via transformers 5.13.1 MetalConfig (kernels 0.15.2, torch 2.13.0 on thetorch210 build, Apple M4 Max / macOS 26): HF decode feeds [batch, 1, hidden] to
every layer, so batched generation silently corrupts all sequences after the first.
Kernel-level probe + model-level repro: https://github.com/huggingface/transformers/issues/47331.
Flattening 3D inputs to 2D before the call is a full workaround (verified at model
level), so the integration side can fix this without a kernel change - but the kernel
silently returning garbage for documented-shape inputs seems worth fixing or guarding
here too.
Also noting: this repo ships builds for torch 2.8/2.9/2.10 only; on torch 2.13 thekernels loader picks the torch210 build. If that pairing is unsupported, a load-time
error would be preferable to silent corruption.