Sub-issue of #1198. Explicitly lower priority / deferred — only pick up if the sidecar
cache's cold-start latency on a cache miss (see the sidecar cache sub-issue) proves to matter in
practice. Not scheduled otherwise.
Idea
A kernel variant that decodes I2sGgufLayout.GROUP_128 (BitNet.cpp's x86/AVX-quantized layout —
128-element blocks, high-bit-pair-first, per I2sRepack.kt:69-84's bit arithmetic) directly from
the file's native byte order, the same way the vendored hs_ml_ternary_neon.c decodes
SEQUENTIAL today. This would let GROUP_128 tensors — the common case for published GGUFs —
qualify for true zero-copy mmap the same way the SEQUENTIAL fast-path sub-issue gets it for
NeoGPU-converted files, without ever materializing a repacked payload, cached or not.
GROUP_64 (BitNet.cpp's ARM/NEON-quantized flavor, rarer in practice) is explicitly not
scheduled here — it stays on the repack/sidecar-cache fallback path indefinitely unless it
becomes a real problem on its own.
Constraint
Must not modify the vendored file itself
(skainet-backends/skainet-backend-native-cpu/native/src/vendor/neogpu/hs_ml_ternary_neon.c) —
its "Local modifications: none" invariant
(skainet-backends/skainet-backend-native-cpu/native/src/vendor/neogpu/README.md:70-73) exists
so provenance stays byte-verifiable against upstream. Any new GROUP_128 decode logic is a
sibling kernel, not an edit to the vendored one.
Why this is deferred rather than done alongside the other three
The crash risk is fixed by the off-heap storage sub-issue regardless of on-disk layout, and the
repeated-repack-cost problem is fixed by the sidecar cache sub-issue, also regardless of layout.
This sub-issue only buys back the first-load latency on a cache miss — a smaller win, for a
kernel variant that then has to be maintained per-backend (NEON, any future AVX-512/x86 path)
indefinitely. Worth doing only if that specific latency is shown to matter.
Sub-issue of #1198. Explicitly lower priority / deferred — only pick up if the sidecar
cache's cold-start latency on a cache miss (see the sidecar cache sub-issue) proves to matter in
practice. Not scheduled otherwise.
Idea
A kernel variant that decodes
I2sGgufLayout.GROUP_128(BitNet.cpp's x86/AVX-quantized layout —128-element blocks, high-bit-pair-first, per
I2sRepack.kt:69-84's bit arithmetic) directly fromthe file's native byte order, the same way the vendored
hs_ml_ternary_neon.cdecodesSEQUENTIALtoday. This would letGROUP_128tensors — the common case for published GGUFs —qualify for true zero-copy mmap the same way the SEQUENTIAL fast-path sub-issue gets it for
NeoGPU-converted files, without ever materializing a repacked payload, cached or not.
GROUP_64(BitNet.cpp's ARM/NEON-quantized flavor, rarer in practice) is explicitly notscheduled here — it stays on the repack/sidecar-cache fallback path indefinitely unless it
becomes a real problem on its own.
Constraint
Must not modify the vendored file itself
(
skainet-backends/skainet-backend-native-cpu/native/src/vendor/neogpu/hs_ml_ternary_neon.c) —its "Local modifications: none" invariant
(
skainet-backends/skainet-backend-native-cpu/native/src/vendor/neogpu/README.md:70-73) existsso provenance stays byte-verifiable against upstream. Any new
GROUP_128decode logic is asibling kernel, not an edit to the vendored one.
Why this is deferred rather than done alongside the other three
The crash risk is fixed by the off-heap storage sub-issue regardless of on-disk layout, and the
repeated-repack-cost problem is fixed by the sidecar cache sub-issue, also regardless of layout.
This sub-issue only buys back the first-load latency on a cache miss — a smaller win, for a
kernel variant that then has to be maintained per-backend (NEON, any future AVX-512/x86 path)
indefinitely. Worth doing only if that specific latency is shown to matter.