CUDA: add fast walsh-hadamard transform by am17an · Pull Request #23615 · ggml-org/llama.cpp
r/LocalLLaMA
•
Generative AI
AI Hardware
Open Source AI
Implemented(by u/am17an ) FWHT for CUDA, speed-up for cases when we quantize the k-cache. 1-2% boost on pp & 7-9% boost on tg. Performance on a 5090 with -ctk q8_0 -ct q8_0 Model Test t/s master t/s cuda-fwt Speedup gemma4 26B. A4B Q4_K_M pp2048 13587.89 13809.20 1.02 gemma4 26B. A4B Q4_K_M pp2048 12425.01 12553.32 1.01 gemma4 26B. A4B Q4_K_M pp2048 12158.21 12291.42 1.01 gemma4 26B. A4B Q4_K_M pp2048 11710.89 11913.97 1.02 gemma4 26B. A4B Q4_K_M pp2048 10982.21 11214.12 1.02 gemma4 26B. A4B Q4_K_M pp2048 9702.60 9776.75 1.01 gemma4 26B. A4B Q4_K_M tg128 223.81 243.90 1.09 gemma4 26B.