mirror of https://gitee.com/namelin2022/ollama
Browse Source
KV cache quantization has a dependency on the flash attention kernel. We currently cannot use flash attention with gpt-oss as it requires additional operations. The model definition does not call flash attention, so it works regardless of the setting but the cache will pick up the quantization type. This updates the flash attention setting earlier in the loading flow so that all downstream settings are also set correctly. Fixes: #11671mxyng/convert v0.11.2
committed by
Jesse Gross
1 changed files with 4 additions and 0 deletions
Loading…
Reference in new issue