Browse Source

kvcache: Log batch size if we can't find a slot

In some cases, we can't find a cache slot when using sliding window
attention. It would be helpful in this (and other cases) to know what
the batch size is.

Bug #10127
jmorganca/cuda-compression-none
Jesse Gross 1 year ago
committed by Jesse Gross
parent
commit
074bac8447
  1. 2
      kvcache/causal.go

2
kvcache/causal.go

@ -239,7 +239,7 @@ func (c *Causal) findStartLoc() (int, error) {
}
}
return 0, fmt.Errorf("%w (length: %v)", ErrKvCacheFull, len(c.cells))
return 0, fmt.Errorf("%w (cache: %v batch: %v)", ErrKvCacheFull, len(c.cells), c.curBatchSize)
}
func (c *Causal) updateSlidingWindow() {

Loading…
Cancel
Save