Browse Source

lower default num parallel to 2

this is in part to "pay" for #10452, which doubled the default context length. The combination isn't fully neutral though, because even though the old 4x2k limit and the new 2x4k limit are memory equivalent, the 1x fallback is larger with 4k
jmorganca/cuda-compression-none
Devon Rifkin 1 year ago
committed by GitHub
parent
commit
fe5b9bb21b
No known key found for this signature in database GPG Key ID: B5690EEEBB952194
  1. 2
      server/sched.go

2
server/sched.go

@ -58,7 +58,7 @@ var defaultModelsPerGPU = 3
// Default automatic value for parallel setting
// Model will still need to fit in VRAM. If this setting won't fit
// we'll back off down to 1 to try to get it to fit
var defaultParallel = 4
var defaultParallel = 2
var ErrMaxQueue = errors.New("server busy, please try again. maximum pending requests exceeded")

Loading…
Cancel
Save