Branch: drifkin/array-head-count-simple

History

Jesse Gross dbb149e6f7 ollamarunner: Preallocate worst case graph at startup Currently, the KV cache and graph are lazily allocated as needed. The cache is fully allocated on first use of the corresponding layer whereas the graph grows with the size of the context. This can be an issue if another application allocates more VRAM after we do our calculations - Ollama will crash in the middle of inference. If we instead allocate the maximum needed memory at startup of the runner, we will either succeed or fail at that point rather than at some surprising time in the future. Currently, this only generates a worst case batch for text, which means that vision models may get a partial allocation and continue to lazily allocate the rest.		1 year ago
..
common	Runner for Ollama engine	2 years ago
llamarunner	llm: set done reason at server level (#9830)	1 year ago
ollamarunner	ollamarunner: Preallocate worst case graph at startup	1 year ago
README.md	Runner for Ollama engine	2 years ago
runner.go	Runner for Ollama engine	2 years ago

`runner`

Note: this is a work in progress

A minimial runner for loading a model and running inference via a http web server.

./runner -model <model binary>

curl -X POST -H "Content-Type: application/json" -d '{"prompt": "hi"}' http://localhost:8080/completion

curl -X POST -H "Content-Type: application/json" -d '{"prompt": "turn me into an embedding"}' http://localhost:8080/embedding