Building and serving local LLM models with llama.cpp

For running local LLMs, ollama is an easy way to get up and running, but if you need or want to start tuning model parameters, llama.cpp offers more flexibility (and is generally faster than ollama)

I installed following steps here.

To run on an nvidia GPU I Installed the CUDA toolkit following steps here.

After installing Nvida CUDA toolkit, and attempting to build with it enabled, I got this error:

$ cmake -B build -DGGML_CUDA=ON

CMAKE_BUILD_TYPE=Release
-- Warning: ccache not found - consider installing it for faster compilation or disable this warning with GGML_CCACHE=OFF
-- CMAKE_SYSTEM_PROCESSOR: x86_64
-- GGML_SYSTEM_ARCH: x86
-- Including CPU backend
-- x86 detected
-- Adding CPU backend variant ggml-cpu: -march=native 
-- CUDA Toolkit found
-- The CUDA compiler identification is unknown
CMake Error at ggml/src/ggml-cuda/CMakeLists.txt:59 (enable_language):
  No CMAKE_CUDA_COMPILER could be found.

  Tell CMake where to find the compiler by setting either the environment
  variable "CUDACXX" or the CMake cache entry CMAKE_CUDA_COMPILER to the full
  path to the compiler, or to the compiler name if it is in the PATH.

Following steps online, it was suggested to add this line to /etc/environment, but this still doesn’t resolve the error:

CUDACXX=/usr/local/cuda~13.3/bin/nvcc

I noticed in this additional steps section, it mentioned to update your path, so added these 2 lines to .bashrc and then this resolved the above config issue:

export PATH=${PATH}:/usr/local/cuda-13.3/bin
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/cuda-13.3/lib64

After building with these options llama-cli is able to run. If I point it at the Gemma model I downloaded with ollama though I get errors, and posts online seem mixed on whether you can do this or not, but most seem to imply you need to download a GGUF model specifically for llama.cpp:

./llama-cli --model /usr/share/ollama/.ollama/models/blobs/sha256-4e30e2665218745ef463f722c0bf86be0cab6ee676320f1cfadf91e989107448
Loading model... \0.00.696.653 E llama_model_load: error loading model: done_getting_tensors: wrong number of tensors; expected 2012, got 601
0.00.696.666 E llama_model_load_from_file_impl: failed to load model

After testing with other more recently downloaded models with ollama (e.g. Qwen 3.8 27b), this approach does work, and appears the error is model specific.

I also discovered llama.cpp build detects your GPU model and builds a family specific build. I just replaced a 2060 with a 5070ti and had to rebuild after the change, as the last build binaries were giving errors that the GPU was not detected.

To start a cli chat session using llama and a specific model, I used:

./build/bin/llama-cli --model [path to model file]

I also found out by trial and error that pointing to the sha named model files downloaded by ‘ollama pull’ can be run by llama.ccp

To run a llama server that you can use with harnesses like opencode, pi and oh my pi, I used:

./build/bin/llama-server --model [path to model] --chat-template-kwargs '{"reasoning_effort":"medium"}' --host 0.0.0.0 --port 8090 --alias qwen3.8-27b --ctx-size 49152 --parallel 1 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0

At this point I also started working with Claude to benchmark the token per second responses with increasing the context size to find a sweetspot where I could run 100% of the model on the GPU with the largest context size without spilling onto the CPU and system ram. Claude seemed particularly good at this, proposing changes, testing, and then retesting tps and checking memory usage.

At this point I started moving to the llama router so I could switch between multiple models from my harness:

./build/bin/llama-server --host 0.0.0.0 --port 8090 --models-preset ./router-models.ini --models-max 1

Here’s an example of my router-models.ini:

; Global defaults applied to every model unless overridden below.
[*]
n-gpu-layers = 999
flash-attn = on
cache-type-k = q8_0
cache-type-v = q8_0
ctx-size = 16384
parallel = 1

[qwen3.8-27b-general]
model = [path to model]
ctx-size = 49152
chat-template-kwargs = {"reasoning_effort":"medium"}

[qwen3.8-27b-agent]
model = [path to model]
ctx-size = 49152
chat-template-kwargs = {"reasoning_effort":"low"}
reasoning-budget = 512
temp = 0.6
repeat-last-n = 256
repeat-penalty = 1.15

[qwen3.6-35b-a3b-agent]
model = [path to model]
ctx-size = 131072
n-cpu-moe = 18
no-mmap = true
reasoning-budget = 512
temp = 0.6
top-p = 0.95
top-k = 20
min-p = 0.0