Local AI Engine (llama.cpp)

What is llama.cpp?

llama.cpp is the engine that runs AI models locally on your computer. Think of it as the software that takes an AI model file and makes it actually work on your hardware - whether that's your CPU, graphics card, or Apple's M-series chips.

Originally created by Georgi Gerganov, llama.cpp is designed to run large language models efficiently on consumer hardware without requiring specialized AI accelerators or cloud connections.

Why This Matters

Privacy: Your conversations never leave your computer Cost: No monthly subscription fees or API costs Speed: No internet required once models are downloaded Control: Choose exactly which models to run and how they behave

Accessing Engine Settings

Find llama.cpp settings at Settings () > Llama.cpp under Model Providers:

Model Management

The Models section at the top of the Llama.cpp settings page lets you manage all your local GGUF models.

Download from Hub

Browse and download models directly from the Hub tab in the left sidebar. Downloaded models will appear here automatically.

Import Local Files

Click Import to link a GGUF model file already on your computer. This is useful for models downloaded via your browser from Hugging Face, or models shared with other apps — Jan links to the file in place without copying it.

Delete a Model

Click the trash icon next to any model to remove it. This deletes the model file from Jan's data folder (linked files leave the original intact).

Engine Management

Hardware Backends

Jan offers different backend versions optimized for your specific hardware. Think of these as different "drivers" - each one is tuned for particular processors or graphics cards.

Performance Settings

These control how efficiently models run:

Memory Settings

These control how models use your computer's memory:

KV Cache Types Explained

• f16: Full 16-bit precision, uses more memory but highest quality
• q8_0: 8-bit quantized, balanced memory usage and quality
• q4_0: 4-bit quantized, uses least memory, slight quality loss

Advanced Settings

These settings are for fine-tuning model behavior and advanced use cases:

Text Generation Control

RoPE (Rotary Position Embedding) Settings

Mirostat Sampling

Output Constraints

Troubleshooting Common Issues

Models won't load:

• Try a different backend (switch from CUDA to CPU or vice versa)
• Check if you have enough RAM/VRAM
• Update to latest engine version

Very slow performance:

• Make sure you're using GPU acceleration (CUDA/Metal/Vulkan backend)
• Increase GPU Layers in model settings
• Close other memory-intensive programs

Out of memory errors:

• Reduce Context Size in model settings
• Switch KV Cache Type to q8_0 or q4_0
• Try a smaller model variant

Random crashes:

• Switch to a more stable backend (try avx instead of avx2)
• Disable overclocking if you have it enabled
• Update graphics drivers

Quick Setup Guide

For most users:

1. Use the default backend that Jan installs
2. Enable Auto Update Engine for automatic compatibility
3. Leave all performance settings at defaults
4. Only adjust if you experience problems

If you have an NVIDIA graphics card:

1. Select the appropriate CUDA backend from the dropdown (e.g., avx2-cuda-12-0)
2. Make sure GPU Layers is set high in model settings
3. Keep Flash Attention enabled
4. Set Main GPU Index if you have multiple GPUs

If models are too slow:

1. Check you're using GPU acceleration (CUDA/Metal/Vulkan backend)
2. Enable Continuous Batching
3. Increase Batch Size and uBatch Size
4. Close other applications using memory

If running out of memory:

1. Enable Auto-Unload Old Models
2. Change KV Cache K/V Type to q8_0 or q4_0
3. Reduce Context Size in model settings
4. Enable MLock if you have sufficient RAM
5. Try a smaller model

For advanced users:

1. Experiment with Mirostat sampling for more consistent outputs
2. Use Grammar/JSON Schema files for structured generation
3. Adjust RoPE settings for models with extended context support
4. Fine-tune thread counts based on your CPU