From the onset, my goal for this project was to install the ‘best’ large language model onto my 16GB GeForce RTX 4080 graphics card. A generic term like ‘best’ gets thrown around far too often, and so I will state for the record it means something different to everyone based on their personal needs. This guide builds upon the recently published article: Understanding Local AI and Large Language Models. You will need to understand the nomenclature to make use of this (or any) tutorial.
While this article documents the steps I took to get started, there are certain aspects that will remain specific to my installation. For example, I have a second NVME SSD (A:) designated for Debian and its contents. Much of the details assembled together in this article were generated by Claude (Opus 5.5 / Extra). While I tried to keep the content as interesting as possible, the majority of this tutorial is copy/paste. Alternatively, you could always feed these instructions into your favorite AI agent, and let it walk you though the steps.
Project Overview
This guide sets up a local coding agent on a Windows 11 PC with a 16GB NVIDIA graphics card, a very common VRAM size across NVIDIA’s RTX 40 and 50 series. Strictly speaking, llama.cpp’s llama-server runs the Swift 1.5 Qwen3.8 27B model inside WSL2 Debian, and the Pi coding agent connects to it. This particular setup fits a 102,400-token context entirely in 16GB of VRAM. The reference system (GeForce RTX 4080 FE, Core i9-11900K, 32GB DDR4-3600 RAM) generates about 70 tokens per second on short prompts and about 40 near a full context.
Every 16GB card can hold this same model and context. Generation speed depends mainly on memory bandwidth, so other cards scale roughly with it:
| 16GB GPU | Architecture (compute capability) | Memory bandwidth | Short-prompt speed |
|---|---|---|---|
| RTX 4060 Ti 16GB | Ada (8.9) | 288GB/s | ~28 t/s (estimate) |
| RTX A4000 | Ampere (8.6) | 448GB/s | ~40 t/s (estimate) |
| RTX 5060 Ti 16GB | Blackwell (12.0) | 448GB/s | ~43 t/s (estimate) |
| RTX 4070 Ti Super | Ada (8.9) | 672GB/s | ~65 t/s (estimate) |
| RTX 4080 (reference) | Ada (8.9) | 717GB/s | ~69 t/s (measured) |
| RTX 4080 Super | Ada (8.9) | 736GB/s | ~71 t/s (estimate) |
| RTX 5070 Ti | Blackwell (12.0) | 896GB/s | ~86 t/s (estimate) |
| RTX 5080 | Blackwell (12.0) | 960GB/s | ~92 t/s (estimate) |
Estimates scale the reference result by bandwidth; compute limits, prompt processing speed, and speculative-decoding hit rates vary by card. 16GB laptop GPUs work the same way but run lower clocks and bandwidth than their desktop namesakes.
Local Swift 1.5 27B on a 16GB NVIDIA GPU (WSL2 + llama-server + Pi)
Requirements:
- Windows 11 with WSL2 (Windows Sub-system for Linux, included) and a Debian distribution (free via Microsoft Store)
- Any NVIDIA GPU with 16GB of VRAM (see the table above) and a current Windows driver
- 32GB of system RAM recommended; 16GB works with a smaller prompt cache (Step 1)
- About 25GB free on the drive that holds the WSL disk
- Integrated graphics (iGPU) on the CPU (Intel non-F models, AMD Ryzen 7000 or newer, or Ryzen G models), enabled in the BIOS, so other apps can render off the 16GB card (Step 1). Without it, expect a smaller context
Configuration summary:
| Setting | Value | Why |
|---|---|---|
| Model file | Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf (10.44GB) |
Includes the MTP head that speculative decoding uses |
| Context | 102,400 tokens | The largest size that fits in 16GB with q8_0 caches |
| KV cache | q8_0 for K and V, q8_0 for the MTP draft cache | Half the VRAM of f16 with negligible quality loss |
| Speculative decoding | MTP draft plus two n-gram matchers | About 55% faster generation than n-gram alone |
| Prompt cache | 12GiB in system RAM | Reuses earlier prompt prefixes; sized to fit WSL’s memory limit |
| Chat template | froggeric fixed Qwen template | Fixes tool-call rendering and prompt-cache invalidation |
| Endpoint | 127.0.0.1:8080, alias swift-27b |
Local only, with a short model ID for Pi |
Model facts: Swift 1.5 Qwen3.8-27B · GSQ-RCO is a Qwen3.8-27B fine-tune (GGUF arch qwen35) that the description card says uses 58.5% fewer thinking tokens than the base model. The IQ3_XXS tier has a development KLD of 0.0978; IQ3_S (12.12GB with MTP) is the next step -up in quality; on a 16GB card it leaves room for roughly 50,000 tokens of context. The card’s example uses --repeat-penalty 1.0; this guide uses 1.1 (see Step 5).
VRAM budget at 102,400 tokens: model weights 10.44GB (9.7GiB), q8_0 KV cache about 3.4GiB, and compute buffers, MTP draft context, and recurrent state about 1.5GiB. Total dedicated use is about 15.6GB, roughly 60MB below the point where Windows starts moving GPU memory into system RAM (16.0GB minus the hardware reservation, 336MB on the reference card). This fits only when other apps render on the integrated GPU (Step 1). If other GPU apps must share the GPU, use -c 98304 for about 150MB more margin. Without MTP, the 10.09GB standard file fits 104,000 tokens but generates at about 44 tokens per second on the reference card. Step 7 Part C finds the exact ceiling for your system.
Step 1: Windows prep
The Windows NVIDIA driver is the only GPU driver you install; WSL maps it in as /usr/lib/wsl/lib/libcuda.so (NVIDIA CUDA on WSL guide).
- Visit geforce.com, and update the NVIDIA driver to the current Windows 11 release.
- In PowerShell:
wsl --update - Create or edit
%UserProfile%\.wslconfigso WSL can hold the prompt cache, the page-cached model file, and Claude Code at the same time:
[wsl2]
memory=26GB
swap=8GB
# Optional: keep the swap file off C:
# swapFile=A:\\wsl\\swap.vhdx- Apply it:
wsl --shutdown, then reopen Debian. - Confirm inside Debian:
free -hshows roughly your memory setting as total, andnvidia-smilists your GPU with about 16,376MiB total. Note the “CUDA Version” in the top right ofnvidia-smi; that is the highest toolkit your driver supports.
Leave 6GB for Windows itself. If Windows feels starved while the model runs, drop memory to 24GB and -cram to 8192.
Size WSL’s memory and the prompt cache (-cram in Step 5) to your system RAM:
| System RAM | .wslconfig memory |
-cram (MiB) |
|---|---|---|
| 64GB | 48GB | 24576 |
| 32GB (reference) | 26GB | 12288 |
| 16GB | 11GB | 2048 |
Move app rendering off the 16GB card. The selected model needs nearly all 16GB of VRAM. In Windows Settings, go to System > Display > Graphics, add Chrome and any other GPU-accelerated apps you keep open, and set each one to Power saving so it renders on the CPU’s integrated graphics. If that option is missing, enable the integrated GPU in the BIOS first. Hardware acceleration stays on, and the monitor stays connected to the NVIDIA card. If your CPU has no integrated graphics, the desktop and these apps keep 0.4 to 0.9GB of VRAM; start the Step 7 Part C search at 81920 and expect a ceiling around 90,000 tokens. On the reference system, setting Chrome to Power saving moved its rendering to the i9-11900K’s UHD 750 integrated graphics and freed about 0.4 GB of the 0.9 GB that Windows and apps had been using on the RTX 4080, which raised the clean context from 81,920 to 102,400 tokens.
Note that some apps configured for Power savings, such as Chrome, will have a slightly noticeable latency when typing as the signal passes from the processor’s iGPU to video card’s GPU output.
Step 2: CUDA toolkit inside Debian WSL
Install only a versioned cuda-toolkit-X-Y package. Never install cuda, cuda-X-Y, or cuda-drivers under WSL; those pull a Linux driver that overwrites the WSL mapping.
# Build prerequisites
sudo apt update
sudo apt install -y build-essential cmake git curl wget ccache libssl-dev pkg-config pipx
# NVIDIA network repo for your Debian release (debian12 or debian13)
DEB=debian$(. /etc/os-release; echo $VERSION_ID)
echo "$DEB"
wget https://developer.download.nvidia.com/compute/cuda/repos/${DEB}/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
# List available toolkits, then install the newest one at or below the
# "CUDA Version" that nvidia-smi reported (13.4 shown as an example; 13.4.2 is current as of October 2026)
apt-cache search '^cuda-toolkit-[0-9]+-[0-9]+$' | sort -V
sudo apt install -y cuda-toolkit-13-4
# PATH for nvcc
echo 'export PATH=/usr/local/cuda/bin:$PATH' >> ~/.bashrc
source ~/.bashrc
nvcc --versionIf the debian13 repo URL returns 404, set DEB=debian12. Do not add /usr/local/cuda/lib64/stubs to LD_LIBRARY_PATH; its stub libcuda would shadow the real WSL driver.
Step 3: Build llama-server with CUDA
Build from master so the Qwen3.8 architecture, draft-mtp, ngram-map-k4v, ngram-mod, and --reasoning-preserve are all present. Compiling only for your card’s compute capability (8.9 for RTX 40, 12.0 for RTX 50, 8.6 for RTX A4000) keeps the build short; the first command reads it from nvidia-smi (build docs). RTX 50 cards need CUDA 12.8 or newer, which the 13.x toolkit from Step 2 covers.
cd ~
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
CC=$(nvidia-smi --query-gpu=compute_cap --format=csv,noheader | head -1 | tr -d .); echo "Compute capability: $CC"
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=$CC -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j 12 --target llama-server llama-cli llama-bench
# Put the binaries on PATH
mkdir -p ~/.local/bin
ln -sf ~/llama.cpp/build/bin/llama-server ~/.local/bin/llama-server
ln -sf ~/llama.cpp/build/bin/llama-bench ~/.local/bin/llama-bench
# Verify
llama-server --version
llama-server --list-devices # should list CUDA0 with your GPU's name
llama-server --help | grep -E 'draft-mtp|reasoning-preserve|cache-ram'-j 12 leaves headroom on the 11900K’s 16 threads; nvcc is memory hungry, so lower it if the build is killed. The q8_0/q8_0 KV pair is in the default flash-attention kernel set, so GGML_CUDA_FA_ALL_QUANTS is not needed.
To update later: for a llama.cpp update alone, run cd ~/llama.cpp && git pull && cmake --build build --config Release -j 12 --target llama-server llama-cli llama-bench. After installing a new CUDA toolkit, delete the build folder first (rm -rf build) and rerun both cmake commands above, because CMake caches the old compiler path and a plain rebuild keeps using it. A real rebuild shows compile lines; output that says only Built target means nothing changed. Re-run Step 7 Part C after either update.
Step 4: Download the model and chat template
Keep the model inside the WSL ext4 disk (on your A: drive), not under /mnt/a, so loads avoid the slow 9P file bridge.
# Hugging Face CLI (provides the hf command)
pipx install huggingface_hub
pipx ensurepath && source ~/.bashrc
mkdir -p ~/models/swift15 && cd ~/models/swift15
# The MTP variant of IQ3_XXS plus the checksum list
hf download ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF \
Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf SHA256SUMS \
--local-dir .
grep 'IQ3_XXS-mtp.gguf' SHA256SUMS | sha256sum -c -
# froggeric fixed Qwen template, saved under a descriptive filename
curl -L -o froggeric--Qwen-Fixed-Chat-Templates--chat_template.jinja \
https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates/resolve/main/chat_template.jinja
grep -m1 template_version froggeric--Qwen-Fixed-Chat-Templates--chat_template.jinjaThe last line should print a qwen3.8-froggeric-v22.x version string. The froggeric template fixes tool-call rendering and prompt-cache invalidation in the official Qwen templates and defaults preserve_thinking to true, which pairs with --reasoning-preserve.
The current release is v22.5 (September 2026), still the single chat_template.jinja at the repository root. Its author recommends --reasoning-format deepseek when Pi or another coding agent connects, so thinking arrives in the separate reasoning_content field instead of the reply text. llama-server’s default (auto) normally does the same; if Pi ever shows raw <think> text or stops mid tool call, add --reasoning-format deepseek to the script.
The model card’s usage section still says to authenticate while the repository is private, but the repository is public. If hf download returns 401, run hf auth login with a read token.
Step 5: Launch script
swift-serve is a small Bash script that starts llama-server with every setting from the configuration summary in the Overview. Save the block below into a file named swift-serve, then follow “How to run it” under the code.
#!/usr/bin/env bash
set -euo pipefail
MD="$HOME/models/swift15"
exec llama-server \
-m "$MD/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf" \
--alias swift-27b \
--no-ui \
--host 127.0.0.1 --port 8080 \
-ctk q8_0 -ctv q8_0 \
--flash-attn on \
--parallel 1 \
-cram 12288 \
-ngl all \
--no-mmproj --no-mmproj-offload \
--chat-template-file "$MD/froggeric--Qwen-Fixed-Chat-Templates--chat_template.jinja" \
-c 102400 \
--spec-type draft-mtp,ngram-map-k4v,ngram-mod \
--spec-draft-type-k q8_0 --spec-draft-type-v q8_0 \
--spec-draft-n-max 3 --spec-draft-n-min 0 \
--spec-ngram-map-k4v-size-n 12 --spec-ngram-map-k4v-size-m 8 --spec-ngram-map-k4v-min-hits 2 \
--spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 8 --spec-ngram-mod-n-max 12 \
--temperature 1.0 --top-k 20 --top-p 0.95 --min-p 0.0 \
--presence-penalty 0.0 --repeat-penalty 1.1 \
--reasoning on --reasoning-preserve --reasoning-effort medium \
"$@"How to run it
- Move the file into
~/.local/bin, the folder Step 3 created forllama-server. Adjust the source path to wherever you saved it:mkdir -p ~/.local/bin mv ~/swift-serve ~/.local/bin/swift-serve - If you wrote the file with a Windows editor, strip Windows line endings, or Bash fails with
bad interpreter:sed -i 's/\r$//' ~/.local/bin/swift-serve - Make it executable:
chmod +x ~/.local/bin/swift-serve - Confirm the folder is on your PATH. Debian adds
~/.local/binat login once the folder exists, so open a new terminal first. Ifwhich swift-serveprints nothing, add it yourself:echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc source ~/.bashrc which swift-serve # /home/<you>/.local/bin/swift-serve - Start the server by typing the name, from any folder:
swift-serve - Leave that terminal open. The server runs in the foreground and loads the model in 10 to 60 seconds; it is ready when the log shows
listening on http://127.0.0.1:8080. Open a second Debian terminal for the Step 7 checks and for Pi. PressCtrl+Cin the server terminal to stop it.
To test a change without editing the file, add flags after the name; they override the ones inside. Example: swift-serve -c 65536.
If you would rather not move the file, run it from its folder with ./swift-serve after chmod +x swift-serve, or with bash swift-serve.
| Flag | Value | Effect |
|---|---|---|
-ctk / -ctv |
q8_0 | KV cache at 8 bits, about half the VRAM of f16 |
--flash-attn |
on | Required for a quantized V cache |
--parallel |
1 | One slot, so the full 102,400 context goes to one session |
-cram |
12288 | Host RAM prompt cache in MiB; restores earlier agent prefixes without reprocessing |
-ngl |
all | Every layer on the GPU |
-c |
102400 | Context window in tokens. Fits a 16GB card with app rendering on the iGPU; Step 7 Part C finds your ceiling |
--spec-type |
draft-mtp, ngram-map-k4v, ngram-mod | Speculative decoding from the model’s MTP head plus two n-gram matchers that hit on repeated code and file contents |
--spec-draft-n-max / -n-min |
3 / 0 | MTP drafts up to 3 tokens per step; its draft cache uses q8_0 via –spec-draft-type-k/v |
--spec-ngram-map-k4v-* |
n 12, m 8, hits 2 | Look up a 12-token key, draft 8 tokens, after 2 prior hits |
--spec-ngram-mod-* |
match 24, min 8, max 12 | Defaults are 48 and 64; these shorter drafts reject less often |
| Sampling | temp 1.0, top-k 20, top-p 0.95, min-p 0 | Matches the model card’s example |
--repeat-penalty |
1.1 | Model card example uses 1.0; try both |
--reasoning |
on | Thinking enabled |
--reasoning-preserve |
(flag) | Keeps reasoning from earlier turns in history; stabilizes prompt caching in agent loops |
--reasoning-effort |
medium | Passed to the chat template; Swift 1.5 reports results at low, medium, and xhigh |
| –no-ui | (flag) | Disables the built-in web chat UI at http://localhost:8080. Pi uses the API, which stays on |
Step 6: Install and configure Pi
Pi connects to the running llama-server as an OpenAI-compatible endpoint through ~/.pi/agent/models.json. Pi requires Node.js 22.19 or newer; its installer adds Node if missing (Pi README).
# Pinned installer (updates later with: pi update)
curl -fsSL https://pi.dev/install.sh | sh
# or, if you manage Node yourself:
# npm install -g --ignore-scripts @earendil-works/pi-coding-agent
node --version # 22.19 or newer
mkdir -p ~/.pi/agentCreate ~/.pi/agent/models.json:
{
"providers": {
"llama-cpp": {
"baseUrl": "http://127.0.0.1:8080/v1",
"api": "openai-completions",
"apiKey": "none",
"models": [
{
"id": "swift-27b",
"name": "Swift 1.5 Qwen3.8 27B (local)",
"reasoning": true,
"input": ["text"],
"contextWindow": 102400,
"maxTokens": 32768,
"samplingParams": {
"temperature": 1.0,
"top_p": 0.95,
"top_k": 20,
"min_p": 0.0
}
}
]
}
}
}id matches the server’s --alias. samplingParams mirrors the server flags so Pi cannot silently override them per request. contextWindow and maxTokens let Pi plan compaction against the real 102,400 limit; if a Pi update rejects either field, delete it.
Run it:
cd ~/projects/some-site # any project directory
piInside Pi, open /model, pick swift-27b, and press Ctrl+S to make it the default. /thinking changes reasoning effort per session; the server’s --reasoning-effort medium applies when Pi sends none.
Pi also has a native llama.cpp router mode (/login llama.cpp, then /llama to load models from --models-dir). It does not accept -m, so per-model flags like the speculative settings would move into a llama.cpp model-preset INI. The single-model setup above keeps them all in one script.
Step 7: Verify, benchmark, tune
Follow these steps in order. The setup is working when step 4 shows every layer on the GPU and step 6 prints an answer with a tokens/sec number.
Part A: Start the server and confirm it works
- Open terminal 1. In Windows Terminal, open a Debian tab. Start the server and save its log to a file:
swift-serve 2>&1 | tee ~/swift-serve.logThe log scrolls in this window and is also written to
~/swift-serve.log. Leave this window open the whole time; closing it stops the server. - Open terminal 2. Open a second Debian tab. Run every remaining command here.
- Wait until the model is loaded. This command repeats every 2 seconds and stops when the server is ready:
until curl -sf http://127.0.0.1:8080/health; do sleep 2; done; echoIt prints
{"status":"ok"}. If it never stops, look at terminal 1 for an error and match it in the troubleshooting table below. - Confirm the model is fully on the GPU.
grep -iE 'cuda|gpu|offload|layer|buffer|mtp' ~/swift-serve.log | head -40Log wording varies between llama.cpp builds, so this search is broad. Look for:
- A layer count where both numbers match, such as
offloaded 65/65 layers to GPU. The total can differ; what matters is that no layers stay on the CPU. - A
CUDA0 KV buffer sizeof roughly 3,500MiB. - A line mentioning the MTP draft context, which confirms the MTP head loaded. If the search still returns only the MTP line, this build does not print those load details at the default log level. Skip to step 5, the reliable check: about 13,500 to 15,600MiB used means the whole model is on the GPU. Well under 12,000MiB means layers fell back to the CPU.
- A layer count where both numbers match, such as
- Check VRAM headroom.
nvidia-smi --query-gpu=memory.used,memory.total --format=csvUsed memory should be at or below about 15,600MiB of 16,376MiB, with Task Manager’s Shared GPU memory no more than about 0.3GB above its idle value.
- Send a test prompt and record the speed. Create a reusable test command once:
cat > ~/.local/bin/swift-test <<'EOF' #!/usr/bin/env bash curl -s http://127.0.0.1:8080/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"swift-27b","messages":[{"role":"user","content":"Write a PHP function that converts a page title into a URL slug."}]}' \ | python3 -c ' import json, sys r = json.load(sys.stdin) print(r["choices"][0]["message"]["content"][:800]) print("\ngenerated tokens/sec:", r.get("timings", {}).get("predicted_per_second")) ' EOF chmod +x ~/.local/bin/swift-testThen run it:
swift-testThe model thinks before answering, so the first reply can take 30 seconds or more. Write down the
generated tokens/secnumber. If this works, the server is ready for Pi (Step 6).
Part B: Pick the fastest speculative setting
Default: the script uses MTP plus both n-gram matchers, which runs at about 69 t/s on the reference RTX 4080 versus about 44 t/s with n-gram only. Repeat steps 7 and 8 after llama.cpp or CUDA updates to confirm MTP still wins.
Some users report MTP slowing llama.cpp down on Qwen3.8-27B (unsloth discussion), so measure all three options on your card. Flags added after swift-serve override the same flag inside the script.
- Test n-gram only. In terminal 1, press
Ctrl+Cto stop the server, then run:swift-serve --spec-type ngram-map-k4v,ngram-mod 2>&1 | tee ~/swift-serve.logIn terminal 2, repeat step 3, then run
swift-testand write down the tokens/sec. - Test with no speculation. In terminal 1, press
Ctrl+C, then run:swift-serve --spec-type none 2>&1 | tee ~/swift-serve.logIn terminal 2, repeat step 3, then run
swift-testand write down the tokens/sec. - Save the winner. Compare the three numbers from steps 6, 7, and 8. If the default (step 6) is fastest, change nothing. Otherwise open the script:
nano ~/.local/bin/swift-serveChange the
--spec-type draft-mtp,ngram-map-k4v,ngram-modline to the faster value (ngram-map-k4v,ngram-modornone). PressCtrl+O,Enter,Ctrl+Xto save and exit. In terminal 1, pressCtrl+Cand startswift-serveagain.
For a fairer comparison on agent work, run each test twice and use the second result; the first request after a restart includes prompt processing.
Part C: Find your context ceiling
Every 16GB card has the same total VRAM, but how much is free depends on the hardware reservation, your displays, and any apps still rendering on the card. The script’s 102,400 tokens fits the reference system with app rendering on the integrated GPU. This procedure finds the largest size that fits yours. Each 1,000 tokens of context costs about 36MB of VRAM with q8_0 caches.
- Record your idle baseline. Stop the server with
Ctrl+C, wait 10 seconds, and note Shared GPU memory for the NVIDIA card in Task Manager > Performance. - Create a depth test. It sends a prompt of exactly the number of tokens you ask for, built from llama.cpp’s source code, so the context is actually filled:
cat > ~/.local/bin/swift-depth <<'EOF' #!/usr/bin/env bash # Usage: swift-depth TOKENS Sends a prompt of exactly TOKENS tokens of source code and reports speed. python3 - "${1:-60000}" <<'PY' import json, sys, glob, os, urllib.request, urllib.error target = int(sys.argv[1]) base = "http://127.0.0.1:8080" headers = {"Content-Type": "application/json"} key = os.path.expanduser("~/.config/llama/api-key") if os.path.exists(key): headers["Authorization"] = "Bearer " + open(key).read().strip() def post(path, obj): req = urllib.request.Request(base + path, json.dumps(obj).encode(), headers) try: return json.load(urllib.request.urlopen(req, timeout=3600)) except urllib.error.HTTPError as e: print(e.read().decode()[:500]); sys.exit(1) src = os.path.expanduser("~/llama.cpp") files = sorted(glob.glob(f"{src}/src/*.cpp") + glob.glob(f"{src}/common/*.cpp") + glob.glob(f"{src}/tools/server/*.cpp")) text = "\n".join(f"// FILE: {os.path.basename(f)}\n" + open(f, errors="ignore").read() for f in files) tokens = post("/tokenize", {"content": text})["tokens"] if len(tokens) < target: print(f"Only {len(tokens)} tokens of source available; using all of it.") body = post("/detokenize", {"tokens": tokens[:target]})["content"] res = post("/v1/chat/completions", {"model": "swift-27b", "max_tokens": 4096, "messages": [{"role": "user", "content": body + "\n\nIn two sentences, what does the first file in this dump do?"}]}) t = res.get("timings", {}) print(res["choices"][0]["message"]["content"][:400]) print("\nprompt tokens:", t.get("prompt_n"), "| prompt t/s:", round(t.get("prompt_per_second") or 0), "| generated t/s:", round(t.get("predicted_per_second") or 0, 1)) PY EOF chmod +x ~/.local/bin/swift-depth - Test the first size. Start at 81920 in terminal 1, then fill about 90% of it from terminal 2:
swift-serve -c 81920 2>&1 | tee ~/swift-serve.logswift-depth 73000The chat template adds a few hundred tokens, so request about 10% less than the context size. Run one depth test per server start; a second run reuses the cached prefix, and
prompt tokensthen shows only the new part. - Judge the run. Watch Task Manager during processing and generation.
- Pass: Shared GPU memory stays within about 0.3GB of your idle baseline for the whole run, and generated speed declines only gradually with depth. On the reference card that is about 44 t/s at 80,000 tokens and about 39 t/s at 100,000.
- Fail: shared memory climbs during the run, generation collapses far below your short-prompt speed, or the server fails to load.
- Step up until it fails. After each pass, press
Ctrl+Cand try the next size with its depth test: 90112 (swift-depth 81000), 98304 (swift-depth 88000), 102400 (swift-depth 92000), 106496 (swift-depth 96000). The last size that passed is your ceiling. - Lock it in. Replace
102400with your ceiling in both files:sed -i 's/-c 102400 /-c YOUR_CEILING /' ~/.local/bin/swift-serve sed -i 's/"contextWindow": 102400/"contextWindow": YOUR_CEILING/' ~/.pi/agent/models.json
Re-run this check after a GPU driver, CUDA toolkit, or llama.cpp update, since compute buffer sizes can change between versions. For a repeatable speed baseline across updates, stop the server and run llama.cpp’s standard benchmark, which measures raw speed without speculative decoding and varies little between runs: llama-bench -m ~/models/swift15/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf -fa 1 -ctk q8_0 -ctv q8_0 -ngl 99 -p 512 -n 128 -r 5. Record pp512 and tg128 before and after each update.
Troubleshooting
| Symptom | Fix |
|---|---|
| CUDA out of memory at load | swift-serve -c 81920, then 65536 if needed. Close GPU-accelerated apps (browsers, Discord) |
invalid argument on a --spec-* or --reasoning-* flag |
Build is too old: git pull and rebuild (Step 3) |
draft-mtp errors about a missing head |
You loaded the non-MTP file; use ...IQ3_XXS-mtp.gguf |
| Template or tool-call parse errors | Remove --chat-template-file once to test the embedded template, then re-download the froggeric file |
| WSL process killed under load | free -h; raise memory in .wslconfig or lower -cram |
nvidia-smi not found or no GPU in WSL |
Update the Windows driver, wsl --update, wsl --shutdown; confirm /usr/lib/wsl/lib/libcuda.so.1 exists |
| Pi does not list swift-27b | Server not running, or id in models.json differs from --alias |
| Log warns common_fit_params: failed to fit params to free device memory | Predicted VRAM use leaves less than the 1GiB safety margin. On Windows the driver can spill the overflow into shared system RAM, which loads fine but runs slowly. Check Task Manager > Performance > GPU: Shared GPU memory above about 0.5GB means spillover. Fix in this order until the warning is gone: add –spec-draft-type-k q8_0 –spec-draft-type-v q8_0, then lower -c (98304, 90112, 81920). Optional: NVIDIA Control Panel > Manage 3D settings > CUDA – Sysmem Fallback Policy > Prefer No Sysmem Fallback, so an overflow fails at load instead of slowing silently. |
If quality disappoints: IQ3_S with MTP (12.12GB, KLD 0.0513 versus 0.0978) roughly halves the divergence from BF16. It adds about 1.7GB, so pair it with -c 65536 or lower.
Step 8: Security hardening (prompt injection and /mnt)
Claude Code, Pi, the shared MCP servers, and your project folders all run as one Debian user, so one set of controls protects all of them. Prompt injection cannot be prevented outright: any web page, GitHub issue, library doc, or file an agent reads can carry instructions. The goal is to make a hijacked agent unable to reach Windows, unable to reach your network, unable to leak secrets, and unable to slip a backdoor past you into a site you upload.
| Layer | Protects against | Applies to |
|---|---|---|
| 8.1 WSL boundary | Reading or writing Windows drives, launching Windows programs | Everything in Debian |
| 8.2 Root lock | An agent undoing 8.1 or 8.3 with sudo | Everything in Debian |
| 8.3 Network block | Reaching Windows services, your router, NAS, or other PCs | Everything in Debian |
| 8.4 Claude Code sandbox | Shell commands writing outside the project or contacting unapproved sites | Claude Code shell commands |
| 8.5 Pi rules | Pi acting on injected instructions with no approval step | Pi |
| 8.6 Credential scoping | Stolen tokens being useful | All agents and MCPs |
| 8.7 Upload gate | Backdoored PHP or JS reaching production | Your manual uploads |
Credentials that already exist in Debian. Even with no SSH or site logins, these are present and are what an injected instruction would try to steal: the Claude Code login (~/.claude/.credentials.json), any MCP tokens (GitHub, Tavily, Context7), a Hugging Face token if you ran hf auth login, and the llama API key if you created one. 8.4 and 8.6 handle them.
8.1 Close the Windows escape hatches
WSL mounts every Windows drive under /mnt and lets Linux programs launch Windows executables such as cmd.exe and powershell.exe, which run as your Windows user. Both are turned off in /etc/wsl.conf. Your workflow does not need either: Windows Explorer reads the project folders at \\wsl.localhost\Debian\home\Debian\projects, which works from the Windows side with automount off.
First, confirm systemd is running (8.3 needs it) and look at the current file so you keep any existing [user] or [network] section:
ps -p 1 -o comm= # should print: systemd
cat /etc/wsl.confWrite the new file. If the old one had a [user] section (for example default=Debian), add it back at the bottom before saving:
sudo cp /etc/wsl.conf /etc/wsl.conf.bak 2>/dev/null
sudo tee /etc/wsl.conf > /dev/null <<'EOF'
[boot]
systemd=true
[automount]
enabled=false
mountFsTab=false
[interop]
enabled=false
appendWindowsPath=false
EOFOptional: also remove WSLg (Linux GUI apps and the /mnt/wslg sockets) by adding guiApplications=false under [wsl2] in %UserProfile%\.wslconfig.
Apply it from PowerShell with wsl --shutdown, reopen Debian, and verify:
ls /mnt # no c, a, or other drive letters
cmd.exe /c ver # command not found
echo "$PATH" | tr ':' '\n' | grep -c /mnt # 0
nvidia-smi # GPU still visibleIf nvidia-smi fails, restore the backup with sudo cp /etc/wsl.conf.bak /etc/wsl.conf and run wsl --shutdown.
What stops working: code ., explorer.exe ., copying to the Windows clipboard with clip.exe, and anything that opens a Windows browser. Sign-in flows such as Claude Code’s print a URL instead; paste it into Chrome yourself. A Playwright MCP that drives Windows Chrome breaks by design; install Chromium inside Debian with npx playwright install --with-deps chromium.
8.2 Lock root
With root, an agent could remount C: (mount -t drvfs), turn interop back on, or flush the firewall in 8.3. Make every sudo call require your password, with no cached approval:
sudo grep -rn NOPASSWD /etc/sudoers /etc/sudoers.d/ # should print nothing
echo 'Defaults timestamp_timeout=0' | sudo tee /etc/sudoers.d/90-no-cache > /dev/null
sudo chmod 440 /etc/sudoers.d/90-no-cache
sudo visudo -c # must print "parsed OK"
sudo -k; sudo -n true # must say a password is requiredIf the first command finds a NOPASSWD line for your user, remove it with sudo visudo (or sudo visudo -f on the file it names). Never type your sudo password into a prompt that an agent started.
8.3 Block Windows and your local network
In the default NAT mode, Debian can reach the Windows host and every device on your LAN. This firewall blocks all private and link-local addresses while leaving loopback (llama-server, the Claude Code sandbox proxy) and DNS working. Your .wslconfig does not set networkingMode, so NAT applies; keep it that way, because mirrored mode shares Windows’ own localhost services with Debian.
sudo apt install -y nftables
NS=$(awk '/^nameserver/{print $2; exit}' /etc/resolv.conf); echo "DNS: $NS"
sudo tee /etc/nftables.conf > /dev/null <<EOF
#!/usr/sbin/nft -f
flush ruleset
table inet wsl_egress {
chain output {
type filter hook output priority 0; policy accept;
oif "lo" accept
ct state established,related accept
ip daddr $NS udp dport 53 accept
ip daddr $NS tcp dport 53 accept
ip daddr { 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 169.254.0.0/16, 100.64.0.0/10 } counter reject
ip6 daddr { fc00::/7, fe80::/10 } counter reject
}
}
EOF
sudo systemctl enable --now nftablesVerify:
curl -sS -m 5 -o /dev/null -w '%{http_code}\n' https://example.com # 200: internet works
GW=$(ip route | awk '/default/{print $3}')
curl -sS -m 5 "http://$GW" ; echo "exit $?" # fails at once: Windows host blocked
swift-test # llama-server still answersThis does not limit which internet sites agents reach. 8.4 does that for Claude Code’s shell commands.
8.4 Claude Code: sandbox shell commands, gate web tools
Claude Code’s built-in sandbox runs every shell command in an isolated namespace: writes are limited to the project folder, and network access goes through a proxy that only allows domains you list (sandbox docs). MCP servers and the WebFetch tool run outside it, so they are gated with approval prompts instead.
sudo apt install -y bubblewrap socat
sysctl kernel.apparmor_restrict_unprivileged_userns 2>/dev/null # 1 needs the AppArmor step in the docs; 0 or no output is fineIn Claude Code, run /mcp and note the exact server names (for example tavily, playwright, github, context7). Then merge this into ~/.claude/settings.json, replacing the mcp__ names with yours and keeping any keys the file already has:
{
"sandbox": {
"enabled": true,
"failIfUnavailable": true,
"allowUnsandboxedCommands": false,
"network": {
"strictAllowlist": true,
"allowedDomains": [
"registry.npmjs.org",
"github.com",
"*.githubusercontent.com",
"api.wordpress.org",
"downloads.wordpress.org",
"repo.packagist.org"
]
},
"credentials": {
"files": [
{ "path": "~/.claude/.credentials.json", "mode": "deny" },
{ "path": "~/.pi/agent/auth.json", "mode": "deny" },
{ "path": "~/.config/llama", "mode": "deny" },
{ "path": "~/.cache/huggingface", "mode": "deny" }
]
}
},
"permissions": {
"blockReadsOutsideWorkingDirectories": true,
"deny": [
"Read(//mnt/**)",
"Edit(//mnt/**)",
"Read(~/.pi/agent/auth.json)",
"Read(~/.config/llama/**)",
"Read(~/.cache/huggingface/**)",
"Read(**/.env)"
],
"ask": [
"WebFetch",
"mcp__tavily",
"mcp__playwright",
"mcp__github"
]
}
}What each part does:
failIfUnavailableandallowUnsandboxedCommands: false: Claude Code refuses to start without the sandbox, and an injected instruction cannot talk it into running a command outside it.strictAllowlist: shell commands can reach only the listed domains, with no prompt to approve new ones. Add what your projects need (for example a CDN for a theme build); keep the list short, because every allowed domain is a possible exfiltration route.credentialsand theReaddeny rules: shell commands and Claude’s file tools cannot read the token files.blockReadsOutsideWorkingDirectoriesalso stops sandboxed commands from reading your home directory outside the project.ask: every web fetch and every call to the listed MCP servers waits for your approval, so you see what is fetched and when the agent acts on web content.
Restart Claude Code and confirm the sandbox. Ask Claude (do not type these yourself) to run touch ~/sandbox-probe and curl --noproxy '*' https://example.com. The first must fail with Read-only file system and the second with Could not resolve host.
If Claude Code later refuses commands with a Sandbox is required message, the sandbox lost a dependency mid-session; reinstall bubblewrap and socat and restart Claude Code. The refused commands do not run unsandboxed.
8.5 Pi: contain by policy
Pi runs tools without asking first and has no built-in sandbox (Pi security doc). Layers 8.1 to 8.3 still bind it, because they apply to every process in Debian. On top of that:
- Keep web tools away from Pi. Do not configure Tavily, Playwright, or other web-reading MCP servers for Pi. Do web research in Claude Code, where each fetch is approved, have it save findings to a notes file in the project, read the notes yourself, then give them to Pi.
- Keep project trust on “ask”. Leave Pi’s
defaultProjectTrustat its default of"ask", and decline trust for any folder you did not create, such as a downloaded theme or plugin. - Check instruction files. Pi loads
AGENTS.mdandCLAUDE.mdfrom a folder even when trust is declined. Open them before running Pi in a folder that came from somewhere else. - Stronger option, later. For a boundary around the whole Pi process, Pi’s isolation guide covers Docker, OpenShell, and the Gondolin micro-VM.
8.6 Scope the credentials that do exist
- GitHub MCP: replace any classic token with a fine-grained token limited to the repositories it needs, read-only permissions (Contents, Metadata), and an expiry of 90 days or less. If no project needs it, remove the server.
- Tavily and Context7: use keys dedicated to this machine, with usage caps where the provider offers them, so a leaked key is cheap to revoke.
- Hugging Face: the model repo is public, so log out once downloads are done:
hf auth logout. - If something looks wrong: revoke the affected token from the provider’s website first, then investigate.
8.7 Review gate before upload
The most damaging realistic outcome is not a stolen file but a backdoor you upload yourself: an injected instruction that makes the agent add an eval(base64_decode(...)) line, a hidden admin user, or a call to an outside domain in code that looks routine. Put each project under git and review every change before it leaves Debian.
Once per project, and before every agent session:
cd ~/projects/your-site
git init 2>/dev/null; git add -A && git commit -qm "baseline before agent session"Create a scanner that flags patterns common in WordPress backdoors and malicious JavaScript:
cat > ~/.local/bin/wp-preflight <<'EOF'
#!/usr/bin/env bash
# Usage: wp-preflight [project-dir] Flags code to review before upload.
DIR="${1:-.}"
echo "== Files changed since last commit =="
git -C "$DIR" status --short 2>/dev/null || echo "(not a git repo)"
echo; echo "== Risky PHP/JS patterns =="
grep -rnIE --include='*.php' --include='*.phtml' --include='*.inc' --include='*.js' \
--exclude-dir=node_modules --exclude-dir=vendor --exclude-dir=.git \
'eval\s*\(|assert\s*\(|base64_decode\s*\(|gz(inflate|uncompress|decode)\s*\(|str_rot13\s*\(|create_function|\b(shell_exec|system|passthru|proc_open|popen|pcntl_exec)\s*\(|\$_(GET|POST|REQUEST|COOKIE)\[[^]]*\]\s*\(|wp_(insert|create)_user|add_role\s*\(|fsockopen|curl_exec|wp_remote_(get|post)|atob\s*\(|new Function|fromCharCode|document\.write' \
"$DIR" || echo "none"
echo; echo "== URLs added since last commit =="
git -C "$DIR" diff HEAD 2>/dev/null | grep '^+' | grep -oE 'https?://[^"'"'"' )<>]+' | sort -u
echo; echo "== Executable or server-config files =="
find "$DIR" -type f \( -perm -u+x -o -iname '*.phar' -o -iname '*.exe' -o -iname '*.bat' -o -iname '*.cmd' -o -iname '*.ps1' -o -iname '*.lnk' -o -iname '*.hta' -o -iname '.htaccess' -o -iname '.user.ini' \) \
-not -path '*/.git/*' -not -path '*/node_modules/*' -not -path '*/vendor/*'
EOF
chmod +x ~/.local/bin/wp-preflightBefore each upload:
cd ~/projects/your-site
wp-preflight
git diff HEADThe scanner produces false positives (wp_remote_get and base64_decode have legitimate uses), so treat each hit as a line to read, not a verdict. Confirm every flagged line and every new URL is something you asked for. When the diff is clean, commit it, then pull the files from Windows. On the Windows side, never open or run .bat, .cmd, .ps1, .lnk, .hta, or Office files from a project folder; only upload them.
8.8 Session habits
- Separate research from building. A session that reads the web should end by writing notes to a file. Start a fresh session for code changes, so injected text from a page is not sitting in the context while the agent edits files.
- Know the warning signs. Stop the session if an agent proposes contacting a domain you did not mention, decodes or writes base64 blobs, reads token or config files unrelated to the task, edits files outside the task, or asks you to approve something “to fix” a blocked command. Then run
git diff HEAD, andgit checkout -- .to discard the changes if needed.
Final check after all layers: ls /mnt shows no drive letters, cmd.exe is not found, sudo -n true asks for a password, sudo nft list ruleset shows the wsl_egress table, the two sandbox probes fail, and swift-test still answers.
