Running an LLM, then forgetting it

llmoxide is a private chat app for open-weight language models. The conversation never leaves your machine and never touches the disk, and when you close the window it gets overwritten.

Everything in it is written from scratch in Rust, with no ML libraries: the model file loader, the decoders for compressed weights, the tokenizers, and the GPU kernels. It runs Gemma 4, dense Qwen3, and Qwen 3.8 27B, whose hybrid layers we'll get to. Every model is checked against llama.cpp tensor by tensor, and greedy output matches it byte for byte.

Writing an engine from nothing turns out to be a great way to learn what an LLM is. There's a lot less to it than the hype suggests. It's a short list of operations, each simple, and a long list of details that will quietly ruin the output if you get one wrong. This post walks the short list, mentions the worst of the details, and then explains why the app exists at all.

At the bottom of the page is a lab: a 135M-parameter model running in your browser on the same kind of code this post describes, with its attention patterns, intermediate predictions and memory on display. You can load it now and follow along, or read first.

If you're not a programmer, How do neural networks work? covers the first half of this without the code.

1A neuron is a dot product

Multiply two lists of numbers element by element and add up the results. That's a dot product, and it measures how much two vectors point the same way. A row of dot products, one per row of a matrix, is a matrix-vector product:

/// y = W x. W is stored [out][in]: one row per output.
fn matvec(y: &mut [f32], w: &[f32], x: &[f32]) {
    let n = x.len();
    for (o, out) in y.iter_mut().enumerate() {
        *out = dot(&w[o * n..(o + 1) * n], x);
    }
}

Each output asks one question of the input (how much does x look like this row?), and the rows are what training learned. Nearly all of an LLM's work is this loop. Qwen 3.8 27B runs it about 27 billion times per token, once for every weight. Almost everything else in this post is about feeding it the right vectors, and doing it fast.

2Layers, and the bend between them

Stacking two matrix multiplies gets you nothing: A(Bx) is just (AB)x, one matrix. To stack usefully you need a non-linear function between them. Modern LLMs use SiLU, x · sigmoid(x), which is roughly ReLU with the corner rounded off.

The feed-forward block in every model in this post is a gated variant called SwiGLU. It uses three matrices instead of two: one to make values, one to make gates, and one to bring the result back down to the model's width.

let gate = Wg · x;     // d_model -> ffn_dim, e.g. 576 -> 1536
let up   = Wu · x;     // d_model -> ffn_dim
for i in 0..ffn_dim {
    gate[i] = silu(gate[i]) * up[i];   // each value scaled by its own gate
}
let out  = Wd · gate;  // ffn_dim -> d_model

This block holds most of a model's parameters: roughly three quarters of everything outside the embedding table. People often describe it as where the model's facts live. Attention moves information between positions, and the FFN transforms it in place.

3Text in, vectors out

Matrices want vectors, and we have text. Two steps bridge the gap.

Tokenizing turns text into integer ids. Every model here uses byte-pair encoding: start from the raw bytes, then repeatedly merge the adjacent pair with the best rank in a table of learned merges (49,152 tokens and 48,900 merges for the lab's model) until nothing merges. Common words end up as one token, and rare ones as several.

Before any merging, a pre-tokenizer splits the text into chunks, and merges never cross a chunk boundary. This is the first of many places where an engine can be silently wrong. The lab's model, SmolLM2, cuts every digit out on its own first and only then applies GPT-2's split. Qwen's split differs in how it handles punctuation before a word. Run a model's vocabulary through the wrong split and nothing errors. You get valid token ids, just not the ones the model was trained on, and the output quietly gets worse. So llmoxide's splitters are ports of llama.cpp's, and they're tested id for id against llama-tokenize on thousands of tokens of prose and code.

Embedding turns each id into a vector, and it's the cheapest step in the model: row id of a big table. No arithmetic at all.

for (i, &tok) in tokens.iter().enumerate() {
    embed.dequant_row_into(tok as usize, &mut h[i * d..(i + 1) * d]);
}

Training arranges that table so tokens used in similar ways point in similar directions. In the lab, click a token to see its 576 numbers and its nearest neighbors. king's are King, kings, queen, emperor and prince.

4Keeping the numbers in range

Pass a vector through dozens of layers and its scale drifts. RMSNorm rescales a vector to unit root-mean-square, then multiplies each element by a learned gain:

fn rms_norm_mul(x: &mut [f32], w: &[f32], eps: f32) {
    let ms = x.iter().map(|v| v * v).sum::<f32>() / x.len() as f32;
    let scale = 1.0 / (ms + eps).sqrt();
    for (v, g) in x.iter_mut().zip(w) { *v *= scale * g; }
}

Every block normalizes its input before working on it (that's called pre-norm), and one last norm runs before the output. Gemma also normalizes each block's output before adding it back. Qwen and llama don't, so a shared code path has to know which model it's running.

5Attention

Everything so far works on each position by itself. Attention is the one step where positions exchange information, and it's what lets it find the noun it refers to.

Each position projects its vector three ways: a query (what am I looking for?), a key (what do I contain?) and a value (what do I hand over if picked?). A position scores its query against every earlier key with a dot product, turns the scores into weights that sum to 1 with a softmax, and takes the weighted sum of the values. Here's the loop the lab runs, lightly trimmed:

for i in 0..t {                              // each new position
    let seen = base + i + 1;                 // causal: it sees 0..=itself
    for h in 0..n_heads {
        let kvh = h / group;                 // GQA: heads share K/V
        let q = &q_all[i * q_dim + h * hd..][..hd];
        for p in 0..seen {
            scores[p] = dot(q, key(p, kvh)) / (hd as f32).sqrt();
        }
        softmax(&mut scores[..seen]);
        let out = &mut attn[i * q_dim + h * hd..][..hd];
        for p in 0..seen {
            for (o, v) in out.iter_mut().zip(value(p, kvh)) { *o += scores[p] * v; }
        }
    }
}
// then: h += Wo · attn

A few details matter:

• Causal. Position i only sees positions up to i. That's what makes generation possible: the past never needs recomputing when the future arrives.
• Multi-head. The query is cut into heads (9 of width 64 in the lab's model), and each head attends separately with its own pattern. In the lab you can watch different heads specialize: some look at the previous token, some at the start of the text, some at the matching word.
• Grouped-query attention. Several query heads share one key/value head: 9 share 3 in SmolLM2, and 24 share 4 in Qwen 3.8. Keys and values are what gets cached (more in part 2), so this divides the memory per token by the group size.
• The 1/√d. It keeps dot products of wide vectors from saturating the softmax. Gemma 4 is the exception: its scale is folded into its norm weights, and applying 1/√d on top flattens attention until every position emits the same filler token.

6Where things are: RoPE

Attention as written doesn't care about order: shuffle the keys and the weighted sum comes out the same. Position has to be put in, and every model here does it with rotary position embeddings. Treat the query and key as lists of 2D pairs, and rotate each pair by an angle proportional to the token's position, with each pair spinning at its own frequency:

fn rope(head: &mut [f32], pos: usize, base: f32, mode: Rope) {
    let half = head.len() / 2;
    let mut theta = pos as f64;
    for i in 0..half {
        let (sin, cos) = (theta.sin() as f32, theta.cos() as f32);
        let (a, b) = match mode {
            Rope::Adjacent => (2 * i, 2 * i + 1),  // llama
            Rope::Halves   => (i, i + half),       // qwen (NeoX)
        };
        let (x0, x1) = (head[a], head[b]);
        head[a] = x0 * cos - x1 * sin;
        head[b] = x0 * sin + x1 * cos;
        theta *= (base as f64).powf(-2.0 / head.len() as f64);
    }
}

Rotating both vectors changes their dot product only by the difference of their angles. So the score between a query at position m and a key at n depends on m − n, which is how far apart they are, and not on where they sit in the text. Try it:

Note the two pairings in the code. Qwen rotates dimension i with i + d/2, while llama-family files rotate neighbors, because llama's converter permutes the Q and K weights to match. Pick the wrong one and nothing crashes. The model just produces confident garbage, a failure mode this post will keep coming back to.

7The block, and the residual stream

Now put it together. The model keeps one vector per token, the residual stream, and each block reads from it and adds back to it. A block never replaces the stream. It only writes a correction into it.

And that's the whole model. The lab's model is the next few lines, run 30 times with d_model = 576:

h = embed(tokens)
for layer in layers:
    h = h + attention(rms_norm(h) * attn_norm)
    h = h + ffn(rms_norm(h) * ffn_norm)
logits = W_out · (rms_norm(h) * output_norm)    // last position only

The additive stream is why the logit lens works: since every layer only adds to h, you can push the stream through the output head after any layer and ask what the model would say if it stopped there. In the lab, for The capital of France is, SmolLM2 predicts . or a space for most of its depth, then the, and Paris only shows up at the very last layer.

8From logits to text

The output head is one more matrix multiply, a score for every token in the vocabulary. A softmax turns the scores into probabilities. Temperature divides the scores before the softmax, sharpening the distribution below 1 and flattening it above. Top-p keeps the smallest set of tokens whose probability adds up to p, so the long tail of nonsense can never get sampled. Then you draw one token, append it, and run again.

Generation stops at an end-of-turn token. A chat is plain token sequence underneath, laid out by a chat template. SmolLM2 and Qwen use ChatML:

<|im_start|>system
You are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>
<|im_start|>user
What is the capital of France?<|im_end|>
<|im_start|>assistant

Those markers are single control tokens. llmoxide inserts them by id and tokenizes the text between them, so a user who types <|im_end|> gets that string shattered into ordinary pieces, and can't forge a turn boundary. (llama.cpp parses special-token spellings out of text by default, so there, typing one in really does end the turn.) The template has its own traps. Some Qwen files omit the "add BOS" flag, and the default for that differs by vocabulary type. Prepend a stray <|endoftext|> and the model reads your prompt as the tail of some other document. That one was found by diffing greedy output against llama.cpp, after it had looked like a numeric near-tie.

9What Qwen 3.8 adds

Everything above is a complete, working LLM: it's what runs in the lab. Qwen 3.8 27B keeps that skeleton and changes two things that matter, one small and one big.

Attention, with a gate and partial rotation

Its attention layers have three changes:

• QK-norm. Each query and key head gets RMS-normalized (with learned per-dimension gains) before rotation, which keeps attention scores in a stable range as the model gets deep.
• Partial RoPE. Only the first 64 of each head's 256 dimensions rotate, with base 10,000,000. The other 192 are position-free, so part of every head matches on content alone.
• A gate on the output. The query projection is fused: it emits [query | gate] per head. After the weighted sum of values, each head's output is multiplied by sigmoid(gate), so a head can switch itself off for a token. Forget the gate and every activation still looks plausible, and the output is subtly wrong.

The hybrid: three of every four layers aren't attention

Here's the big change. Attention's memory is the KV cache, a key and value for every past token, in every attention layer, forever. It grows with the context, and at 262k tokens of context it's enormous. Qwen 3.8 makes only every fourth layer full attention. The other 48 are gated delta-net layers, a kind of linear attention with a fixed-size memory.

A delta-net layer replaces "keep every key and value" with a single matrix S per head (128×128 here) that maps keys to values: an associative memory. To read it, multiply by the query. To write, it uses the delta rule: ask what S currently recalls for this token's key, and nudge it toward the actual value by however much it was wrong. Here's llmoxide's CPU reference, per token and value head:

let g = (a[h] * softplus(alpha[h] + dt_bias[h])).exp();  // decay, in (0, 1)
let b = sigmoid(beta[h]);                                  // how hard to write
for x in s.iter_mut() { *x *= g; }                         // forget a little
for j in 0..128 {                                          // row j of S: value dim j
    let row = &mut s[j * 128..(j + 1) * 128];
    let pred  = dot(row, k);                               // what S recalls for k
    let delta = (v[j] - pred) * b;                         // how wrong that was
    for (r, &kk) in row.iter_mut().zip(k) { *r += delta * kk; }
    o[j] = dot(row, q) / (128f32).sqrt();                  // read with the query
}

Around that core, the fused Q/K/V stream goes through a short causal convolution (width 4) and a SiLU first, q and k are L2-normalized, and the output passes through a gated RMSNorm before its projection. The cost is the point. Per token, a delta-net layer does the same work at position 10 as at position 100,000, and its state never grows: about 3 MB per layer, roughly 150 MB for all 48, at any context length. The 16 attention layers keep exact recall over the whole context, and they're the only ones paying per-token memory.

This is also the one part of the model that's a recurrence: token t's state needs token t−1's. Prefill can batch the matrix multiplies around it, but the loop itself runs in order. On the GPU it's one dispatch with a workgroup per value head, looping over the tokens inside.

It also comes with its own set of traps, every one of them silent:

• There are 48 value heads but only 16 key/query heads. Value head h uses key head h % 16 (ggml repeats by tiling). The h / 3 you'd guess from Hugging Face's repeat_interleave produces plausible garbage.
• ssm_a is stored already negated (the converter bakes in -exp(A_log)). Negate it again and the state explodes.
• The L2 norm's epsilon floors the norm (x / max(‖x‖, ε)). It isn't added under the square root the way RMSNorm's is.
• Some metadata keys are misnamed: ssm.time_step_rank is actually the value-head count.

With all of that right, llmoxide's 27B matches llama.cpp at 567 intermediate tensors within 4×10⁻⁴, and greedy decoding is byte-identical.

10Loading weights

Models ship as a single GGUF file. It's a simple format: a header, typed key/value metadata (architecture, sizes, the tokenizer's whole vocabulary and merge list, the chat template), a table of tensors, and then the tensor bytes, each aligned.

The weights are quantized: stored as small integers in blocks that share a scale. Q8_0 keeps 32 signed bytes and one f16 scale per block. The k-quants used for big models (Q4_K, Q6_K) nest blocks in 256-weight super-blocks with their own scales, getting down to 4.5 and 6.6 bits per weight. You never decompress the whole thing. Each dot product decodes its row on the fly:

GgmlType::Q8_0 => {
    let mut acc = 0f32;
    for (bi, blk) in row.chunks_exact(34).enumerate() {
        let d = f16_at(blk, 0);                           // the block's scale
        let qs: &[i8] = bytemuck::cast_slice(&blk[2..]);  // 32 weights
        let xb = &x[bi * 32..bi * 32 + 32];
        acc += d * qs.iter().zip(xb).map(|(&q, &b)| q as f32 * b).sum::<f32>();
    }
    acc
}

The loader reads the header and tensor table, checks every tensor's shape against the config (so a converter quirk fails at load time instead of 60 layers into a reply), and then hands the data region over. Natively, that's an mmap: no copy, and the OS pages weights in as they're touched. The GPU path then uploads it, and at 22 GB that turned up two failures that return all zeros with no error. Mapping every buffer at once doubles memory with staging copies and the writes just get dropped, so llmoxide uploads one buffer at a time and reads back samples to confirm. And a whole forward pass submitted as one command buffer (about 1,500 dispatches) trips Metal's GPU watchdog, so it's chunked at 64.

The browser is stricter still. wasm32 can address only 4 GB, and the larger models don't fit. So the header gets parsed from a prefix of the file, which is enough to plan the whole upload, and the weights stream from the JS File to the GPU 32 MB at a time without ever entering wasm memory. One more trap: Blob.slice must get its offsets as f64, because the i32 overload saturates past 2 GB, and every tensor after that loads from the wrong place.

11Prefill and decode

Generating a reply has two phases. Prefill runs the whole prompt through the model at once. Decode then runs one token at a time, each one depending on the one before. They're the same function. In the lab's engine the only difference is t, the number of rows going through the matrix multiplies:

pub fn matmul(y: &mut [f32], w: &TensorView, x: &[f32], t: usize, row: &mut [f32]) {
    for o in 0..w.out_dim() {
        w.dequant_row_into(o, row);             // read and decode this row ONCE
        for i in 0..t {                         // ...then use it for every token
            y[i * n_out + o] = dot(row, &x[i * n_in..(i + 1) * n_in]);
        }
    }
}

The expensive part is reading the weights. In decode, t = 1, so every weight in the model is read from memory to do one multiply-add. Decode speed is therefore set by memory bandwidth, not by compute. For the 27B that's simple arithmetic: 22.4 GB per token at 9.4 tokens a second is about 210 GB/s, and the M4 Pro's measured ceiling for a kernel that only reads those bytes is 225–242 GB/s. There's at most 10% left to find in decode, and that's why llmoxide stopped looking.

Prefill shares each weight read across every prompt token, so it becomes compute-bound, and the right tool is a real GEMM. llmoxide has a hand-written Metal one that dequantizes a tile of weights once into threadgroup memory and runs it on the GPU's 8×8 matrix units at about 4.4 TFLOP/s. That took 27B prefill from 14 to 79 tokens a second.

The lab shows both phases. Its prefill and per-token decode times come straight from the engine running in your tab.

12The KV cache

Because attention is causal, a token's keys and values never change once computed. So they're kept: every attention layer appends each new token's K and V to a cache, and decode computes only the newest token's. Without the cache, every token would re-run the whole conversation.

The cache is the conversation's memory, and its size is easy to work out: 2 × layers × kv_heads × head_dim × bytes per token.

Keeping the cache between turns means a follow-up question only prefills the new tokens. The lab's chat shows this: the second turn reports something like prefilled 19, reused 81. But reuse only works if the new prompt is token for token an extension of the old one, and that's harder than it sounds. llmoxide once regressed because a replayed assistant turn encoded an empty thinking block as "\n" + "" + "\n", which tokenizes to [198, 198], while the original generation prompt had written "\n\n", which BPE merges into the single id 271. Same string, different tokens, and the whole conversation re-prefilled every turn.

The delta-net layers make it stricter. Their state has already absorbed every token, so it can't be rewound to an earlier point. Cache reuse is append-only.

The cache is also the conversation, sitting in memory as numbers. Hold that thought.

13Knowing it's right

Look back at how many traps in this post don't crash. Wrong pre-tokenizer, wrong RoPE pairing, a missing gate, a doubled negation, a stray BOS: every one produces fluent, confident, slightly worse text. "The output looks fine" is not a test.

So llmoxide is checked against llama.cpp the hard way. llama-eval-callback dumps every intermediate tensor, and a validator compares each one to llmoxide's CPU reference path, which passes everything it computes through a trace hook under llama.cpp's own names. The GPU path is then diffed against the CPU path at every checkpoint (771 of 771 match on the 27B), and greedy output is compared byte for byte. Compare elements against the tensor's own magnitude rather than whole-tensor sums, which cancel. And check a token other than the first, because position 0 only attends to itself and matches even when attention is completely broken.

The lab's engine got the same treatment before going in this page. Greedy output on SmolLM2-135M and on Qwen3-0.6B is byte-identical to llama-completion.

14Where your conversations go

People tell AI things they don't tell anyone else: symptoms, money trouble, relationship problems, legal worries, the code from work they're not supposed to paste anywhere. Then they treat the conversation as if it evaporated.

A hosted model turns every prompt into a record on someone else's server. It may be logged for abuse monitoring, kept for a retention period, read by a reviewer, or held for litigation. In 2025, a US court ordered OpenAI to preserve ChatGPT conversations, including ones users had deleted, as evidence in a copyright case. You can't delete what you don't hold, and whatever a provider's policy says today, the copy is theirs.

A local model fixes the server and nothing else. Inference stays local, but the conversation still leaves copies all over the machine:

Most of that list isn't a bug in anyone's app. It's what a normal program on a normal OS leaves behind.

15Why it matters more now

Recovering that residue used to take a forensics specialist. Now it takes a sentence. An AI agent with shell access will happily run every one of these, and it knows where to look better than most people do:

$ sqlite3 ~/Library/Application\ Support/<some-chat-app>/*.db "select content from messages"
$ grep -i "ollama run\|llama-cli -p" ~/.zsh_history
$ sudo strings -n 12 /dev/nvme0n1p3 | grep -i "diagnos"   # an unencrypted Linux swap partition
$ tmutil listbackups   # ...and the history database is in every one of them

So the person who can read your old conversations is no longer just a lab with a warrant. It's anyone who gets a few minutes at your unlocked laptop with an agent, malware that ships with one, a stolen or resold machine, a backup drive, or a family computer. As agents make everyone capable of this kind of digging, the only reliable defense is for the residue not to exist in the first place.

Browsers solved a version of this years ago. Incognito mode doesn't hide you from the network. It just doesn't leave a history on the machine. AI needs the same thing.

16What llmoxide does

The design goal is narrow and testable: after you close the window or press New chat, nothing on the machine holds the conversation. Not on disk, not in memory, not on the GPU.

Every claim like this gets tested, because "we clear the cache" is exactly the kind of statement that's easy to make and easy to get wrong. Resetting a KV cache's length looks like clearing it and isn't. A clear_buffer that was queued but never submitted looks the same from the CPU side. So wipe_check prefills a prompt, confirms the GPU buffers are full of it, wipes, and reads every buffer back. On the 27B: 41,487,330 nonzero words across 147 buffers before, 0 after. It refuses to pass if nothing was there to begin with. The read-only file tools get the same treatment: every read restores the file's access time, and a checker confirms that atime, mtime and ctime are unchanged to the nanosecond and that no filesystem event fired.

17Incognito, not invisible

It's not perfect, and it says so. What it doesn't cover:

• That you used it. The model file's access time, the process launch in the system log, the GPU running hot. What was asked can be made unrecoverable. That something was asked can't.
• Root on a running machine, or anything already watching live: a keylogger, screen capture, an EDR agent logging every open.
• Pixels and input. Rendered text sits in the compositor until it's repainted, and keystrokes pass through the OS input system like any app's.
• A SIGKILL before the wipe, though locked pages still never reached swap, and the kernel zeroes freed pages before reusing them.

That's the same deal browser incognito offers, and it's the right one. The threat that's growing fastest is the cheap, after-the-fact one: someone, or something, going through a machine later. Closing that off for everyone matters more than chasing an attacker with root, who has easier options anyway.

The lab below is weaker than the app, and it's worth being exact about how. It runs in a browser tab, which can't mlock, and the text you type also lives in the page's DOM and in the message posted to the worker, which are the browser's copies, not the engine's. What it does do is real: the engine's heap zeroes on free, and Forget overwrites both KV caches and shows you the count of nonzero floats before and after. Nothing you type is sent anywhere. The only network request is the one-time download of the model file.

This is the dense transformer from part 1, written as a small Rust crate (crates/lab) on top of llmoxide's GGUF loader, tokenizer, and sampler. It's compiled to wasm with SIMD and runs in a Web Worker, so the page stays responsive. Every forward pass can record its attention weights and residual stream, and that's what the panels draw.

It's a 135M-parameter model, 200 times smaller than the 27B. It's coherent for a sentence or two and then wanders. That's what a model this size is like, and it's small enough to download in seconds and fast enough to chat with on one CPU thread.

Try the real thing

The lab is the teaching version. The app is the real one: Qwen 3.8 27B or Gemma 4 on your GPU, in a native window that leaves nothing behind. It's one Rust binary with no Python and no runtime to install:

git clone https://github.com/bigmah/llmoxide && cd llmoxide
cargo run --release -p llmoxide-hub --bin llmoxide-fetch -- gemma4-e4b-q4   # 5.3 GB, once
cargo run --release                                                        # the chat window

It's developed on macOS, and the privacy layer calls macOS APIs directly, so Linux and Windows need small ports before the app builds there. The repo's privacy doc has the full threat model and how each claim is checked. Issues and PRs are welcome.