Alisher Sharipov

I'm building a local LLM that actually gets better the longer I use it

I've been running local models for a while now, and there's one thing that keeps bothering me.

· 5 min read

Every time a better model comes out, I switch to it, get everything configured again, explain my preferences, correct the same mistakes, and basically start over. The new model might be smarter, but it hasn't learned anything from the months I spent working with the previous one.

I want to see if I can change that.

I'm building Shura, a local AI coding assistant that I want to keep improving over time. Not just by upgrading the base model, but by learning from the work we do together.

And I'm trying to do it on my regular desktop, without a dedicated server or expensive GPUs.

The hardware and what I've got working

My setup is pretty modest:

  • Ryzen 5 5600
  • 32 GB DDR4
  • RX 7800 XT (16 GB VRAM)
  • Fedora Linux + ROCm

I'm currently running a Qwen3.6-35B-A3B finetune, a MoE model with roughly 3B active parameters. Most of the model sits on the GPU, and some of the experts are offloaded to system RAM.

After quite a bit of experimenting, I'm getting around 78 tok/s decode with MTP on normal coding workloads (up to ~28k context). For longer tasks, I can go up to 200k context, with roughly 1,000 tok/s prefill on a 100k-token input in my test setup.

  • 78 tok/sdecode with MTP on normal coding work, up to ~28k context
  • 200kcontext for longer tasks
  • ~1,000 tok/sprefill on a 100k-token input, in my test setup

Getting there wasn't straightforward. I tried several optimizations that looked promising on paper and turned out to make almost no difference. I'll be documenting those failures too, since I wish I'd had that information when I started.

I also cared a lot about the PC staying usable. The model always leaves 1.5 GB of VRAM free for the desktop, and if the browser or a video suddenly needs more in the middle of an answer, it gives the memory back in about a quarter of a second and keeps going. If I start a game, it gets out of the way completely. If I'm on a call, it won't start loading a huge model into VRAM. All of this works in my test setup, and I want it to feel like the model just isn't there until I need it.

Now for the part I haven't figured out yet

I don't want Shura to just remember things about me. I want it to become genuinely more useful at the specific work I do.

Say I correct the same coding mistake three times. Why should I have to correct it a fourth time? And if I spend weeks teaching it how a particular project works, why should that knowledge disappear when I switch models?

Here's what I'm planning to build:

  1. Memory

    Keep track of confirmed facts, project decisions, and useful context. Every memory should have a source, and I should be able to inspect, edit, export, or delete it.

  2. Rules

    If I keep correcting the same behavior, Shura can suggest a rule. I decide whether to accept it, and the rule has to pass tests before it becomes active.

  3. More compute when it matters

    For harder problems, generate multiple approaches and use tests to figure out which one actually works instead of trusting the first answer.

  4. Actual weight updates

    This is the part I'm most interested in. I want to train small LoRA adapters using examples I've explicitly confirmed, without replacing the quantized base model I use every day.

How one change goes through the loop Step 1: I correct the same thing again. Step 2: Shura suggests a rule and I say yes or no. Step 3: the rule is tested on my own tasks. Then it splits: if the tasks got better, the rule is kept; if anything got worse, it is rolled back. Either way the loop starts again with the next correction. 1 I correct the same thing again, in a real task 2 Shura suggests a rule I say yes or no 3 Tested on my own tasks old ones included Better keep the rule Worse roll it back then the next correction starts the loop again
Every change goes round this loop. It is kept only if it passes on my own tasks; otherwise it is rolled back.

The hope is that the model eventually gets better at solving my kinds of problems, not just better at following my formatting preferences.

A few principles I'm trying to stick to

  • Changes get tested against my own tasks before I keep them. If I find regressions, I want to be able to roll back.
  • Nothing gets used for learning unless I've explicitly confirmed it. No screen recording, background monitoring, or silently collecting everything I do.
  • Everything stays local. No cloud training or uploading private conversations.
  • The assistant shouldn't constantly hammer my SSD or make my desktop unusable.
  • When I eventually switch to a better base model, my confirmed examples, memories, and rules should come with me. I'll retrain any model-specific adapters rather than assume they'll transfer.

I'm also trying to keep this useful on ordinary consumer hardware. If the only way to make it work is to run training jobs all day on a bunch of expensive GPUs, that kind of defeats the point for me.

Why am I sharing this now?

Because I want to document the whole process, including the experiments that don't work.

I'll be posting benchmarks, failed ideas, implementation details, and hopefully some actual before-and-after results. I don't want to claim I've built a self-improving AI when I haven't even proven that the first adapter helps.

I'd also love to hear from anyone who's experimented with LoRA on MoE models, especially on AMD hardware. Or anyone working on something similar.

If you're interested in testing Shura on your own machine or contributing, that would be great too. The code is on GitHub.

One question I'm curious about: if you had a local model that you could keep improving for years, what would you want it to learn first?

Shura on GitHub Subscribe (RSS)