My self-hosted advisor model heard "tennis card," answered about tennis rackets, and wouldn't take the correction. So I regraded my whole model fleet.
I run a small "AI team" on my Mac: four role personas in Open WebUI, each backed by a local model served with MLX, grounded in my private documents through RAG. Private context stays on my machine; cloud models get the public stuff. The heavyweight seat on that team is the Advisor, the persona I go to with strategy questions.
I'm also a card collector. I started when I was a kid, left the hobby in 1995, and came back in 2018. So last week I asked my Advisor, then backed by DeepSeek's R1 Distill 70B, a simple question about João Fonseca, the Brazilian tennis player.
Me
should I buy his tennis card thinking it will go up in value?
Advisor · thought for 175s
…several factors come into play when buying a specific tennis racket: limited editions or special releases can appreciate, standard rackets depreciate with use…
Me
I am talking about a sports card
Advisor · thought for 37s
…if you're buying a specific tennis racket as an investment, focus on limited editions, player endorsements, brand and model…
Me
Why are you saying racket when I said sports card
Advisor
…if you're referring to a "sport scard," it might be helpful to provide…
Strike one: I typed "card" and the model silently substituted "racket" in the first sentence of its own reasoning trace, then never questioned it. Strike two: I corrected it explicitly, and instead of re-reading my message, it re-read its own previous answer and repeated the racket analysis almost verbatim. Strike three: when I asked why, its reasoning garbled my words into "sport scard" and the reply broke off mid-thought.
Strike two is the one that matters. A model that misreads a word is having a bad day. A model that treats your explicit correction as confirmation of its own earlier output has a structural problem, and it turns out this one is known: the R1 distills are Llama and Qwen models fine-tuned on DeepSeek-R1's chain-of-thought traces, which were overwhelmingly math and code. The distillation buys benchmark reasoning and quietly regresses conversational instruction-following. The long reasoning channel tends to re-derive from its own prior output rather than your latest message, which is precisely the loop I got stuck in, at 37 to 175 seconds of "thinking" per swing.
The memory math
Two quantization decisions did the real work here. First: never download full precision for local inference. The bf16 copy of a model doubles the RAM and halves the speed (Apple Silicon inference is memory-bandwidth-bound) for a quality difference you can't measure; 8-bit is already the "no compromises" tier, and 4-bit is where most of the value lives. Second: gpt-oss ships with its expert weights natively in MXFP4, which is over 90% of the parameters, so the only real variant choice is what happens to the remaining scaffolding. The Q4 build quantizes that too and saves 3 GB over the bf16-scaffold build, with the knowledge-bearing experts bit-for-bit identical.
To make the 120B fit alongside everything else, the 27B dropped from 8-bit to 4-bit:
gpt-oss-Q4 - 58GB
qwen 3.8 27b 4Bit - 15GB
Llama 3.1 8B - 9GB
82 GB of weights under a 96 GB wired limit, with room for KV caches. The dedicated server also fixed a quieter problem: MLX servers handle one request at a time, so when three personas share a model, a long Advisor prompt makes the messaging agent go silent. Deep-reasoning seats and latency-sensitive seats shouldn't share a lane.
Same question that started all this, asked to the rebuilt Advisor: it understood "card" on the first pass, pulled context from my own documents, declined to invent a price it didn't have data for, and told me what evidence would actually settle the question. Twelve seconds, end to end. The old model spent fifteen times longer being wrong about rackets.
What I'd generalize from a week of this: benchmarks measure the wrong thing for a daily-driver persona, so keep a personal failure case and test every candidate against it, the way you'd check a card in hand instead of trusting the listing photo. Match the model to the seat, not the leaderboard: reasoning depth for the advisor chair, latency and tone for the messaging chair. And the cheapest experiment is the model you already have loaded; run that one before downloading 62 GB.
