Fine-tuned NER model for brand detection on Reddit content

Why this is hard

I run a pipeline that reads a large volume of forum comments in a consumer-gear niche and turns them into structured aggregates: which products people actually recommend, which materials get defended, how sentiment moves over time. Every downstream number depends on one upstream step — pulling brand, product-model, and spec mentions out of the raw text. If extraction is noisy, every chart above it is decorative.

Community text breaks the assumptions that general-purpose NER models are built on. There is no capitalization discipline. Product names are abbreviated to in-group shorthand a model has never seen. Material and spec strings look like part numbers, not words. Category nouns get discussed with exactly the same syntax as brand names, so surface form tells you nothing. And the same string can be a brand in one sentence and a generic descriptor in the next.

Off-the-shelf zero-shot extraction handled the easy half. Prompted with plain label names, a general zero-shot NER model landed around 0.65 F1 on our data. Its errors were the expensive kind: it confidently tagged generic category words as brands, and it silently missed the smaller makers that are exactly what a recommendation engine needs to surface. Precision failures inflate rankings; recall failures make the long tail invisible.

Labels without a labeling team

Fine-tuning needs annotated spans, and hand-annotating a few thousand comments is a week of work I was never going to do. So a frontier LLM did the annotation and I spent the effort on making its output trustworthy instead. Total labeling bill: about $9 for ~5,000 comments.

The one design decision that made this work: never ask the model for character offsets. LLMs are bad at counting characters and will hand you spans that are off by a few positions, which is worse than no label at all because it teaches the model wrong boundaries. Instead the LLM returns the verbatim substring it found, plus the label, and deterministic code locates every occurrence and computes exact offsets. The LLM does the part it's good at — semantics — and the type system does arithmetic.

// LLM returns strings, not indices.
{ "entities": [
    { "text": "Acme Works", "label": "brand" },
    { "text": "Model 42", "label": "product model" },
    { "text": "AB-12", "label": "material spec" }
] }

// Code computes spans, validates with a schema, and drops
// anything it cannot locate verbatim in the source text.

Everything about the annotation stage is idempotent and resumable, keyed on the source document, so a rate limit or a bad batch costs cents rather than a rerun. Responses are schema-validated before they're allowed into the dataset; anything the code can't align to the original text is skipped rather than guessed at.

The tokenization problem nobody warns you about

Character offsets aren't what the model trains on — it wants token indices. That conversion is where a surprising amount of quality lives, because domain entities are full of punctuation that generic tokenizers happily shred. A spec string like AB-12 or 1.4116 has to survive as one unit; a comma glued to a brand name has to be separated from it.

/[A-Za-z0-9]+(?:[-./][A-Za-z0-9]+)*|[^\s]/g

That single punctuation-aware pattern — keep alphanumerics joined across hyphens, dots, and slashes; emit every other non-space character on its own — fixed a whole class of boundary errors. When a span still can't be aligned to token boundaries cleanly, the example is dropped. A small clean dataset beats a large approximately-correct one, and there is no way to notice a slowly-corrupted training set from the loss curve.

Real user text also brings its own hazards. Lone Unicode surrogates from broken emoji sequences crashed the tokenizer's Rust extension outright, and batched pre-tokenized encoding hit a library bug that required falling back to per-example calls. Neither had anything to do with machine learning, and both cost real hours.

Negatives are the actual product

The thing that moved precision most wasn't more positive examples — it was buying negatives on purpose. Roughly 30% of the training set is comments with no entities at all, sampled from the model's own failure vocabulary: the exact generic terms it kept mislabeling as brands. Teaching a model what isn't an entity is teaching it the boundary of the class.

More is not better, though. An aggressive expansion to ten times the adversarial negatives made the model measurably worse — it learned to be timid, and recall fell further than precision rose. The useful dose turned out to be small and targeted.

Ten runs, and the bug that ate five of them

Getting to a good model took ten training runs, and most of the failures were plumbing rather than modeling. The one worth writing down: the training pipeline feeds the encoder a per-token mask that looks exactly like a binary is-this-a-real-token flag. It isn't. It carries incremental word indices — zero for special, prompt, continuation, and padding tokens, and an increasing counter for the first sub-token of each real word.

Building it as ones and zeros produces no error, no warning, and a perfectly plausible loss curve. The model simply reads the entire comment as a single word and learns nothing about spans. Five runs died before I stopped trusting the shape of the data and read the base library's training loop.

The lesson I actually took from this: in ML plumbing, silent success is the dangerous failure mode. Assert on the shape and semantics of your batches before you spend a GPU hour on them.

Results

Training runs on a single mid-tier cloud GPU in about 24 minutes, which makes iteration essentially free — the expensive resources here were labeling and debugging time, not compute.

Scaling the encoder helped least where the model already had general knowledge (brands, +5.4%) and most where the vocabulary is purely domain-specific (product models +13.2%, spec strings +16.4%). That gradient is a useful signal in itself: it tells you which classes are bottlenecked on data rather than on capacity.

Two operational details mattered more than the headline number. First, per-class confidence thresholds instead of one global cutoff — the classes have genuinely different score distributions, and tuning them separately beat any single value.

Second, the large model consistently reached its best validation loss around epoch two and overfit after. With a dataset this small, the honest fix is early stopping and more data, not more training — and knowing that is worth more than another hyperparameter sweep.

Caveat I'd want a reviewer to hold me to: these scores are agreement with LLM-generated labels on a locked 225-example validation set, not with human-verified ground truth. It's the right metric for tracking iteration, and the wrong one to quote as absolute accuracy.

What generalizes

• Let the LLM emit strings; compute offsets in code. Almost every span-alignment bug I didn't have traces back to this one rule.
• Buy negatives deliberately, from your model's own failure vocabulary — and keep the dose small.
• Lock the validation set before the second run. Two of my early 'regressions' were just split noise.
• Read the training loop of any library you fine-tune. Data structures that look self-explanatory often aren't.
• A sub-billion-parameter model on a commodity GPU is a rounding error next to labeling and engineering time. Fine-tuning small models is far cheaper than most teams assume.
• Own the inference path: the finished model runs as a local service, so the marginal cost of extracting entities from another million comments is electricity.

The long-form version of this write-up — with the full run log, the domain specifics, and the ugly parts left in — is published on the site the pipeline feeds.