How Hivewire turns forty articles into one story
When something happens, forty outlets write about it and every raw news feed hands you forty entries. Our entire product is collapsing those into one cited story. Here's how the clustering works at the level that matters for trusting the output: the architecture and the judgment calls. The config values stay ours.
The problem is harder than "group similar headlines"
Two articles about the same event routinely share almost no vocabulary. A wire recap and a local dispatch can describe the same explosion in entirely different words. Meanwhile two articles about different events can be nearly identical: "Fed cuts rates" and "Markets rally after Fed cut" sit next to each other in any similarity space, but one is the decision and the other is the reaction. A reader who asked for one story per event means exactly that distinction. Any system built purely on text similarity fails in both directions at once: it splits stories that belong together and merges stories that don't. That's the actual problem, and it shapes every stage below.
Stage one: a shortlist, not a verdict
Every incoming article is embedded using its headline and opening, where news writing packs the facts. We compare that embedding against the events we're currently tracking. Old events age out of the comparison set: news has a working memory, and yesterday's storm shouldn't be a candidate for today's earnings call.
Vector similarity is genuinely good at two things: definitely not and almost certainly yes. Articles with no plausible match become new stories immediately. Articles that match one event overwhelmingly, with no runner-up close behind, attach directly. That resolves most of the volume cheaply and quickly, which lets us afford to be careful with the rest.
Stage two: a model reads the ambiguous ones
The middle band contains plausible matches and cases with several candidates too close to call. A language model reads the article against each candidate event and returns one of three verdicts: same event, related but distinct, or unrelated. That three-way split is the heart of the system. "Related but distinct" is what similarity scores cannot express: the rate cut and the market rally are part of one storyline and should be linked, but they are not one story. Collapsing them loses information a reader can never get back. Same-event verdicts attach the article; related verdicts connect the stories without merging them; unrelated spawns a new story even at high similarity.
Stage three: the system repairs itself
Clustering runs in real time, so every decision starts with incomplete information. The first two articles about a developing event can look unrelated until the third makes the connection obvious. Matching decisions aren't final. A merge pass continuously compares events with the same verdict machinery and folds duplicates together when the evidence arrives. When one new article matches several existing events at once, we treat that as a signal that those events were fragments of one story and queue them to merge. Early fragmentation heals instead of persisting.
Every story gets classified, and re-scored as it grows
New stories receive a headline, summary, topics, geography, and category. We also score significance on a 0–10 scale using explicit factors: the scope of who's affected, historical weight, transformative potential, and plain news value. Significance isn't static; a story that keeps attracting coverage gets re-scored, and the bigger a story already is, the more scrutiny its updates get. That score feeds sort=top, adjusted for corroboration and freshness. It is an editorial judgment made mechanically, the same way every time.
Why commentary can't swallow the news
Every story carries a category: news, feature, analysis, opinion, or podcast. Each category comes with different clustering rights, and the asymmetry is deliberate.
An opinion column about the rate cut belongs with the rate-cut story as one of its cited perspectives, so commentary that matches a news event joins it. But the reverse is forbidden: only news stories can act as cluster targets. A standalone op-ed becomes its own story, marked opinion, and it never accumulates other articles or becomes a merge target. Without that rule, a widely syndicated column titled "The Fed's big mistake" could start absorbing actual rate-cut reporting. The resulting "story" would be an op-ed wearing forty citations. One-directional clustering is why category=news on /v1/stories gives you events, while category=opinion,analysis gives you the discourse about them.
What we deliberately don't do
We don't return full article text. A story is a summary plus citations, and the reporting stays with the outlets that wrote it. We also don't cluster across a scraped long tail. The corpus is a curated set of publications that do original reporting because clustering quality depends on corpus quality; every deduplication mistake in a feed of aggregator copies multiplies.