Relevance, not keyword hits

The first version of Clairwire had a simple definition of coverage: if a source was set up to watch a client, everything it returned went into that client's feed. Search X for the client's name, and every result counts.

That works until real clients show up. A client with a common name gets matched by strangers who share it. A competitor mentioned in passing shows up as if the article were about them. A client who writes for a publication gets every article they've written, because their name is in the byline. None of that is coverage, but all of it was being counted, and "counted" means it showed up in the mention total, fed the sentiment chart, and could trigger an alert. The numbers were drifting away from what people were actually saying about the client, and nothing on the screen said so.

The mistake underneath is worth naming, because it turns up in lots of systems: treating how data was fetched as proof of what it is. A search query decides what gets collected. It can't tell you whether each result is relevant.

Five answers instead of one#

Now, when a mention is analyzed, the model has to make an explicit relevance decision and give its reason. There are five possible answers:

Only direct and indirect count toward a client's stats, alerts, reports and briefs, and that rule is enforced the same way everywhere. Right after analysis, anything else is stopped before it reaches the alerts queue, and any alert already raised on it is resolved:

// Context is retained for labeled briefing context, never client alerts.
if (!['direct', 'indirect'].includes(analysis.relevance ?? 'uncertain')) {
  await this.prisma.alert.updateMany({
    where: { organizationId, clientId, mentionId, status: { in: ['new', 'acknowledged'] } },
    data: { status: 'resolved', resolvedAt: new Date() },
  })
  return
}

Look at the fallback in analysis.relevance ?? 'uncertain'. A missing decision is treated as uncertain, not as coverage. When the system doesn't know, it doesn't count.

Giving context its own category matters more than it might seem. Clients often want to follow a competitor or an industry topic. Mixing that into their own numbers would be wrong, but throwing it away would be wrong too. A separate bucket lets it be used where it helps, as background in the daily brief, without inflating anyone's coverage.

Don't hide the evidence#

It would have been easy to go one step further and only show people the mentions the system is sure about. I decided against it. The mentions inbox still shows uncertain and context items, clearly labeled. The whole point of recording a reason for every decision is that a person can check it, and nobody can check what they can't see.

When the model gets it wrong, an analyst can mark a mention as "not about this client". That one action moves it out of the client's coverage and resolves any alerts it raised, and undo puts everything back the way it was. The model's original decision is never overwritten. The human correction is stored on top of it, so you can always see both what the system thought and what a person decided.

Writing a brief from a day of mentions#

The daily brief is where relevance pays off most visibly. A busy client can collect hundreds of mentions in a day, and a brief that tries to summarize all of them isn't a brief anymore.

So the day's direct and indirect mentions are ranked twice: once by reach (how many people are likely to see them) and once by velocity (how fast they're spreading). The top of each ranking is merged into one list, with duplicates removed. A few context stories, such as competitor news, are added separately and labeled as context, so they can inform the brief without being presented as the client's own coverage.

The model that writes the brief only sees that short, pre-selected list, together with the reason each story made the cut. It never sees the raw firehose. The output is more focused that way, and the most expensive and least predictable part of the system, the language model, only ever works on a small input that the rest of the code has already vetted.

The weekly summary that a PR person reviews and then forwards to their own client follows a related rule. The model only gets headlines, source names and numbers, never the full text of the mentions. That text was written by strangers on the internet, and the summary ends up in an email sent outside the organization. Keeping it out of the prompt means nothing a stranger wrote can steer what gets sent.

Shipping the change without emptying anyone's dashboard#

Changing what counts as coverage is a data migration, even if the database schema barely changes. Apply the stricter rule to existing data, and a client could log in the morning after the deploy to find half their history gone. It would be technically more accurate, it would look exactly like a bug, and it would be a terrible way for anyone to find out the product had changed.

So the rollout had three parts.

Old mentions keep their old status. Every mention analyzed before the change was tagged legacy, and legacy still counts as coverage alongside direct and indirect. The shared filter every coverage query goes through spells that out:

/** Keep historical coverage during rollout; new analysis must establish relevance. */
export const clientCoverageWhere = {
  status: { notIn: ['ignored', 'irrelevant'] },
  analysis: { is: { relevance: { in: ['direct', 'indirect', 'legacy'] } } },
} satisfies Prisma.MentionWhereInput

Keeping that rule in one exported constant, rather than repeating it in every query, is what made the migration safe. Changing what "coverage" means is a one-line edit in one file.

New mentions follow the new rules from day one. Anything analyzed after the deploy has to earn direct or indirect.

Reclassifying the backlog is a deliberate action. Old mentions can be re-analyzed under the new rules, but only when someone chooses to, for a specific client and period. Alerts are switched off for that run, so a months-old story doesn't page anyone just because it was scored again. None of it happens automatically just because the code shipped.