Method
How we measured accuracy
One benchmark on real RC records: the test, the comparison and how to read it.
Summary
On 150 held-out RC records matched against five insurer masters (701 scored labels), our system picked an exact listing 77.0% of the time. The best standard matcher picked one 66.2% of the time.
The paired gap is +10.8 points (95% confidence interval +5.7 to +16.3). When our system returns a listing, it is right 84.9% of the time, against 66.9%.
Its HIGH confidence level is not yet better than a standard matcher's score at the same coverage.
Labels were made by AI agents to written guidelines, and the test set is deliberately hard. What to keep in mind, and the next steps for the benchmark, are at the end of this page.
Results
| System | Exact | Exact or acceptable |
|---|---|---|
| Our system | ||
| As it runs today | 77.0% | 80.7% |
| Standard matchers | ||
| Filters, insurer model name, then BM25 (tuned; the best standard matcher) | 66.2% | 69.3% |
| Fuzzy matching with filters (tuned) | 65.5% | 67.8% |
| BM25 search with filters (tuned) | 65.2% | 68.0% |
| Plain BM25 search (default settings) | 60.3% | 61.9% |
| Plain fuzzy matching, token-set ratio (default settings) | 56.5% | 58.1% |
Reading the result
The gap
+10.8 points
Against the best standard matcher, resampling whole vehicles 2,000 times: 95% confidence interval +5.7 to +16.3. On 125 labels ours was right and it was wrong; on 49 it was the other way round. Against plain BM25 the gap is +16.7 points (+11.6 to +21.8).
Where the gap comes from
Mostly the variant. Counting brand and model only, the gap is smaller: 86.0% against 82.6% (a check on three of the five masters).
One convention differs
For CNG and LPG vehicles, the labels and our system follow different conventions. Scored the labels' way, our system gets 78.5% exact.
Confidence
Our 159 HIGH picks were right 86.2% of the time; the best standard matcher's 159 most confident picks, 85.5%. On about 44,000 real RC records, 33.3% of results are HIGH.
Speed
About 7 ms per vehicle for all five masters on a laptop CPU (6.9 ms). Standard matchers with filters take 0.4 to 2 ms. Speed is not a reason to choose ours.
Wrong picks and “no listing”
Our system
- Returns a listing
- 89%
- Right when it returns a listing
- 84.9%
- Wrong or non-exact listing (share of all labels)
- about 13%
- “No listing” (share of all labels)
- 11%
Best simple matcher
- Returns a listing
- 97%
- Right when it returns a listing
- 66.9%
- Wrong or non-exact listing (share of all labels)
- about 32%
- “No listing” (share of all labels)
- 3%
Our system says “no listing” more often. In exchange, it gives far fewer wrong picks that look like answers.
How the test was built
What was measured
For each vehicle and insurer master: did the system pick the right listing?
Exact means a listing the label marks as exact; duplicates of one vehicle count as one listing. Exact or acceptable also counts listings marked acceptable.
Where a master has no right listing, the system is right only if it returns none. 49 of the 750 labels were “unsure” and are left out.
The test set
150 real RC records (vehicle fields only), each matched against five insurer masters: 750 labels, 701 of them scored.
Held out: its model families were not used to tune our system. Built to be hard: EVs, CNG and hybrids, unknown makes, missing listings, and about 40% from model families our system had never seen.
The labels
Made by AI agents following written guidelines: two labellers, a reconciler and a reviewer. A check by people is the next step, listed below.
Labellers started from lists that included our system's suggestions, which could favour us. They also searched the full masters: for 39% of match labels, an exact listing was outside the list.
An AI spot-check of 89 tuning-set labels found none clearly wrong. On a pessimistic reading, our lead would be 8 to 10 points.
The comparison
BM25 search and fuzzy matching, each with filters (category, brand, fuel, cc and a no-match threshold), plus a two-step matcher: the insurer's model name first, then BM25 for the variant.
An AI agent built them in a few hours on 14 October 2026, tuned them on our tuning sets only, and scored them once on this test with the same scorer as our system.
What to keep in mind
About the test
- Labels. Made by AI agents; a check by people comes next.
- Hard cases. Whether our lead grows or shrinks on everyday traffic has not been measured.
- Small sample. Each score is uncertain by about ±5.5 points. The gap is clear of zero; the order among the tuned standard matchers is noise.
- Tuning. On the two sets we tuned on, our system scores about 90%. Tuning inflates that, so we quote only the held-out test.
About the comparison
- Quick baselines. A strong in-house team would likely close part of the gap, by an amount we have not measured. On raw insurer files, the baselines would probably do slightly worse.
- Not tested. Real in-house systems, LLM-based matchers and any vendor's product. Published vendor figures don't say what they measure, so we don't compare with them.
- Scope. Five insurer masters; private cars and two-wheelers only.
Next steps for the benchmark
Each of these makes the comparison stronger. The first one you can run with us now, free.
- A test on your own vehicles: the free 300-vehicle comparison.
- People with insurer-mapping experience checking the test labels.
- A comparison with an LLM-based matcher on the same test set.
- A run with fewer input fields (maker, model, class and category only).
- A head-to-head with a vendor's product on the same RC records.
We can walk you through the method and the scripts on a call.