Standardise first
Lowercase names, remove punctuation and expand abbreviations before comparing anything. The Splink documentation says inputs should be standardised and should include several columns that are not highly correlated. A name alone is a weak key.
Compare inside blocks, not every pair
Comparing every record in list A with every record in list B grows with the product of the two sizes. Blocking limits the comparison to records that share something, such as a postcode area or a web domain. Rules settle the easy pairs, and only the unclear ones need a model.
- Block on a shared field.
- Settle exact and near-exact matches by rule.
- Send only unclear pairs to a model.
Why a model helps and what it costs
A preprint on 2,518 ambiguous product pairs found a fuzzy rule scored F1 0.445 against 0.958 for a small language model, at about USD 1.30 per 1,000 pairs. A forum commenter makes the same point about aliases. Both are single reports on specific data, so test on yours.
Test the false links
A wrong link joins two different companies and corrupts a ledger, which costs more than a missed link. Label a set of look-alike pairs, hold some back, and count wrong links on them. Link nothing without a second matching field, and keep an unsure list for a person.
- Half true pairs, half look-alikes.
- Report false links on the look-alikes.
- Merge and delete nothing.
Sources and limits
- Splink Checked 2026-10-12.
- Splink is a Python package for probabilistic record linkage that deduplicates and links records without unique identifiers.
- No training data is required, as models can be trained with an unsupervised approach.
- It is described as capable of linking a million records on a laptop in about a minute.
- Input data should be standardised, with consistent column names and formatting, and have several columns that are not highly correlated.
- Zenodo: Cost-Aware Evaluation of LLMs for Ambiguous Product Entity Matching Checked 2026-10-12.
- A single-author preprint (8 May 2026) on 2,518 ambiguous product pairs reports a fuzzy rule baseline at F1 0.445 and a small language model at F1 0.958 for USD 1.30 per 1,000 pairs; the figures are specific to that benchmark.
- Hacker News: entity resolution discussion Checked 2026-10-12.
- A commenter argues that reliable matching needs a model with the encoded world knowledge of an LLM, after spending much effort on older fuzzy-matching and entity-recognition tools, and gives the example of telling that 'WMT' and 'wally world' are the same company.