A Bloom filter can be wrong in only one direction — which is exactly what can let a claims system skip an expensive lookup for the honest majority.
Consider the most thankless check a claims system runs: deciding, for every new claim, whether it has been seen before. At first notice of loss, an insurer wants to know whether this VIN, this claimant, this device, or this exact document has already appeared — in its own claims history, in a cross-industry fraud and duplicate database, or on a watch list. Most claims are legitimate and new, so in a design like this the check would usually come back “no.” Proving “no” can be expensive. It can mean querying very large datasets, sometimes through an external shared service that adds cost and latency. Multiply that by every incoming claim, and a pipeline can spend real time and money confirming, over and over, that nothing is wrong.
The obvious fix — keep a full in-memory index of everything ever seen — is too large to hold for hundreds of millions of records. The clever fix is to ask a smaller question. We rarely need certainty that a claim is a duplicate or is on a list — the full lookup will confirm that anyway. What we need is a cheap, reliable way to know when a claim is definitely not on the list, so we can skip the lookup entirely. That asymmetry is the whole idea behind the Bloom filter.
On a recent data migration involving millions of customer records, I needed a reliable way to catch duplicate claims. I proposed this...
(CNN) — President Donald Trump has shared another fake quote on social media, this time a fabricated comment in which “Denzel Washington” supposedly heaps praise upon him. The phony quote that Tru...