Entity Resolution: The Security Technology You Probably Haven't Heard Of

The concept is probably unfamiliar, but it underpins much of the world’s security—in telecommunications, banking and national security. It’s entity resolution (ER), the technology for consolidating disparate data about someone or something, which isn’t nearly as easy as it sounds.

At its core, ER is simple. The system brings together all the information in your data that relates to a single customer, criminal, asset or threat so that you can make informed decisions. ER helps analysts find the needle in the haystack and rapidly builds a complete picture of all relevant information. The market for ER systems is predicted to become a multi-billion-dollar industry in the next decade, with mostly US companies leading the way.

ER is also used by police for mapping organised criminal networks, immigration officers for reviewing your passport and other relevant data, casinos for detecting fraudsters and banks for conducting know-your-customer (KYC) checks to prevent sanctions evasion. Global KYC regulation requires banks to collate information on individuals so that they can conduct due diligence investigations and avoid facilitating criminal activities.

Outside the security domain, ER can enable marketing departments to better understand their customers and help finance teams to identify payments to suppliers. ER works with internal data and can be augmented by external data providers, bringing together relevant information in one view.

Entity-based resolution can also be used to identify links in obscured or deceptive behaviours known as ‘channel separation’, such as the use of a burner phone. This helps track down criminals, fraudsters, terrorists and spies who separate identifiers across different data channels or collection methods.

Basic ER systems rely on record matching, such as linking passport number 1234567 to another record with the same number. Doing this manually is time consuming and doesn’t scale up easily. If you have millions or billions of records, human investigators can’t do the job, not even with AI or advanced search functions.

In contrast, computers can compare vast numbers of records quickly, but they struggle to intuitively understand which records should be joined. ER tools have rapidly improved thanks to decades of research in data science, computer science, mathematics and behavioural science. The latest, most advanced ER systems have better intuitive understanding. For example, they can decide to merge two passport records only if they share additional attributes, such as the holder name and date of birth or the expiry date.

Consider this example. Should an analyst merge two similar accounts linked to one email address? The first is registered under Bob Smith (date of birth 07/09/1997) and the second to Robert E Smith (date of birth 09/07/1977). It could be the same person or perhaps father and son, given the ambiguity of date format (DD/MM/YYYY or MM/DD/YYYY) and the possibility of input error. By layering more information such as residential address, IP address or device type, ER can refine the resolution process and lessen the analyst’s workload.

The image below shows a simplified processing chain within an ER system, broken down into components of an end-to-end process, with each step refined over the years.

ER processes are based on decades-old mathematical models that use a process called ‘blocking’ to extract more value from data. Blocking breaks down a single record into smaller chunks that are easier for computers to process at scale and compare. This enables techniques like fuzzy matching, which calculates how similar two words are, meaning the names ‘Robert’, ‘Bob’ and ‘Rob’ would be matched together.

However, comparing every bit of data to every other attribute is still a very intensive computer process, so different ER systems take shortcuts to address this. Sometimes entities and blocks are matched using probabilistic and statistical approaches, whereas others are preprogrammed together. For example, blocks may be weighted differently depending on the uniqueness of the data field. A more unique data link, such as a credit card number which is definitely linked to just one person, would be given more weight than a less unique one, such as an email address, phone number, birth year or gender which could be common across many individuals.

The most advanced ER systems balance computation requirements, assessing the rarity of each data field and apply an even more specialised technique called entity-based resolution. This technique dynamically reassesses every single entity based on every new data record. This results in more accurate matches, less noise, and ultimately deeper analytical insights.

Despite advances in ER tools, practical implementation can still be challenging. Success depends on data structure, use cases and technology architecture. Name matching alone is hugely complex—even in English, let alone other languages. For instance, a computer doesn’t inherently know that ‘Tony’ is short for ‘Anthony’ or that in Russian ‘Sacha’ can be both ‘Alexandra’ (feminine) and ‘Aleksandr’ (masculine).

Recent developments have improved fuzzy matching across languages and cultures and in handling unstructured data. For organisations with global datasets and international remits, such as national security agencies, extracting the most knowledge and wisdom from currently available data can be the difference between life and death.

Global organisations are increasingly looking to AI to solve their resourcing challenges and help process overwhelming amounts of data. ER and AI can work hand in hand. AI can extract entities from unstructured data, while ER tools can produce meaningful insights from the data.

Organisations are experimenting with the data-science approaches to make the most of the opportunity sitting in their data. But those with ambitious AI goals should consider building on practical use cases. Having a complete understanding of their data would provide a solid foundation for an AI that is powerful and context aware. ER alone can be transformational across sectors; but especially for security analysts, security operations centre managers and investigators.

Kyle McCurdy is a technologist and former British diplomat. He has 15 years’ experience in operationalising emerging technologies, working in cyber, partnering with new companies and innovating within government.

This article appeared originally at The Strategist (ASPI).