Improving company deduplication at scale
Software Engineer at TheirStack
We have rebuilt how we find duplicate companies. The new system has merged more than a million duplicate company records into the profile of the company they belong to, so jobs and technologies that used to be spread across several profiles of the same employer now sit together.
Duplicates come from the way we collect data. We gather jobs from more than 362,000 websites: applicant tracking systems like Workday, Greenhouse and Ashby, job boards like LinkedIn, Indeed and JOIN, and thousands of company careers pages. A company that hires on several of them shows up once per source, and each source describes it in its own way. One spells the name differently, another adds the legal entity, a third lists no website at all. Left as separate records, they split the company's jobs across several profiles and make it appear more than once in a company search.
We already merged the obvious duplicates as jobs came in. The ones left were records that don't look alike at first sight, and the new system is built for those. The difficulty runs both ways: merge too little and duplicates stay, merge too much and two different companies end up mixed in one profile.
A schematic view: each colored dot represents a company record. We link the records that belong to the same employer and show each group as one profile.
Signals, and how much each one counts
Every record carries signals about the company behind it, and not all of them count the same. Some are strong enough on their own to tell us two records are the same company. Others are weak and only count when several point the same way. Records linked this way form a group, and each group becomes one company profile, with its jobs and technologies together.
Signals can also contradict each other: two records may share one while everything else says they are different employers. In that case we keep them apart, because two profiles for one company is a smaller problem than one profile mixing the jobs of two.
An example: QuidelOrtho
QuidelOrtho, the diagnostics company, appears in our data as six records from LinkedIn, Indeed, Workday and JOIN. Five of them are some spelling of "QuidelOrtho", a few with a legal suffix like "Corporation" or "Sales Company, LLC".
The sixth, "Ortho-Clinical Diagnostics, Inc.", has a different name, but enough signals agree to link it to the QuidelOrtho group.
"Ortho-Clinical Diagnostics K.K." looks almost the same as that record, yet it stays a separate profile. It shares a single signal with the group, and nothing else confirms it is the same employer.
More than a million duplicate records merged
The count is of records merged, not companies: a company that had five duplicate records accounts for five of them. For companies whose records were merged, a search now brings together jobs previously split across those profiles, with fewer duplicate entries in company searches. You can look any company up in the app to see its jobs and technologies together.
We will keep improving how we deduplicate companies, so that fewer duplicates reach your searches and each company's jobs and data stay together in one profile.