How frequently we scrape jobs and update our jobs and company datasets
We have scrapers that run on different frequencies to ensure we capture the most up-to-date job listings:
By following this schedule, we ensure that after one day, we have captured most of the jobs posted the previous day.
In the following GIF, you can see the date and time we discovered jobs posted on a specific date (April 24th).
As you see, while we capture the majority of new job postings within the first and second day, there is a long tail of job discoveries that extends over weeks after that.
This means that the number of jobs posted today or yesterday may appear lower initially than its final value, which will increase as more jobs are discovered.
For customers in the Startup Plan or above, if they're interested in a specific job segment and want to access new jobs as soon as possible, we can increase the frequency of our scrapers for those jobs. Contact us if this is your case.
We update our company dataset every day, including new technologies we found in recent jobs, the total number of jobs found for them and all the other company information we have.
This means that for jobs found in the current day, if the company didn't exist in our database before, we wouldn't have data until the day after, so company filters wouldn't work on those jobs. However, this happens in a minimal percentage of all jobs and you'd still see these jobs in job searches and their company data would appear empty until the day after.