How to backfill a job board with TheirStack

New to backfilling? Read How to backfill a job board first for an overview of strategies, sourcing methods, and quality standards.

Why TheirStack for backfilling

TheirStack's job data platform was built with job board operators in mind:

• Original job link to company website — When a job originates from a company's career page, we include the URL (final_url) so you can redirect users to the correct source. Filter for career-page-only jobs with final_url_exists.
• Standardized job descriptions — Descriptions are normalized to Markdown across all sources, so your front end renders them consistently.
• Company enrichment — Most jobs include company logo, domain, industry, headcount, revenue, type, location, and technologies used — everything you need for a company profile page. Media assets like logos are hosted on our infrastructure with stable URLs.
• Real-time data — New jobs are added every minute. Webhook delivery means your board updates in near real-time.
• Active or expired status — Stale listings destroy candidate trust and search rankings, so knowing which jobs are still open matters as much as adding new ones. TheirStack detects when a posting goes offline at its source and records the date (closed_at) — delist expired jobs in real time with the job.closed webhook event, filter searches with is_closed, or pull the closed-jobs dataset.
• 20+ filters — Use job_title_or, industry_id_or, technology_slug_or, country_code_or, and more to get only the jobs that match your niche.
• 353k data sources — Career pages, ATS platforms, and job boards worldwide. Learn more

How to get the data into your job board

TheirStack offers three ways to ingest job data: webhooks for both real-time streaming and historical backfilling, the API for on-demand pulls, and datasets for high-volume bulk loads. As a rule of thumb, use webhooks for a filtered feed below roughly 2M jobs/month. Between roughly 2M and 5M jobs/month, both Webhooks and Datasets can make sense depending on how broad your coverage is and where you want to store and filter the data. Contact us to compare the options. See How to choose the best way to access TheirStack data for the full decision guide.

Option 1: Webhooks (Recommended for filtered feeds below ~2M jobs/month)

Webhooks push jobs to your endpoint as soon as they match your criteria. No polling, no cron jobs. Set up a webhook to listen for the new.job event, apply your filters, and start receiving jobs automatically.

Webhooks handle both ongoing updates and historical backfilling:

• Ongoing updates: Once your webhook is active, new jobs matching your filters are delivered in near real-time as they are indexed.
• Historical backfill: To seed your board with existing jobs, set a wider date range in your webhook filters (e.g., posted_at_max_age_days: 90), enable "Send all matching jobs", and wait a few minutes for all matching jobs to be delivered. Once the backfill is complete, you can either adjust the date range back to a shorter window or cancel and recreate the webhook with your ongoing filters.

See How to set up a webhook for the full walkthrough.

Option 2: API polling

Use the Jobs API to fetch jobs on a schedule (e.g., every 15 minutes). The same filters and fields are available.

See Fetch jobs periodically for implementation details.

Option 3: Datasets (Consider from ~2M to 5M jobs/month)

If you need high-volume inventory — for example, launching with 50,000+ listings or maintaining a broad, multi-country board — datasets are the most cost-effective option. Instead of pulling jobs one API call at a time, you receive the full file via S3 in CSV or Parquet format.

Datasets are a good fit when:

• You want a large initial seed beyond what webhooks can deliver — Download a historical snapshot to populate your board on day one. For smaller backfills, webhooks can handle this too (see Option 1 above).
• You operate at high volume — Between roughly 2M and 5M records/month, datasets may become more cost-effective than per-record API or webhook credits. Above roughly 5M records/month, datasets are usually the better fit. Contact us with your scope and filters so we can compare both routes.
• You load data into a warehouse first — If your pipeline goes S3 → warehouse → job board (e.g., via dbt or Airflow), datasets slot in naturally.

A common pattern is to combine datasets with webhooks: use a dataset for a large initial bulk load, then subscribe to new.job webhooks to keep your board current going forward.

See Datasets for delivery formats, update frequencies, and the jobs data dictionary for the full field reference.

Further reading