Skip to main content
Blog

Job postings for hundreds of companies without scraping

Scraping careers pages works for five domains and breaks at five hundred. How a domain-filtered job database compares, with live numbers and limits.

November 3, 2026•Updated: November 4, 2026

Yes. Instead of scraping each company's careers page, ATS and job board listings yourself, you can query a job database that already collects those sources, merges the copies of each posting and lets you filter by company domain. You send the whole domain list in one request and get back structured postings (title, description, location, dates, salary when published) for every company on it. Scraping it yourself is reasonable for five domains you care about deeply. At five hundred, most of the work goes into keeping hundreds of scrapers, sources and duplicates in line.

Scrape a handful of domains yourself, query a job database for hundreds

If your list is a handful of companies and you need every posting the minute it appears on their own site, a scraper you control is a fair choice. If your list runs to hundreds or thousands of domains, or you need the data refreshed every week, a job database you query by domain costs less engineering time, covers more sources per company and gives you one schema to load.

A job database trades control for coverage and less upkeep

Scraping careers pages, ATSs and boards yourselfQuerying a job database by domain
CostEngineering time to build and repair scrapers, plus proxies and compute that grow with the listPaid per posting returned; a domain with no postings costs nothing
FreshnessAs fresh as your schedule and your scrapers' uptimeSet by the provider's crawl; ours finds 73% of new postings the same day they are posted and 91% by the end of the next day
CoverageThe sources you chose to build for, per companyCareer sites, ATSs and job boards for every company in the list, merged
EffortOne parser per ATS or site layout, repaired when layouts change; your own deduplicationOne request with the list; paging through results
LimitsBlocking, CAPTCHAs and rate limits; jobs you never knew to look forOnly companies whose domain the provider has on record; roles posted only on a small company's own site can arrive late or not at all

Why scraping works for five domains and breaks at five hundred

A company's postings rarely live in one place. The same role is created in an ATS, published on the careers page and syndicated to LinkedIn, Indeed and other boards. We took the five domains our own domain-list guide uses as its example and counted their open jobs posted in the last 30 days: 849 postings, as of 3 November 2026. Of those, 604 carry a LinkedIn or Indeed URL and 213 a Greenhouse, Lever or Ashby URL (a posting can carry both, since we match the posting URL and the URL where we found it). If you had scraped only the boards, or only the ATSs, you would have missed part of the list. One of the five samples we read shows the chain: a software engineering role found through an Indeed listing whose final URL is the company's own careers page.

With five domains you can write a scraper per site, check the output by eye and fix what breaks. Each new domain adds a site layout or an ATS you may not have a parser for yet, more requests to get blocked, and more copies of the same posting to merge. Across all companies, 466,768 direct employers with a domain on record published at least one job that is still open in the last 30 days, 4,291,441 postings in all (as of 3 November 2026). That is about nine open postings per company on average. Your list may be busier or quieter, but every one of those companies is a source you would otherwise visit on a schedule.

Deduplication is the part that grows fastest. Every source you add brings copies of postings you already have, and reposts of the same role on later dates. We keep the first time we see a posting and treat the same title from the same company within 30 days as a repost, which the sources and coverage page explains in full. A homemade pipeline has to write and keep that logic itself.

One request carries the whole domain list

The Jobs API takes the whole list in company_domain_or. This request returns open postings from direct employers posted in the last 30 days, for every domain in the list:

curl --request POST \
  --url "https://api.theirstack.com/v1/jobs/search" \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer <YOUR_API_KEY>" \
  -d '{
    "company_domain_or": ["stripe.com", "notion.so", "linear.app", "vercel.com", "figma.com"],
    "posted_at_max_age_days": 30,
    "is_closed": false,
    "company_type": "direct_employer",
    "limit": 500,
    "offset": 0
  }'

Each request returns up to 500 postings, so you page with offset until a page comes back short. A posting costs 1 API credit, and a domain with no postings costs nothing. For the five domains above, the full pull is 849 postings, so 849 credits. To size a list before you pull it, send the same filters with include_total_results: true and limit: 1; counting results costs 1 credit when anything matches and nothing when the list has no postings.

For the next refresh, save the latest discovered_at you stored and send it as discovered_at_gte, keeping the same domain list and a posted_at_* filter. For our five sample domains, 325 of the 849 postings were discovered on or after 27 October 2026, so a weekly refresh would have paid for those 325 and skipped the rest. The periodic fetching guide covers the pattern and the job_id_not alternative. If you'd rather have new postings pushed to you, a webhook on the same search sends each job.new event with the same job schema the API returns, at 1 API credit per event. Webhook events aren't deduplicated against postings you already pulled through the API, so pick one channel for each list. Large one-time loads can also come as Parquet or CSV datasets.

The full walkthrough, with the app's no-code version, pagination in Python and extra filters, is in how to scrape job data for a list of company domains. Agents can run the same search through the search_jobs tool of our MCP server:

Using the TheirStack MCP, find the open jobs posted in the last 30 days by direct employers at stripe.com, notion.so, linear.app, vercel.com and figma.com, and tell me how many each company has.

Scraping wins when a domain isn't on record or a role lives only on the careers page

A domain filter only finds companies we have a domain for. Of the 6,727,433 open postings from direct employers in the last 30 days, 4,291,441 (64%) belong to companies with a domain on record, as of 3 November 2026. For the domains that come back empty, try company_linkedin_url_or or company_name_case_insensitive_or, and expect some misses.

A role that exists only on a small company's own site and is syndicated nowhere is the one most likely to reach us late or not at all, as our freshness page says. If your list is a few dozen companies whose careers pages are the only place they hire, a scraper pointed at those pages will see some roles first.

You also choose every field and every schedule when the scraper is yours. If you need something a careers page shows and a job database doesn't store, or you need to check a page every few minutes, owning the scraper is the way to get it.

Run the domain search on your own list

Website domain: stripe.com, notion.so, linear.app, vercel.com, figma.com
Date posted: last 30 days
Job Closed: open jobs only
Company type: Direct employer

Open this search in the app with the five sample domains, then replace them with your own list under the Website domain filter.

TheirStack: integrations with AI assistants, automations and CRMs

TheirStack works with the AI assistants, automation tools and CRMs you already use: ask Claude or ChatGPT through the MCP server, send new matches to Clay, n8n, Make or Zapier by webhook, and create leads in Salesforce or HubSpot.

Get the job postings for your whole domain list

Paste your company domains and see every open posting we found for them, from careers pages, ATSs and job boards, in one search.

Start free: 200+ free credits.