How to Scrape a Directory (and Fill the Gaps With an API)

A directory scrape gives you company names and websites. It rarely gives you people or emails. This guide scrapes the listings, then asks LeadOcean for the contacts.

Write the scraper only if the directory holds companies that no database lists. If you just need a segment such as small logistics firms, skip the scrape and filter a database (Step 5 shows how).

TL;DR

• Scrape the listing pages with Python, keep name and domain, dedupe on domain. About 45 minutes.
• Send each domain to LeadOcean. A company lookup costs 1 record when it finds a company, and a miss costs 0.
• Pull the decision makers with people search filtered by domain. The example run spends about 26 records.

Prerequisites

• Python 3.10 or newer with requests and beautifulsoup4 installed.
• A LeadOcean API key from app.leadocean.io/register, exported as $LEADOCEAN_API_KEY. Free gives 1,000 records for life, no card, 5 requests a second.
• A directory you may scrape. Read its terms and its robots.txt first (RFC 9309, September 2026). If either forbids automated access, stop.

Step 1: Read the page structure

Open one listing page, view source and find the repeating block. Note three things: the CSS selector for one listing, the field that holds the company website, and how the "next page" link works.

curl -s -A "my-research-bot (you@example.com)" \
  "https://directory.example.com/logistics?page=1" | head -60

Send a real contact in the user agent. Scrape one page at a time, not in parallel. If the site loads listings with JavaScript, the HTML you fetch will be empty. Look for the JSON the page calls and use that instead.

Step 2: Write the scraper

This script walks pages until one comes back empty, waits a second between requests and writes raw.csv. The selectors are placeholders. Replace them with the ones from Step 1.

import csv, time, requests
from bs4 import BeautifulSoup

HEADERS = {"User-Agent": "my-research-bot (you@example.com)"}
BASE = "https://directory.example.com/logistics"

rows, page = [], 1
while True:
    r = requests.get(BASE, params={"page": page}, headers=HEADERS, timeout=30)
    if r.status_code != 200:
        break
    cards = BeautifulSoup(r.text, "html.parser").select("div.listing")
    if not cards:
        break
    for c in cards:
        name = c.select_one("h3").get_text(strip=True)
        link = c.select_one("a.website")
        rows.append({"name": name, "url": link["href"] if link else ""})
    page += 1
    time.sleep(1)

with open("raw.csv", "w", newline="") as f:
    w = csv.DictWriter(f, fieldnames=["name", "url"])
    w.writeheader()
    w.writerows(rows)
print(len(rows), "listings")

Step 3: Clean and dedupe on domain

Directories repeat companies and often link to a profile page instead of the company site. Reduce each URL to a bare domain and keep one row per domain.

import csv
from urllib.parse import urlparse

seen, out = set(), []
for row in csv.DictReader(open("raw.csv")):
    host = urlparse(row["url"]).netloc.lower().removeprefix("www.")
    if host and host not in seen:
        seen.add(host)
        out.append({"name": row["name"], "domain": host})

with open("domains.csv", "w", newline="") as f:
    w = csv.DictWriter(f, fieldnames=["name", "domain"])
    w.writeheader()
    w.writerows(out)
print(len(out), "unique domains")

Drop rows where the domain belongs to the directory itself. Those are profile links, not company sites.

Step 4: Look up each domain

POST /v1/companies/enrich takes a domain and returns the company record. A domain we hold nothing for is a 404 and costs nothing (LeadOcean API reference, October 2026).

curl -s -X POST https://api.leadocean.io/v1/companies/enrich \
  -H "x-api-key: $LEADOCEAN_API_KEY" \
  -H "content-type: application/json" \
  -d '{"domain": "acme.com"}'

Then pull the people. Search costs one record per person returned, so filter to buyers and keep the page small.

curl -s -X POST https://api.leadocean.io/v1/people/search \
  -H "x-api-key: $LEADOCEAN_API_KEY" \
  -H "content-type: application/json" \
  -d '{"domain": ["acme.com"], "jobLevel": ["C-Team", "VP", "Director"], "limit": 25}'

To run the whole file, loop over domains.csv and stay under 5 requests a second on Free:

import csv, os, time, requests

KEY = {"x-api-key": os.environ["LEADOCEAN_API_KEY"]}
for row in csv.DictReader(open("domains.csv")):
    r = requests.post("https://api.leadocean.io/v1/companies/enrich",
                      headers=KEY, json={"domain": row["domain"]}, timeout=30)
    if r.status_code == 404:
        continue
    if r.status_code == 429:
        time.sleep(int(r.headers.get("Retry-After", 1)))
        continue
    r.raise_for_status()
    print(row["domain"], r.json()["data"].keys())
    time.sleep(0.25)

The 0.25 second pause keeps you at 4 requests a second.

Step 5: Skip the scrape when a filter will do

Many directories are just a company list by industry, size and country. Size that segment first, for free. On 2026-10-21 the leadocean_count_companies MCP tool, which wraps GET /v1/companies/count, returned this for industry Logistics and Supply Chain, HQ country US, headcount 11-50:

{
  "total": 363,
  "totalIsExact": true,
  "facets": {
    "hq_state": [
      {"value": "California", "count": 41},
      {"value": "Florida", "count": 27},
      {"value": "Texas", "count": 27}
    ]
  },
  "credits": 0
}

That count is companies only. It is not a mailable-contact count. If the number is close to what the directory offers, search the database and skip the scraper.

What you get

Your domains.csv ends up as one row per company website. The LeadOcean calls return the record below. This is the response shape from the API reference, with placeholder values.

{
  "success": true,
  "data": {
    "company_data": { "company_name": "Acme", "company_domain": "acme.com" },
    "company_detected_technologies": {},
    "company_metrics": {},
    "meta": { "fetched_at": "2026-10-01", "schema": "leadocean.company.v1" }
  },
  "meta": { "credits": 1 }
}

People search adds meta.nextCursor when more rows exist. Pass it back unchanged to get the next page. One search pages to 10,000 rows, then meta.depthCapped is true.

Troubleshooting

These are the errors the API reference describes.

FAQ