How to Scrape a Web Page Into Structured Data (and When an API Beats It)

Web scraping is a program that requests a web page, reads its HTML and copies chosen fields into rows. This guide builds one scraper, shows what breaks, and shows the point where an API is the cheaper route.

TL;DR

• Every scraper runs four steps: request, parse, extract, store. The first working version takes about 20 minutes.
• The scraper in this guide spends 0 LeadOcean records. The API comparison at the end spends up to 25 of your 1,000 free records.
• Scrape public pages that have no API. For people and company data, call an API: the scraper will be the part that breaks.

Prerequisites

• Python 3.10 or newer with requests and beautifulsoup4 (pip install requests beautifulsoup4).
• A public page you are allowed to fetch. Check its robots.txt and terms first.
• For Step 5 only: a LeadOcean key from app.leadocean.io, exported as $LEADOCEAN_API_KEY.

Step 1: Request the page

The request is the easy part. Send a GET, identify yourself, and check the status code before you parse anything.

import requests

URL = "https://acme.com/team"
headers = {"User-Agent": "acme-research-bot/1.0 (ops@acme.com)"}

resp = requests.get(URL, headers=headers, timeout=15)
resp.raise_for_status()   # stop on 403, 404, 429
html = resp.text
print(resp.status_code, len(html))

A 403 or a challenge page means the site does not want automated reads. Stop there. Working around a bot check is where scraping turns into a legal and reliability problem.

Step 2: Parse the HTML into a tree

Parsing turns a string of markup into a tree you can query by tag and class.

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")
cards = soup.select("div.team-member")
print(len(cards), "cards found")

This works on static pages, where the data is in the first response. If cards is empty but the page shows people in your browser, the content loads with JavaScript. You need a headless browser such as Playwright. It is slower and more likely to trip anti-bot checks. Clay's guide lists the same three methods: HTML parsing, headless browsers and official APIs (clay.com, October 2026).

Step 3: Extract the fields

Extraction is where scrapers rot. A selector like div.team-member h3 is a bet that the site never renames a class.

rows = []
for card in cards:
    name = card.select_one("h3")
    title = card.select_one("p.role")
    link = card.select_one("a[href*='linkedin.com']")
    rows.append({
        "name": name.get_text(strip=True) if name else None,
        "title": title.get_text(strip=True) if title else None,
        "linkedin_url": link["href"] if link else None,
    })

Guard every field with a None check. One redesign and every selector returns nothing, silently.

Step 4: Store the rows

Write to a file or table, with the source URL and fetch time on every row.

import csv, datetime

stamp = datetime.datetime.now(datetime.timezone.utc).isoformat()
with open("team.csv", "w", newline="") as f:
    w = csv.DictWriter(f, fieldnames=["name", "title", "linkedin_url", "source", "fetched_at"])
    w.writeheader()
    for r in rows:
        w.writerow({**r, "source": URL, "fetched_at": stamp})

Step 5: Compare with one API call

For contact data, the scraper above gets you a name, a title and sometimes a profile link. No email, no phone, no verification. LeadOcean's people search returns those fields from one request, filtered, with no HTML to parse.

curl -s -X POST https://api.leadocean.io/v1/people/search \
  -H "x-api-key: $LEADOCEAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "title": ["Data Engineer"],
    "country": ["US"],
    "employeeRange": ["51-200"],
    "limit": 25
  }'

Size the list first. Sizing is free: send count=true as a query parameter with limit=1, or use the free leadocean_count_leads MCP tool. On 21 October 2026 the same filters counted 2,039 people across 1,372 companies. Those people hold a verified, catch_all_valid or catch_all email. Without the email condition they matched 7,156. The 2,039 is the mailable set by email status; it does not mean every address was verified.

To page through results, pass the cursor from the previous response. One search pages to 10,000 rows. A single search call spends one record per person returned, so the 25 above cost 25 of the 1,000 free records.

What you get

The scraper writes a CSV like this (placeholder values):

name,title,linkedin_url,source,fetched_at
Jane Doe,Data Engineer,https://www.linkedin.com/in/janedoe,https://acme.com/team,2026-10-21T09:00:00+00:00

The free count call returned this on 21 October 2026 (trimmed):

{
  "total": 2039,
  "totalIsExact": true,
  "totalAll": 7156,
  "distinctCompanies": 1372,
  "credits": 0
}

A search record has the shape documented in the API reference (response shape from the API reference, placeholder values, inner fields trimmed):

{
  "data": [
    {
      "profile_data": { "...": "name, headline, location" },
      "contact_data": { "...": "present per the reveal rules in the reference" },
      "resume_data": { "...": "past roles, education, when held" },
      "meta": { "...": "record metadata" }
    }
  ],
  "meta": { "nextCursor": "..." }
}

Read api.leadocean.io/openapi.json for the exact inner fields before you map columns.

Where scrapers fail

Scrapers fail in four predictable ways, and each one costs engineering time you do not see on day one.

• Layout drift. A redesign renames a class and every selector returns None. You find out when the CSV goes empty.
• JavaScript rendering. The data is not in the first response. A headless browser fixes it and multiplies runtime and memory.
• Blocking. Rate limits, bot checks and IP bans end a run halfway. Retrying harder makes it worse.
• No verification. A scraped address was valid when the page was published. Nothing tells you whether it still is.

Scraping still wins in one case: a public page with no API and no licensed source. Think of a niche directory or a price list. Everything else is cheaper to buy as data.

What to build, by goal

Bulk CSV export is not on the Free plan. See /pricing for the plan that includes it.

Troubleshooting

FAQ