How to Turn HTML Into Markdown for an LLM (Python Scraper Cheat-sheet)

Clean Markdown comes from deleting before converting. Remove the non-content tags with BeautifulSoup, run markdownify on the rest, and switch to Playwright only for pages that render in JavaScript.

LeadMagic's guide (September 2026) compares markdownify, html2text and Playwright. This page is the paste-ready version. It adds the step after the scrape: once you have a list of company sites, you need the people behind them.

TL;DR

• Fetch with requests, strip noise with BeautifulSoup, convert with markdownify. Playwright replaces requests for JavaScript pages. About 15 minutes.
• The scraper spends 0 LeadOcean records. Sizing the people at a domain is free. Paging 25 of them spends 25 of your 1,000 free records.
• LeadOcean is the lookup step after the scrape. It does not scrape pages for you.

Prerequisites

• Python 3.10 or newer.
• A LeadOcean key from app.leadocean.io. Free has no card. Export it as $LEADOCEAN_API_KEY.
• curl for the API steps.

pip install requests beautifulsoup4 markdownify
export LEADOCEAN_API_KEY="paste-your-key-here"

Step 1: Fetch, strip and convert

Strip first, convert second. Markdown from a full page is mostly menus, so delete the non-content tags before markdownify sees them.

import requests
from bs4 import BeautifulSoup
from markdownify import markdownify as md

NOISE = ["script", "style", "nav", "footer", "header", "noscript", "iframe", "svg"]

def html_to_markdown(html: str) -> str:
    soup = BeautifulSoup(html, "html.parser")
    for tag in soup(NOISE):
        tag.decompose()
    body = soup.body or soup
    text = md(str(body), heading_style="ATX", bullets="-")
    lines = [line.rstrip() for line in text.splitlines()]
    return "\n".join(line for line in lines if line.strip())

def fetch_markdown(url: str) -> str:
    r = requests.get(url, headers={"User-Agent": "Mozilla/5.0"}, timeout=15)
    r.raise_for_status()
    return html_to_markdown(r.text)

if __name__ == "__main__":
    print(fetch_markdown("https://example.com"))

Three tweaks matter more than any library choice:

• Scope to the main content. Many sites wrap the article in <main> or <article>. Use soup.find("main") or soup.find("article") or soup.body instead of the whole body. Menus and sidebars vanish.
• Collapse blank lines. The html_to_markdown function above already drops them. Blank runs cost tokens and carry nothing.
• Cap the length. Slice the Markdown to a fixed number of characters before it goes into a prompt, and cut at a heading boundary when you can.

Keep links if the model needs source URLs. Pass strip=["img"] to markdownify to drop images and keep anchors. Only scrape pages whose terms and robots.txt allow it.

Step 2: Render JavaScript pages with Playwright

Install it once with pip install playwright, then playwright install chromium.

Use Playwright when requests returns an empty shell. It loads the page in a real browser, waits for the network to settle, and hands the rendered HTML to the same function.

from playwright.sync_api import sync_playwright

def render_markdown(url: str) -> str:
    with sync_playwright() as p:
        browser = p.chromium.launch()
        page = browser.new_page()
        page.goto(url, wait_until="networkidle", timeout=30000)
        html = page.content()
        browser.close()
    return html_to_markdown(html)

Try fetch_markdown first. Playwright is slower and heavier, so use it only for pages that need it.

Step 3: Be polite and cache what you fetch

Scraping the same page twice wastes your time and their servers. Cache each result on disk by URL, sleep between requests, and stop on a 403 or a bot check. Do not work around a block.

import hashlib, pathlib, time

CACHE = pathlib.Path("md_cache")
CACHE.mkdir(exist_ok=True)

def cached_markdown(url: str, delay: float = 1.0) -> str:
    path = CACHE / (hashlib.sha1(url.encode()).hexdigest() + ".md")
    if path.exists():
        return path.read_text()
    markdown = fetch_markdown(url)
    path.write_text(markdown)
    time.sleep(delay)
    return markdown

Check each site's robots.txt and terms before you crawl it. A cache also gives you a stable input when you rerun the later steps.

Step 4: Pull the company domains out of the Markdown

A scraped directory, event page or "customers" page is a list of links. Parse them and keep the bare domains. This is the handoff from scraping to data.

import re
from urllib.parse import urlparse

LINK = re.compile(r"\]\((https?://[^)\s]+)\)")
SKIP = {"linkedin.com", "twitter.com", "x.com", "facebook.com", "github.com"}

def domains_from(markdown: str) -> list[str]:
    seen = []
    for url in LINK.findall(markdown):
        host = urlparse(url).netloc.lower().removeprefix("www.")
        if host and host not in SKIP and host not in seen:
            seen.append(host)
    return seen

Run domains_from(fetch_markdown("https://acme.com/customers")) and you get a list such as ["acme.com", "example.com"].

Step 5: Size the buyers at one domain for free

Send count=true as a query parameter with limit 1. It costs 0 records and returns the number of matching people. meta.total is capped at 100,000.

curl -s -X POST "https://api.leadocean.io/v1/people/search?count=true" \
  -H "x-api-key: $LEADOCEAN_API_KEY" \
  -H "content-type: application/json" \
  -d '{
    "domain": ["acme.com"],
    "jobLevel": ["VP", "Director"],
    "limit": 1
  }' | jq '.meta | {total, countOnly}'

jobLevel values are exact strings: C-Team, VP, Director, Manager, Staff, Other. Loop over your domains from Step 4 and drop the ones that return a total of 0 before you spend anything. Keep under 5 requests a second on Free.

Step 6: Page the people at a domain

GET /v1/companies/{domain}/people lists the people LeadOcean holds at a company. It returns thin records and bills one record per person returned. A 404 means nobody matched and costs nothing.

import os, requests

def people_at(domain: str, limit: int = 25) -> list[dict]:
    r = requests.get(
        f"https://api.leadocean.io/v1/companies/{domain}/people",
        headers={"x-api-key": os.environ["LEADOCEAN_API_KEY"]},
        params={"limit": limit},
        timeout=30,
    )
    if r.status_code == 404:
        return []
    r.raise_for_status()
    return r.json()["data"]

Contact points are not in this response. Enrich the people you want with POST /v1/people/enrich and a person_id. Add reveal_email: true to get the address. The roster stops at 10,000 rows, which almost no company reaches.

Step 7: Hand the Markdown to a model, with the people attached

Now the Markdown and the roster live side by side. A typical prompt gives the model one scraped page and asks for structured output, then joins the result to the people you pulled.

def build_prompt(page_markdown: str, people: list[dict]) -> str:
    names = [p["profile_data"]["profile_full_name"] for p in people]
    return (
        "Summarize what this company sells in two sentences, then say which "
        "of these roles is the best first contact.\n\n"
        f"PEOPLE: {', '.join(names)}\n\nPAGE:\n{page_markdown[:8000]}"
    )

The slice keeps the prompt bounded. Send the prompt to whichever model you use. Treat anything you scraped as untrusted text: a page can contain instructions aimed at your model, so never let scraped content trigger actions without a check.

If you would rather skip the code, the hosted MCP server at https://api.leadocean.io/mcp exposes 13 tools to an MCP client. leadocean_count_leads and leadocean_list_filters are free there too.

What you get

The sizing in Step 5 works the same way for any filter set. On 2026-10-21 the free leadocean_count_leads tool returned this count. The filters were jobLevel VP, jobFunction Sales & Business Development and country US.

{"total":129494,"totalIsExact":true,"totalAll":350517,"distinctCompanies":58121,"credits":0}

That is a mailable count: the tool defaults to verified, catch_all_valid and catch_all addresses. totalAll is the same filters with no email condition. The call cost 0 records.

A roster row has this shape. Response shape from the API reference (placeholder values):

{
  "success": true,
  "data": [
    {
      "profile_data": {
        "profile_full_name": "Jane Doe",
        "profile_headline": "VP Sales at Acme"
      },
      "meta": { "source_ids": { "person_id": "1234567890" } }
    }
  ],
  "meta": { "count": 1, "credits": 1, "nextCursor": "CURSOR", "companyIds": ["COMPANY_ID"] }
}

Troubleshooting

FAQ