Nodriver in Python: Extract, Paginate, and Export JSON
!Scrapeless cover with browser pages flowing into a structured document.
A browser scraper needs to know when the records exist, how to identify the next page, and what to do with missing fields. Changing the browser library does not remove those decisions.
Nodriver is an asynchronous Python library that controls a Chromium-based browser through the Chrome DevTools Protocol. It does not require Selenium's ChromeDriver process. We use it here for a bounded collection job on a public practice site: render quote cards, follow pagination, and export records with their source pages.
What Nodriver Does
Nodriver can open pages, select elements, interact with them, and inspect the rendered document. Its asynchronous interface lets a coroutine wait for browser operations without turning every step into a fixed delay.
Direct CDP control describes the connection to the browser. It does not prove that a target will accept an automated session. A site's response can also depend on the network, account state, browser environment, and the action being performed. Use a public practice site to establish that your extraction logic works before investigating a target-specific difference.
The <a href="ultrafunkamsterdam.github.io" rel="nofollow">official Nodriver documentation</a> covers its API, while the <a href="ultrafunkamsterdam/nodriver" rel="nofollow">project repository</a> contains implementation and license details. Check dependency and license requirements before including it in a distributed application.
Install and Check Your Runtime
This example uses Nodriver with BeautifulSoup for the parsing stage. Install the packages in an isolated environment:
python -m pip install nodriver==0.50.6 beautifulsoup4==4.15.0
You also need a compatible local Chrome or Chromium installation. Nodriver normally discovers the executable; an application can supply browser_executable_path when it needs an explicit path. Keep that path in deployment configuration rather than assuming every machine uses the same operating system layout.
Use a separate automation profile. Reusing your personal browser profile can mix private sessions, extensions, and state with the collection job. It also makes a supposedly reproducible example depend on whatever happened in your everyday browser.
Launch, Wait, and Extract Record Cards
Save the following as collect_quotes.py. The page bound is intentional: the exercise reads two pages, then closes the browser.
import asyncio
import json
from pathlib import Path
from urllib.parse import urlsplit
import nodriver as uc
from bs4 import BeautifulSoup
START = "https://quotes.toscrape.com/js/"
async def main():
browser = await uc.start(headless=True)
records, visited = [], set()
try:
tab = await browser.get(START)
for _ in range(2):
await tab.select("div.quote", timeout=15)
current_url = await tab.evaluate("window.location.href")
if current_url in visited:
raise ValueError("Pagination returned to an earlier page")
if urlsplit(current_url).netloc != urlsplit(START).netloc:
raise ValueError("Unexpected collection host")
visited.add(current_url)
soup = BeautifulSoup(await tab.get_content(), "html.parser")
cards = soup.select("div.quote")
if not cards:
raise ValueError("Expected rendered quote cards")
for card in cards:
text = card.select_one("span.text")
author = card.select_one("small.author")
if text is None or author is None:
raise ValueError(f"Incomplete card at {current_url}")
records.append({
"text": text.get_text(" ", strip=True),
"author": author.get_text(" ", strip=True),
"source_url": current_url,
})
if soup.select_one("li.next a") is None:
break
if len(visited) < 2:
next_link = await tab.select("li.next a", timeout=5)
await next_link.click()
await tab.wait_for("div.quote", timeout=15)
Path("nodriver-quotes.json").write_text(
json.dumps(records, ensure_ascii=False, indent=2), encoding="utf-8"
)
print(f"Saved {len(records)} records from {len(visited)} pages")
finally:
browser.stop()
await asyncio.sleep(0.5)
if __name__ == "__main__":
asyncio.run(main())
tab.select() waits for a selector rather than immediately assuming that the page's first document contains records. The scraper then takes one HTML snapshot and parses fields relative to each card. That arrangement makes it easy to save the acquired HTML for diagnosis without changing the field extraction rules.
The short pause after stop() lets scheduled browser cleanup progress before the event loop exits. It is not a content-loading strategy. Page extraction waits on a record selector, which is the meaningful readiness condition for this exercise.
Make Pagination a State Transition
The script locates the next link in the rendered document and clicks the actual element. After navigation, it waits for quote cards and reads the current URL again. That gives the loop a page identity rather than relying on an incremented counter alone.
For a site that updates records without changing the URL, the readiness check must compare another stable property: a page index, a result identifier, or a content change you observed in the application. Do not copy the URL-based duplicate check into that kind of interface and assume it will work.
Avoid a broad text search for “Next” when the page has several controls with that label. A selector scoped to the pagination region gives the action a clearer target. If the site has a documented structured source for its records, inspect that source before building a longer click sequence.
Treat the Output as a Contract
Every exported record has text, author, and source_url. Required fields are checked before a record is appended, and Unicode is preserved when the file is written. Downstream code can therefore distinguish a valid empty optional field from a broken extraction rule.
A production job may need additional fields such as observation time and a source record ID. Add them deliberately. Browser titles and page URLs are useful provenance, but they do not necessarily identify a unique record across category pages or repeat visits.
Keep partial results out of the final export if the collection fails midway and the consumer expects a complete snapshot. An alternative is a separate partial-results file with an explicit status. The consumer should never have to infer completeness from the number of rows.
Decide Who Should Operate the Browser
Local Nodriver gives your application control over the browser process. That also means your deployment owns the executable, profile storage, dependencies, and resource limits. Start with a bounded session and observe its actual memory and process lifetime before increasing concurrency.
Scrapeless Agent Browser is an option when the application should connect to managed browser infrastructure. Its documented Playwright path is a separate client workflow. Do not assume a Nodriver script can connect to it by replacing a URL: a supported CDP client and a tested library connection are different claims.
If the application only needs the final page content, Web Unlocker may fit without an interactive session. Preserve the parser and change the acquisition layer when that meets the task.
What to Check Before Expanding the Job
Run the bounded example first. Inspect the exported authors and source pages, confirm that the second page differs from the first, and verify that the browser process exits. Then test a deliberately missing selector so the failure message is understandable.
Expand collection scope only after those checks work. A browser that opens successfully has passed the launch stage; record validation and complete output remain separate parts of the job.
FAQ
Does Nodriver Require ChromeDriver?
Nodriver controls the browser through CDP and does not use Selenium's ChromeDriver process for this workflow.
Can a Headless Success Guarantee Access to Another Site?
No. It confirms the example's acquisition and extraction path on that environment. A different target requires its own authorized test.
Can Nodriver Replace the Parser?
It can inspect elements directly. Here we use a rendered HTML snapshot and BeautifulSoup to keep browser interaction separate from field parsing.
Legal note: automate only sites and actions you are authorized to access. Respect applicable terms, data rights, and the site's collection boundaries.
Nodriver in Python: Extract, Paginate, and Export JSON
!Scrapeless cover with browser pages flowing into a structured document.
A browser scraper needs to know when the records exist, how to identify the next page, and what to do with missing fields. Changing the browser library does not remove those decisions.
Nodriver is an asynchronous Python library that controls a Chromium-based browser through the Chrome DevTools Protocol. It does not require Selenium's ChromeDriver process. We use it here for a bounded collection job on a public practice site: render quote cards, follow pagination, and export records with their source pages.
What Nodriver Does
Nodriver can open pages, select elements, interact with them, and inspect the rendered document. Its asynchronous interface lets a coroutine wait for browser operations without turning every step into a fixed delay.
Direct CDP control describes the connection to the browser. It does not prove that a target will accept an automated session. A site's response can also depend on the network, account state, browser environment, and the action being performed. Use a public practice site to establish that your extraction logic works before investigating a target-specific difference.
The <a href="ultrafunkamsterdam.github.io" rel="nofollow">official Nodriver documentation</a> covers its API, while the <a href="ultrafunkamsterdam/nodriver" rel="nofollow">project repository</a> contains implementation and license details. Check dependency and license requirements before including it in a distributed application.
Install and Check Your Runtime
This example uses Nodriver with BeautifulSoup for the parsing stage. Install the packages in an isolated environment:
python -m pip install nodriver==0.50.6 beautifulsoup4==4.15.0
You also need a compatible local Chrome or Chromium installation. Nodriver normally discovers the executable; an application can supply browser_executable_path when it needs an explicit path. Keep that path in deployment configuration rather than assuming every machine uses the same operating system layout.
Use a separate automation profile. Reusing your personal browser profile can mix private sessions, extensions, and state with the collection job. It also makes a supposedly reproducible example depend on whatever happened in your everyday browser.
Launch, Wait, and Extract Record Cards
Save the following as collect_quotes.py. The page bound is intentional: the exercise reads two pages, then closes the browser.
import asyncio
import json
from pathlib import Path
from urllib.parse import urlsplit
import nodriver as uc
from bs4 import BeautifulSoup
START = "https://quotes.toscrape.com/js/"
async def main():
browser = await uc.start(headless=True)
records, visited = [], set()
try:
tab = await browser.get(START)
for _ in range(2):
await tab.select("div.quote", timeout=15)
current_url = await tab.evaluate("window.location.href")
if current_url in visited:
raise ValueError("Pagination returned to an earlier page")
if urlsplit(current_url).netloc != urlsplit(START).netloc:
raise ValueError("Unexpected collection host")
visited.add(current_url)
soup = BeautifulSoup(await tab.get_content(), "html.parser")
cards = soup.select("div.quote")
if not cards:
raise ValueError("Expected rendered quote cards")
for card in cards:
text = card.select_one("span.text")
author = card.select_one("small.author")
if text is None or author is None:
raise ValueError(f"Incomplete card at {current_url}")
records.append({
"text": text.get_text(" ", strip=True),
"author": author.get_text(" ", strip=True),
"source_url": current_url,
})
if soup.select_one("li.next a") is None:
break
if len(visited) < 2:
next_link = await tab.select("li.next a", timeout=5)
await next_link.click()
await tab.wait_for("div.quote", timeout=15)
Path("nodriver-quotes.json").write_text(
json.dumps(records, ensure_ascii=False, indent=2), encoding="utf-8"
)
print(f"Saved {len(records)} records from {len(visited)} pages")
finally:
browser.stop()
await asyncio.sleep(0.5)
if __name__ == "__main__":
asyncio.run(main())
tab.select() waits for a selector rather than immediately assuming that the page's first document contains records. The scraper then takes one HTML snapshot and parses fields relative to each card. That arrangement makes it easy to save the acquired HTML for diagnosis without changing the field extraction rules.
The short pause after stop() lets scheduled browser cleanup progress before the event loop exits. It is not a content-loading strategy. Page extraction waits on a record selector, which is the meaningful readiness condition for this exercise.
Make Pagination a State Transition
The script locates the next link in the rendered document and clicks the actual element. After navigation, it waits for quote cards and reads the current URL again. That gives the loop a page identity rather than relying on an incremented counter alone.
For a site that updates records without changing the URL, the readiness check must compare another stable property: a page index, a result identifier, or a content change you observed in the application. Do not copy the URL-based duplicate check into that kind of interface and assume it will work.
Avoid a broad text search for “Next” when the page has several controls with that label. A selector scoped to the pagination region gives the action a clearer target. If the site has a documented structured source for its records, inspect that source before building a longer click sequence.
Treat the Output as a Contract
Every exported record has text, author, and source_url. Required fields are checked before a record is appended, and Unicode is preserved when the file is written. Downstream code can therefore distinguish a valid empty optional field from a broken extraction rule.
A production job may need additional fields such as observation time and a source record ID. Add them deliberately. Browser titles and page URLs are useful provenance, but they do not necessarily identify a unique record across category pages or repeat visits.
Keep partial results out of the final export if the collection fails midway and the consumer expects a complete snapshot. An alternative is a separate partial-results file with an explicit status. The consumer should never have to infer completeness from the number of rows.
Decide Who Should Operate the Browser
Local Nodriver gives your application control over the browser process. That also means your deployment owns the executable, profile storage, dependencies, and resource limits. Start with a bounded session and observe its actual memory and process lifetime before increasing concurrency.
Scrapeless Agent Browser is an option when the application should connect to managed browser infrastructure. Its documented Playwright path is a separate client workflow. Do not assume a Nodriver script can connect to it by replacing a URL: a supported CDP client and a tested library connection are different claims.
If the application only needs the final page content, Web Unlocker may fit without an interactive session. Preserve the parser and change the acquisition layer when that meets the task.
What to Check Before Expanding the Job
Run the bounded example first. Inspect the exported authors and source pages, confirm that the second page differs from the first, and verify that the browser process exits. Then test a deliberately missing selector so the failure message is understandable.
Expand collection scope only after those checks work. A browser that opens successfully has passed the launch stage; record validation and complete output remain separate parts of the job.
FAQ
Does Nodriver Require ChromeDriver?
Nodriver controls the browser through CDP and does not use Selenium's ChromeDriver process for this workflow.
Can a Headless Success Guarantee Access to Another Site?
No. It confirms the example's acquisition and extraction path on that environment. A different target requires its own authorized test.
Can Nodriver Replace the Parser?
It can inspect elements directly. Here we use a rendered HTML snapshot and BeautifulSoup to keep browser interaction separate from field parsing.
Legal note: automate only sites and actions you are authorized to access. Respect applicable terms, data rights, and the site's collection boundaries.
How to Solve Cloudflare Turnstile in Selenium (Python)
Cross-posted from blog.peak.fo. I work on Peak, a Cloudflare Turnstile solving API, so the code below calls it. The automation-fingerprint problem it solves is the same no matter which solver you use.
---
You point Selenium at a page, the login form loads, and sitting on top of it is a Cloudflare Turnstile widget that either spins forever or flips to an error the moment your driver touches the tab. Your script worked yesterday against a staging box with no challenge. Now it's dead in the water. The widget isn't broken, and neither is your code. Selenium carries a fingerprint that Turnstile is built to catch, and a stock WebDriver browser is close to the worst-case profile you can hand it.
Why Selenium specifically trips Turnstile
Turnstile doesn't make you click a bus-in-a-grid puzzle. It runs a batch of browser checks in the background, scores how human the client looks, and issues a token if the score passes. Two things about Selenium push that score the wrong way.
First, the automation fingerprint. A browser driven by WebDriver advertises itself. navigator.webdriver returns true, the Chrome DevTools Protocol leaves traces, the default window has no real profile or history behind it, and the Chrome-for-Testing binaries that Selenium Manager pulls down have their own tells. Turnstile's challenge script reads all of that. None of it is illegal or hidden, it's a clean signal that a program is driving the page.
Second, and this is the part people miss: Turnstile wants to hand your browser a token, and it won't issue one to a client it has already flagged. So you get a widget that renders, maybe even shows a spinner, and then silently refuses to produce a cf-turnstile-response value. There's nothing to click and nothing to retry. The score decided the outcome before you got a chance.
The options, compared
There are three realistic paths, and they're not equal.
undetected-chromedriver
undetected-chromedriver patches the driver to strip the obvious automation flags, so navigator.webdriver reads false and a lot of the CDP noise goes quiet. It helps, mostly with the full-page interstitial, the "Checking your browser" screen Cloudflare throws before the site loads. On a decent residential IP, undetected-chromedriver will often get you past that interstitial without extra work.
Where it falls down is the embedded widget, the .cf-turnstile box sitting inside a login or signup form. Clearing the interstitial and getting the widget to mint a token are different problems. The widget scores you again, in context, and a patched driver on a datacenter IP still looks like a patched driver. undetected-chromedriver raises your pass rate; it doesn't settle the outcome, and that rate drops fast once you run unattended, at volume, or from a flagged IP range.
Fighting the fingerprint harder
You can keep layering: real residential proxies, persistent profiles, human-like mouse movement, stealth patches on top of stealth patches. This can work, but it's a maintenance treadmill. Cloudflare ships a change, your pass rate craters, and you're back reverse-engineering the challenge on a Tuesday night. Fine as a hobby, rough as a dependency.
Solving the token with an API
The approach that stays stable: stop asking the flagged browser to produce the token, and request it from a solving service instead. The service solves against the page's sitekey, hands you a valid token, and you inject that token into the form yourself. Cloudflare's server reads cf-turnstile-response the same way no matter how it got populated. A token solved by the API and a token minted by a real click are indistinguishable at the point of validation.
The rest of this is the Selenium walkthrough for that third path.
A full working Selenium + Python walkthrough
You need two things off the page before you can solve: the page URL and the widget's sitekey. The sitekey lives in the data-sitekey attribute on the .cf-turnstile element.
Step 1: read the sitekey from the live page
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
URL = "https://target.com/login"
driver = webdriver.Chrome()
driver.get(URL)
# Wait for the widget to render, then read its sitekey
widget = WebDriverWait(driver, 15).until(
EC.presence_of_element_located((By.CSS_SELECTOR, ".cf-turnstile"))
)
sitekey = widget.get_attribute("data-sitekey")
print("sitekey:", sitekey)
If .cf-turnstile isn't present, the site may be using the explicit-render JavaScript API instead of the auto-render HTML attribute. In that case the sitekey is passed to turnstile.render() in the page's own script, so grep the page source for sitekey rather than relying on the element attribute.
Step 2: POST the sitekey to Peak
Send the URL and sitekey to the solve endpoint. For most cases the proxyless task type is all you need.
import requests
def solve_turnstile(url, sitekey):
resp = requests.post(
"https://api.peak.fo/solve",
headers={"X-API-Key": "pk_your_api_key"},
json={
"task_type": "turnstiletaskproxyless",
"url": url,
"sitekey": sitekey,
},
timeout=30,
).json()
if not resp.get("success"):
raise RuntimeError(f"solve failed: {resp.get('error')}")
return resp["data"]["token"]
token = solve_turnstile(URL, sitekey) # ~1s
The response comes back shaped like this:
{
"success": true,
"data": { "token": "0.Xy7Nm2..." },
"cost": 0.0009
}
You pay per successful solve. A failed solve costs nothing, so you're not charged for the misses.
Step 3: inject the token and fire the callback
Now drop the token into the cf-turnstile-response textarea that Turnstile injects into the form. Most pages read this field directly on submit.
driver.execute_script("""
let el = document.querySelector('[name="cf-turnstile-response"]');
if (!el) {
el = document.createElement('textarea');
el.name = 'cf-turnstile-response';
el.style.display = 'none';
(document.forms[0] || document.body).appendChild(el);
}
el.value = arguments[0];
""", token)
Setting the field is enough for a plain form POST. But a lot of sites keep the submit button disabled until Turnstile fires its success callback, the function named in the widget's data-callback attribute. If the button stays greyed out after you set the field, read that attribute and call the function directly:
callback = widget.get_attribute("data-callback")
if callback:
driver.execute_script(f"window['{callback}'](arguments[0]);", token)
Step 4: submit
driver.find_element(By.NAME, "email").send_keys("you@example.com")
driver.find_element(By.NAME, "password").send_keys("...")
driver.find_element(By.CSS_SELECTOR, 'button[type="submit"]').click()
The form posts with a token Cloudflare already validated. No widget to wait on, no spinner to babysit.
Handling the invisible variant
Turnstile has a managed mode that renders no visible checkbox. There's still a .cf-turnstile element in the DOM, it's zero-size, so your get_attribute("data-sitekey") call works exactly the same. The widget is still waiting on a token it won't issue to a flagged browser, and the fix is identical: solve against the sitekey, inject the response field, fire the callback if one exists.
The one thing that changes with invisible Turnstile is the callback matters more. Visible widgets often gate on the field; invisible ones almost always gate on the JavaScript callback, because there's no checkbox for a user to tick. So if the invisible variant isn't submitting, check the callback path first.
When to pass your own proxy
The proxyless task works when the target scores the token on its own merits. Some sites tie the token to the IP that requested it, and if your Selenium session submits from a different IP than the one Peak solved from, validation fails. That's when you switch to the proxied task type and hand over the same proxy your browser uses:
json={
"task_type": "turnstiletask",
"url": url,
"sitekey": sitekey,
"proxy": "http://user:pass@ip:port",
}
Rule of thumb: start proxyless. If tokens solve fine but the target rejects the submit, move to the proxied task and match the IP. Point Selenium at the same proxy with --proxy-server so the solve and the submit come from one address.
Common mistakes
- Solving too early. Turnstile tokens are single-use and expire in roughly 300 seconds. Request the token right before you submit, not at the top of a long script.
- Setting an input instead of a textarea. Turnstile uses a textarea named cf-turnstile-response. If you create an input and the page's own validation checks the node type, it can reject it. Match what Turnstile itself injects.
- Ignoring the callback. If the submit button won't enable, the site is gating on data-callback, not the field. Call the function.
- Mismatched IP. Solve through the same proxy you submit from when the target binds the token to the originating address.
- Expecting undetected-chromedriver to do the whole job. It helps with the interstitial. It does not reliably mint widget tokens under automation.
FAQ
Can Selenium solve Cloudflare Turnstile on its own?
Not reliably. A stock WebDriver browser carries automation signals that drop the Turnstile score below the pass line, so the widget often refuses to issue a token at all. undetected-chromedriver improves the odds, mainly on the full-page interstitial, but the embedded widget still scores you again and the pass rate falls under automation. Solving the token through an API and injecting it is the dependable route.
Does undetected-chromedriver bypass Turnstile?
It helps with the "Checking your browser" interstitial and can clear it on a good IP. It does not reliably make the embedded .cf-turnstile widget produce a token when the browser is being driven, especially from datacenter IPs or at volume. Use it to raise your pass rate, not as a guarantee.
What's the difference between the proxyless and proxied task types?
turnstiletaskproxyless solves without a proxy and works when the target scores the token independently of the requesting IP. turnstiletask takes a proxy field and is for sites that bind the token to the IP that solved it, so the solve and the submit come from the same address.
Why does the submit button stay disabled after I set the response field?
The site is gating on Turnstile's success callback rather than the field value. Read the widget's data-callback attribute and call that function with the token via execute_script.
What does this cost?
Peak charges $0.90 per 1,000 successful Turnstile solves, dropping to $0.35 per 1,000 at volume. Failed solves are free, and new accounts get 1,000 free solves with no card.
If you're on Node or driving a browser from JavaScript, the same pattern ports over to Playwright and Puppeteer. For the framework-agnostic version of the solve flow, see bypassing Cloudflare Turnstile with Python.
Start free at peak.fo. 1,000 free solves, no card, pay only for solves that land.
Mentions
Nodriver in Python: Extract, Paginate, and Export JSON !Scrapeless cover with browser pages flowing into a structured document. A browser scraper needs to know when the records exist, how to identify the next page, and what to do with missing fields. Changing the browser library does not remove those decisions. Nodriver is an asynchronous Python library that controls a Chromium-based browser through the Chrome DevTools Protocol. It does not require Selenium's ChromeDriver process. We use it here for a bounded collection job on a public practice site: render quote cards, follow pagination, and export records with their source pages. What Nodriver Does Nodriver can open pages, select elements, interact with them, and inspect the rendered document. Its asynchronous interface lets a coroutine wait for browser operations without turning every step into a fixed delay. Direct CDP control describes the connection to the browser. It does not prove that a target will accept an automated session. A site's response can also depend on the network, account state, browser environment, and the action being performed. Use a public practice site to establish that your extraction logic works before investigating a target-specific difference. The <a href="ultrafunkamsterdam.github.io" rel="nofollow">official Nodriver documentation</a> covers its API, while the <a href="ultrafunkamsterdam/nodriver" rel="nofollow">project repository</a> contains implementation and license details. Check dependency and license requirements before including it in a distributed application. Install and Check Your Runtime This example uses Nodriver with BeautifulSoup for the parsing stage. Install the packages in an isolated environment: python -m pip install nodriver==0.50.6 beautifulsoup4==4.15.0 You also need a compatible local Chrome or Chromium installation. Nodriver normally discovers the executable; an application can supply browser_executable_path when it needs an explicit path. Keep that path in deployment configuration rather than assuming every machine uses the same operating system layout. Use a separate automation profile. Reusing your personal browser profile can mix private sessions, extensions, and state with the collection job. It also makes a supposedly reproducible example depend on whatever happened in your everyday browser. Launch, Wait, and Extract Record Cards Save the following as collect_quotes.py. The page bound is intentional: the exercise reads two pages, then closes the browser. import asyncio import json from pathlib import Path from urllib.parse import urlsplit import nodriver as uc from bs4 import BeautifulSoup START = "https://quotes.toscrape.com/js/" async def main(): browser = await uc.start(headless=True) records, visited = [], set() try: tab = await browser.get(START) for _ in range(2): await tab.select("div.quote", timeout=15) current_url = await tab.evaluate("window.location.href") if current_url in visited: raise ValueError("Pagination returned to an earlier page") if urlsplit(current_url).netloc != urlsplit(START).netloc: raise ValueError("Unexpected collection host") visited.add(current_url) soup = BeautifulSoup(await tab.get_content(), "html.parser") cards = soup.select("div.quote") if not cards: raise ValueError("Expected rendered quote cards") for card in cards: text = card.select_one("span.text") author = card.select_one("small.author") if text is None or author is None: raise ValueError(f"Incomplete card at {current_url}") records.append({ "text": text.get_text(" ", strip=True), "author": author.get_text(" ", strip=True), "source_url": current_url, }) if soup.select_one("li.next a") is None: break if len(visited) < 2: next_link = await tab.select("li.next a", timeout=5) await next_link.click() await tab.wait_for("div.quote", timeout=15) Path("nodriver-quotes.json").write_text( json.dumps(records, ensure_ascii=False, indent=2), encoding="utf-8" ) print(f"Saved {len(records)} records from {len(visited)} pages") finally: browser.stop() await asyncio.sleep(0.5) if __name__ == "__main__": asyncio.run(main()) tab.select() waits for a selector rather than immediately assuming that the page's first document contains records. The scraper then takes one HTML snapshot and parses fields relative to each card. That arrangement makes it easy to save the acquired HTML for diagnosis without changing the field extraction rules. The short pause after stop() lets scheduled browser cleanup progress before the event loop exits. It is not a content-loading strategy. Page extraction waits on a record selector, which is the meaningful readiness condition for this exercise. Make Pagination a State Transition The script locates the next link in the rendered document and clicks the actual element. After navigation, it waits for quote cards and reads the current URL again. That gives the loop a page identity rather than relying on an incremented counter alone. For a site that updates records without changing the URL, the readiness check must compare another stable property: a page index, a result identifier, or a content change you observed in the application. Do not copy the URL-based duplicate check into that kind of interface and assume it will work. Avoid a broad text search for “Next” when the page has several controls with that label. A selector scoped to the pagination region gives the action a clearer target. If the site has a documented structured source for its records, inspect that source before building a longer click sequence. Treat the Output as a Contract Every exported record has text, author, and source_url. Required fields are checked before a record is appended, and Unicode is preserved when the file is written. Downstream code can therefore distinguish a valid empty optional field from a broken extraction rule. A production job may need additional fields such as observation time and a source record ID. Add them deliberately. Browser titles and page URLs are useful provenance, but they do not necessarily identify a unique record across category pages or repeat visits. Keep partial results out of the final export if the collection fails midway and the consumer expects a complete snapshot. An alternative is a separate partial-results file with an explicit status. The consumer should never have to infer completeness from the number of rows. Decide Who Should Operate the Browser Local Nodriver gives your application control over the browser process. That also means your deployment owns the executable, profile storage, dependencies, and resource limits. Start with a bounded session and observe its actual memory and process lifetime before increasing concurrency. Scrapeless Agent Browser is an option when the application should connect to managed browser infrastructure. Its documented Playwright path is a separate client workflow. Do not assume a Nodriver script can connect to it by replacing a URL: a supported CDP client and a tested library connection are different claims. If the application only needs the final page content, Web Unlocker may fit without an interactive session. Preserve the parser and change the acquisition layer when that meets the task. What to Check Before Expanding the Job Run the bounded example first. Inspect the exported authors and source pages, confirm that the second page differs from the first, and verify that the browser process exits. Then test a deliberately missing selector so the failure message is understandable. Expand collection scope only after those checks work. A browser that opens successfully has passed the launch stage; record validation and complete output remain separate parts of the job. FAQ Does Nodriver Require ChromeDriver? Nodriver controls the browser through CDP and does not use Selenium's ChromeDriver process for this workflow. Can a Headless Success Guarantee Access to Another Site? No. It confirms the example's acquisition and extraction path on that environment. A different target requires its own authorized test. Can Nodriver Replace the Parser? It can inspect elements directly. Here we use a rendered HTML snapshot and BeautifulSoup to keep browser interaction separate from field parsing. Legal note: automate only sites and actions you are authorized to access. Respect applicable terms, data rights, and the site's collection boundaries.
Nodriver in Python: Extract, Paginate, and Export JSON !Scrapeless cover with browser pages flowing into a structured document. A browser scraper needs to know when the records exist, how to identify the next page, and what to do with missing fields. Changing the browser library does not remove those decisions. Nodriver is an asynchronous Python library that controls a Chromium-based browser through the Chrome DevTools Protocol. It does not require Selenium's ChromeDriver process. We use it here for a bounded collection job on a public practice site: render quote cards, follow pagination, and export records with their source pages. What Nodriver Does Nodriver can open pages, select elements, interact with them, and inspect the rendered document. Its asynchronous interface lets a coroutine wait for browser operations without turning every step into a fixed delay. Direct CDP control describes the connection to the browser. It does not prove that a target will accept an automated session. A site's response can also depend on the network, account state, browser environment, and the action being performed. Use a public practice site to establish that your extraction logic works before investigating a target-specific difference. The <a href="ultrafunkamsterdam.github.io" rel="nofollow">official Nodriver documentation</a> covers its API, while the <a href="ultrafunkamsterdam/nodriver" rel="nofollow">project repository</a> contains implementation and license details. Check dependency and license requirements before including it in a distributed application. Install and Check Your Runtime This example uses Nodriver with BeautifulSoup for the parsing stage. Install the packages in an isolated environment: python -m pip install nodriver==0.50.6 beautifulsoup4==4.15.0 You also need a compatible local Chrome or Chromium installation. Nodriver normally discovers the executable; an application can supply browser_executable_path when it needs an explicit path. Keep that path in deployment configuration rather than assuming every machine uses the same operating system layout. Use a separate automation profile. Reusing your personal browser profile can mix private sessions, extensions, and state with the collection job. It also makes a supposedly reproducible example depend on whatever happened in your everyday browser. Launch, Wait, and Extract Record Cards Save the following as collect_quotes.py. The page bound is intentional: the exercise reads two pages, then closes the browser. import asyncio import json from pathlib import Path from urllib.parse import urlsplit import nodriver as uc from bs4 import BeautifulSoup START = "https://quotes.toscrape.com/js/" async def main(): browser = await uc.start(headless=True) records, visited = [], set() try: tab = await browser.get(START) for _ in range(2): await tab.select("div.quote", timeout=15) current_url = await tab.evaluate("window.location.href") if current_url in visited: raise ValueError("Pagination returned to an earlier page") if urlsplit(current_url).netloc != urlsplit(START).netloc: raise ValueError("Unexpected collection host") visited.add(current_url) soup = BeautifulSoup(await tab.get_content(), "html.parser") cards = soup.select("div.quote") if not cards: raise ValueError("Expected rendered quote cards") for card in cards: text = card.select_one("span.text") author = card.select_one("small.author") if text is None or author is None: raise ValueError(f"Incomplete card at {current_url}") records.append({ "text": text.get_text(" ", strip=True), "author": author.get_text(" ", strip=True), "source_url": current_url, }) if soup.select_one("li.next a") is None: break if len(visited) < 2: next_link = await tab.select("li.next a", timeout=5) await next_link.click() await tab.wait_for("div.quote", timeout=15) Path("nodriver-quotes.json").write_text( json.dumps(records, ensure_ascii=False, indent=2), encoding="utf-8" ) print(f"Saved {len(records)} records from {len(visited)} pages") finally: browser.stop() await asyncio.sleep(0.5) if __name__ == "__main__": asyncio.run(main()) tab.select() waits for a selector rather than immediately assuming that the page's first document contains records. The scraper then takes one HTML snapshot and parses fields relative to each card. That arrangement makes it easy to save the acquired HTML for diagnosis without changing the field extraction rules. The short pause after stop() lets scheduled browser cleanup progress before the event loop exits. It is not a content-loading strategy. Page extraction waits on a record selector, which is the meaningful readiness condition for this exercise. Make Pagination a State Transition The script locates the next link in the rendered document and clicks the actual element. After navigation, it waits for quote cards and reads the current URL again. That gives the loop a page identity rather than relying on an incremented counter alone. For a site that updates records without changing the URL, the readiness check must compare another stable property: a page index, a result identifier, or a content change you observed in the application. Do not copy the URL-based duplicate check into that kind of interface and assume it will work. Avoid a broad text search for “Next” when the page has several controls with that label. A selector scoped to the pagination region gives the action a clearer target. If the site has a documented structured source for its records, inspect that source before building a longer click sequence. Treat the Output as a Contract Every exported record has text, author, and source_url. Required fields are checked before a record is appended, and Unicode is preserved when the file is written. Downstream code can therefore distinguish a valid empty optional field from a broken extraction rule. A production job may need additional fields such as observation time and a source record ID. Add them deliberately. Browser titles and page URLs are useful provenance, but they do not necessarily identify a unique record across category pages or repeat visits. Keep partial results out of the final export if the collection fails midway and the consumer expects a complete snapshot. An alternative is a separate partial-results file with an explicit status. The consumer should never have to infer completeness from the number of rows. Decide Who Should Operate the Browser Local Nodriver gives your application control over the browser process. That also means your deployment owns the executable, profile storage, dependencies, and resource limits. Start with a bounded session and observe its actual memory and process lifetime before increasing concurrency. Scrapeless Agent Browser is an option when the application should connect to managed browser infrastructure. Its documented Playwright path is a separate client workflow. Do not assume a Nodriver script can connect to it by replacing a URL: a supported CDP client and a tested library connection are different claims. If the application only needs the final page content, Web Unlocker may fit without an interactive session. Preserve the parser and change the acquisition layer when that meets the task. What to Check Before Expanding the Job Run the bounded example first. Inspect the exported authors and source pages, confirm that the second page differs from the first, and verify that the browser process exits. Then test a deliberately missing selector so the failure message is understandable. Expand collection scope only after those checks work. A browser that opens successfully has passed the launch stage; record validation and complete output remain separate parts of the job. FAQ Does Nodriver Require ChromeDriver? Nodriver controls the browser through CDP and does not use Selenium's ChromeDriver process for this workflow. Can a Headless Success Guarantee Access to Another Site? No. It confirms the example's acquisition and extraction path on that environment. A different target requires its own authorized test. Can Nodriver Replace the Parser? It can inspect elements directly. Here we use a rendered HTML snapshot and BeautifulSoup to keep browser interaction separate from field parsing. Legal note: automate only sites and actions you are authorized to access. Respect applicable terms, data rights, and the site's collection boundaries.
How to Solve Cloudflare Turnstile in Selenium (Python) Cross-posted from blog.peak.fo. I work on Peak, a Cloudflare Turnstile solving API, so the code below calls it. The automation-fingerprint problem it solves is the same no matter which solver you use. --- You point Selenium at a page, the login form loads, and sitting on top of it is a Cloudflare Turnstile widget that either spins forever or flips to an error the moment your driver touches the tab. Your script worked yesterday against a staging box with no challenge. Now it's dead in the water. The widget isn't broken, and neither is your code. Selenium carries a fingerprint that Turnstile is built to catch, and a stock WebDriver browser is close to the worst-case profile you can hand it. Why Selenium specifically trips Turnstile Turnstile doesn't make you click a bus-in-a-grid puzzle. It runs a batch of browser checks in the background, scores how human the client looks, and issues a token if the score passes. Two things about Selenium push that score the wrong way. First, the automation fingerprint. A browser driven by WebDriver advertises itself. navigator.webdriver returns true, the Chrome DevTools Protocol leaves traces, the default window has no real profile or history behind it, and the Chrome-for-Testing binaries that Selenium Manager pulls down have their own tells. Turnstile's challenge script reads all of that. None of it is illegal or hidden, it's a clean signal that a program is driving the page. Second, and this is the part people miss: Turnstile wants to hand your browser a token, and it won't issue one to a client it has already flagged. So you get a widget that renders, maybe even shows a spinner, and then silently refuses to produce a cf-turnstile-response value. There's nothing to click and nothing to retry. The score decided the outcome before you got a chance. The options, compared There are three realistic paths, and they're not equal. undetected-chromedriver undetected-chromedriver patches the driver to strip the obvious automation flags, so navigator.webdriver reads false and a lot of the CDP noise goes quiet. It helps, mostly with the full-page interstitial, the "Checking your browser" screen Cloudflare throws before the site loads. On a decent residential IP, undetected-chromedriver will often get you past that interstitial without extra work. Where it falls down is the embedded widget, the .cf-turnstile box sitting inside a login or signup form. Clearing the interstitial and getting the widget to mint a token are different problems. The widget scores you again, in context, and a patched driver on a datacenter IP still looks like a patched driver. undetected-chromedriver raises your pass rate; it doesn't settle the outcome, and that rate drops fast once you run unattended, at volume, or from a flagged IP range. Fighting the fingerprint harder You can keep layering: real residential proxies, persistent profiles, human-like mouse movement, stealth patches on top of stealth patches. This can work, but it's a maintenance treadmill. Cloudflare ships a change, your pass rate craters, and you're back reverse-engineering the challenge on a Tuesday night. Fine as a hobby, rough as a dependency. Solving the token with an API The approach that stays stable: stop asking the flagged browser to produce the token, and request it from a solving service instead. The service solves against the page's sitekey, hands you a valid token, and you inject that token into the form yourself. Cloudflare's server reads cf-turnstile-response the same way no matter how it got populated. A token solved by the API and a token minted by a real click are indistinguishable at the point of validation. The rest of this is the Selenium walkthrough for that third path. A full working Selenium + Python walkthrough You need two things off the page before you can solve: the page URL and the widget's sitekey. The sitekey lives in the data-sitekey attribute on the .cf-turnstile element. Step 1: read the sitekey from the live page from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC URL = "https://target.com/login" driver = webdriver.Chrome() driver.get(URL) # Wait for the widget to render, then read its sitekey widget = WebDriverWait(driver, 15).until( EC.presence_of_element_located((By.CSS_SELECTOR, ".cf-turnstile")) ) sitekey = widget.get_attribute("data-sitekey") print("sitekey:", sitekey) If .cf-turnstile isn't present, the site may be using the explicit-render JavaScript API instead of the auto-render HTML attribute. In that case the sitekey is passed to turnstile.render() in the page's own script, so grep the page source for sitekey rather than relying on the element attribute. Step 2: POST the sitekey to Peak Send the URL and sitekey to the solve endpoint. For most cases the proxyless task type is all you need. import requests def solve_turnstile(url, sitekey): resp = requests.post( "https://api.peak.fo/solve", headers={"X-API-Key": "pk_your_api_key"}, json={ "task_type": "turnstiletaskproxyless", "url": url, "sitekey": sitekey, }, timeout=30, ).json() if not resp.get("success"): raise RuntimeError(f"solve failed: {resp.get('error')}") return resp["data"]["token"] token = solve_turnstile(URL, sitekey) # ~1s The response comes back shaped like this: { "success": true, "data": { "token": "0.Xy7Nm2..." }, "cost": 0.0009 } You pay per successful solve. A failed solve costs nothing, so you're not charged for the misses. Step 3: inject the token and fire the callback Now drop the token into the cf-turnstile-response textarea that Turnstile injects into the form. Most pages read this field directly on submit. driver.execute_script(""" let el = document.querySelector('[name="cf-turnstile-response"]'); if (!el) { el = document.createElement('textarea'); el.name = 'cf-turnstile-response'; el.style.display = 'none'; (document.forms[0] || document.body).appendChild(el); } el.value = arguments[0]; """, token) Setting the field is enough for a plain form POST. But a lot of sites keep the submit button disabled until Turnstile fires its success callback, the function named in the widget's data-callback attribute. If the button stays greyed out after you set the field, read that attribute and call the function directly: callback = widget.get_attribute("data-callback") if callback: driver.execute_script(f"window['{callback}'](arguments[0]);", token) Step 4: submit driver.find_element(By.NAME, "email").send_keys("you@example.com") driver.find_element(By.NAME, "password").send_keys("...") driver.find_element(By.CSS_SELECTOR, 'button[type="submit"]').click() The form posts with a token Cloudflare already validated. No widget to wait on, no spinner to babysit. Handling the invisible variant Turnstile has a managed mode that renders no visible checkbox. There's still a .cf-turnstile element in the DOM, it's zero-size, so your get_attribute("data-sitekey") call works exactly the same. The widget is still waiting on a token it won't issue to a flagged browser, and the fix is identical: solve against the sitekey, inject the response field, fire the callback if one exists. The one thing that changes with invisible Turnstile is the callback matters more. Visible widgets often gate on the field; invisible ones almost always gate on the JavaScript callback, because there's no checkbox for a user to tick. So if the invisible variant isn't submitting, check the callback path first. When to pass your own proxy The proxyless task works when the target scores the token on its own merits. Some sites tie the token to the IP that requested it, and if your Selenium session submits from a different IP than the one Peak solved from, validation fails. That's when you switch to the proxied task type and hand over the same proxy your browser uses: json={ "task_type": "turnstiletask", "url": url, "sitekey": sitekey, "proxy": "http://user:pass@ip:port", } Rule of thumb: start proxyless. If tokens solve fine but the target rejects the submit, move to the proxied task and match the IP. Point Selenium at the same proxy with --proxy-server so the solve and the submit come from one address. Common mistakes - Solving too early. Turnstile tokens are single-use and expire in roughly 300 seconds. Request the token right before you submit, not at the top of a long script. - Setting an input instead of a textarea. Turnstile uses a textarea named cf-turnstile-response. If you create an input and the page's own validation checks the node type, it can reject it. Match what Turnstile itself injects. - Ignoring the callback. If the submit button won't enable, the site is gating on data-callback, not the field. Call the function. - Mismatched IP. Solve through the same proxy you submit from when the target binds the token to the originating address. - Expecting undetected-chromedriver to do the whole job. It helps with the interstitial. It does not reliably mint widget tokens under automation. FAQ Can Selenium solve Cloudflare Turnstile on its own? Not reliably. A stock WebDriver browser carries automation signals that drop the Turnstile score below the pass line, so the widget often refuses to issue a token at all. undetected-chromedriver improves the odds, mainly on the full-page interstitial, but the embedded widget still scores you again and the pass rate falls under automation. Solving the token through an API and injecting it is the dependable route. Does undetected-chromedriver bypass Turnstile? It helps with the "Checking your browser" interstitial and can clear it on a good IP. It does not reliably make the embedded .cf-turnstile widget produce a token when the browser is being driven, especially from datacenter IPs or at volume. Use it to raise your pass rate, not as a guarantee. What's the difference between the proxyless and proxied task types? turnstiletaskproxyless solves without a proxy and works when the target scores the token independently of the requesting IP. turnstiletask takes a proxy field and is for sites that bind the token to the IP that solved it, so the solve and the submit come from the same address. Why does the submit button stay disabled after I set the response field? The site is gating on Turnstile's success callback rather than the field value. Read the widget's data-callback attribute and call that function with the token via execute_script. What does this cost? Peak charges $0.90 per 1,000 successful Turnstile solves, dropping to $0.35 per 1,000 at volume. Failed solves are free, and new accounts get 1,000 free solves with no card. If you're on Node or driving a browser from JavaScript, the same pattern ports over to Playwright and Puppeteer. For the framework-agnostic version of the solve flow, see bypassing Cloudflare Turnstile with Python. Start free at peak.fo. 1,000 free solves, no card, pay only for solves that land.