D4Vinci

GitHub 上这个 Python 项目有点离谱！🤯 网站越不想让你爬，它越想试试。 Scrapling 把网页抓取、动态页面处理、反爬应对整合到一个框架里。 已经冲到 9.8 万 Star 左右。 以前写爬虫最头疼的是网站改版、封锁、验证码，现在有人把这些麻烦做成了工具。 D4Vinci/Scrapling

今日のGitHubトレンド D4Vinci/Scrapling Scraplingは、高速なHTTPリクエスト、ステルスブラウザ、強力なパース機能、並行クローリングを統合したPython用ウェブスクレイピングフレームワークです。 静的ページの抽出やJavaScriptのレンダリング、APIからのデータ収集に対応し、プロキシ回転や自動再試行などの機能を提供します。 Python、CLI、AIエージェントから利用可能で、単一ページからサイト全体まで効率的にデータを収集し、Markdownなどで書き出すことを目的としています。

Scrapling in Practice: A Hands-On Look at a 97k-Star Python Scraping Project A Reddit post recently put Scrapling on my radar: an open-source Python project that combines HTTP fetching, browser-based fetching, parsing, and crawler workflows. As of November 5, 2026, GitHub showed roughly 97k stars and 10.2k forks. Those numbers explain why developers are talking about the project, but they are not a quality score, a count of active users, or proof that the library fits your use case. The better questions are practical: - Can I install it and run a small example? - Which parts of a scraping workflow does it cover? - When should I use an HTTP fetcher, a browser, or a spider? - How do I know the extracted data is valid? This article answers those questions with a runnable example using Scrapling’s documented API and a public training site. You can inspect the source, license, releases, and docs yourself in the GitHub repository. The Reddit post is the discovery point for this article, not its technical source. The walkthrough below is an independent hands-on overview. Original Reddit discussion > Only collect data you are authorized to access. Prefer an official API or data export where available, follow the source’s published policies, and stop if the site asks you to stop. !Scrapling workflow from a page request to validated structured data. What is Scrapling? Scrapling is a Python web scraping framework designed to support a range of tasks, from fetching one page to running a multi-page crawl. At a high level, it combines three layers: 1. Fetching — retrieve a page over HTTP, or render it in a browser when JavaScript is required. 2. Parsing — select HTML elements and extract text or attributes using CSS, XPath, and other search methods. 3. Crawling — schedule requests, follow links, manage crawl scope, collect items, and export results. Scrapling’s spider system also documents features such as concurrency controls, sessions, retries, checkpoints, and built-in item exports. The project publishes a BSD-3-Clause license. Repository and license · Spider architecture !Scrapling’s fetching, parsing, and crawling layers. Install the fetchers Scrapling requires Python 3.10 or newer. Its base installation provides the parser engine; the fetchers extra installs dependencies used by the HTTP and browser fetchers. Create a virtual environment and install the fetcher dependencies: python -m venv .venv Activate it: # macOS or Linux source .venv/bin/activate # Windows PowerShell .venv\Scripts\Activate.ps1 Install Scrapling: python -m pip install --upgrade pip python -m pip install "scrapling[fetchers]" Browser-based fetching also needs its browser dependencies: scrapling install If you only plan to use HTTP fetching, you do not need to install a browser. Check the official installation instructions for platform-specific details. The repository showed v0.5.1 as its latest release when I checked on November 5, 2026. That release includes a breaking change for remote browser connections, so read the project’s migration notes before upgrading an existing remote-browser integration. Release history Start with one HTTP request If the page already contains the data you need in its HTML response, start with an HTTP fetcher. There is no reason to launch a browser unless the page requires it. The example below uses Quotes to Scrape, a public training site for scraping practice: from scrapling.fetchers import Fetcher url = "https://quotes.toscrape.com/" page = Fetcher.get(url) print("HTTP status:", page.status) print("Page title:", page.css("title::text").get()) quotes = [] for quote in page.css("div.quote"): text = quote.css("span.text::text").get() author = quote.css("small.author::text").get() if text and author: quotes.append({ "text": text.strip(), "author": author.strip(), }) print(f"Extracted {len(quotes)} valid quotes") print(quotes[:2]) This uses the documented Fetcher.get() and CSS-selection methods. Scrapling’s response object exposes response metadata as well as parsing methods. HTTP fetcher guide · Selector guide The field checks matter. A request can return an HTTP success status while the page content is an error message, an empty state, or a changed layout. Checking required fields helps catch those cases before invalid records enter a downstream dataset. When should you use a browser? A browser is not automatically the better fetcher. It typically uses more resources and adds more moving parts. Use an HTTP fetcher when the response already contains the data. Consider a browser fetcher when the page genuinely requires JavaScript rendering or browser interaction. Before automating a browser, check whether an official API, feed, or export provides the same data in a more stable format. | Requirement | A reasonable starting point | |---|---| | Static or server-rendered HTML | Fetcher | | Repeated HTTP requests with shared state | A fetcher session | | Data appears only after JavaScript runs | A browser fetcher | | Pagination, multiple pages, crawl statistics, and exports | A spider | Scrapling documents separate HTTP and browser fetching approaches. The project’s guidance describes HTTP fetching as the lighter option and browser fetching as appropriate for JavaScript-dependent pages and browser interactions. Choosing a fetcher Follow pagination with a spider When the job needs more than one page, a spider gives you a place to define link-following behavior and item extraction. Save the following as quotes_spider.py: from scrapling.spiders import Response, Spider class QuotesSpider(Spider): name = "quotes-demo" start_urls = ["https://quotes.toscrape.com/"] # Keep link-following within the training site. allowed_domains = {"quotes.toscrape.com"} # Check robots.txt rules and applicable crawl delays. robots_txt_obey = True async def parse(self, response: Response): for quote in response.css("div.quote"): text = quote.css("span.text::text").get() author = quote.css("small.author::text").get() if text and author: yield { "text": text.strip(), "author": author.strip(), "source_url": response.url, } next_page = response.css("li.next a::attr(href)").get() if next_page: yield response.follow(next_page, callback=self.parse) if __name__ == "__main__": result = QuotesSpider().start() print(f"Items scraped: {result.stats.items_scraped}") print(f"Requests made: {result.stats.requests_count}") print(f"Elapsed seconds: {result.stats.elapsed_seconds:.1f}") print(f"Crawl completed: {result.completed}") result.items.to_jsonl("quotes.jsonl") Run it with: python quotes_spider.py The spider extracts quotes from each page and follows the site’s “next” link. allowed_domains constrains link-following to the training site, while robots_txt_obey enables the spider’s documented robots.txt behavior. The framework can export collected items as JSON, JSONL, CSV, or XML. Spider quickstart The exact number of items can change with the site. Use the statistics printed by your own run rather than copying an expected count into a report. !Scrapling spider run summary and sample JSONL output. Validate the extracted records A scraper can finish without an exception and still produce unusable data. Add explicit checks for the fields your downstream workflow requires: def is_valid_quote(item): return ( isinstance(item.get("text"), str) and bool(item["text"].strip()) and isinstance(item.get("author"), str) and bool(item["author"].strip()) ) valid_items = [item for item in result.items if is_valid_quote(item)] invalid_count = len(result.items) - len(valid_items) print(f"Valid records: {len(valid_items)}") print(f"Invalid records: {invalid_count}") if invalid_count: raise RuntimeError("At least one item failed validation") For production work, consider checking: - required fields are present and non-empty; - dates, prices, and IDs match expected formats; - record counts have not unexpectedly fallen to zero; - duplicate records are understood; - the response contains the expected page section, not just an HTTP 200. Log enough information to debug a failure—status code, URL, elapsed time, content type, and extraction counts. Avoid logging credentials, cookies, or personal data. Adaptive parsing: helpful, but not magic Scrapling includes adaptive element tracking: it can attempt to find elements similar to a previously selected element when a page’s structure changes. The documentation describes matching based on properties such as DOM depth, tag relationships, and attributes. Adaptive scraping documentation This can help with some markup changes, but it cannot guarantee that a similar-looking element still has the same meaning. A page redesign might move a “list price” into the same kind of container previously used for “sale price.” The selector could still find an element while the data becomes semantically wrong. Treat adaptive parsing as a fallback, then validate extracted values and monitor missing-field rates and record counts. Retries and proxies need a reason Scrapling also documents blocked-response handling, retries, and proxy rotation. Those are operational tools, not proof that a proxy was the cause of a failure. Before changing proxy providers, distinguish between: - transport errors such as timeouts and connection failures; - HTTP errors such as 429 or 5xx; - an HTTP success response containing an unexpected page; - a parser that no longer matches the page structure; - missing or malformed extracted fields. A 200 OK only tells you that the request received a successful HTTP response. It does not prove that the page contains the content your parser expects. Scrapling’s spider system lets you customize blocked-response detection and retry handling. Keep retries bounded and observable, and avoid increasing request volume when the source is already rate-limiting or denying access. Proxy and blocked-request documentation How to evaluate the repository beyond its star count Stars and forks are useful discovery signals, but they do not tell you whether a package is maintained, secure, or suitable for a particular workload. Before adopting a library, inspect the evidence you can verify: 1. License: Does it allow your intended use? 2. Releases: Are versions tagged, and do release notes call out breaking changes? 3. Documentation: Can you reproduce the basic workflow from the docs? 4. Code and tests: Can you inspect implementation and test coverage for the features you plan to use? 5. Dependencies: Are the browser and HTTP dependencies appropriate for your deployment? 6. Issue and security workflow: Is there a clear process for reporting bugs or vulnerabilities? The Reddit thread that surfaced Scrapling also attracted comments questioning the project’s sudden increase in stars. Those comments are not proof of manipulation. They are a reminder to treat popularity metrics as context, not a substitute for reviewing releases, source, license, and behavior yourself. Reddit discussion When Scrapling may be a good fit Scrapling is worth evaluating if you want a single Python ecosystem for: - fetching simple pages over HTTP; - rendering pages that require JavaScript; - extracting structured data with familiar selectors; - following links with a spider workflow; - exporting items and inspecting crawl statistics. It may be more than you need for one stable page and a few fields. A focused HTTP client and parser could be easier to maintain. If you already operate a mature Scrapy project, compare the integration or migration cost with the benefit before replacing working infrastructure. Scrapling documents a Scrapy integration path. Final take Scrapling’s appeal is the way it connects fetching, parsing, and crawling behind a consistent Python workflow. The practical value is not established by a large star count alone; it comes from whether the documented APIs work for your use case, whether you can validate the output, and whether the project’s release and maintenance signals meet your needs. Start with a small permitted dataset. Run the example, inspect the output, and decide whether the framework removes enough complexity to justify adopting it.

تبي تستخرج بيانات من المواقع؟ جرّب إطار العمل مفتوح المصدر هذا. طلبات تحاكي بصمة Chrome متصفح يدعم JS ويتعامل مع Cloudflare يدعم MCP وAgent Skills للذكاء الاصطناعي → D4Vinci/Scrapling t.co

Si quieres hacer scraping y extracción de datos de webs, usa este framework open source. ✓ Peticiones que imita la huella de Chrome ✓ Navegador con JS y Captcha de Cloudflare ✓ MCP y Agent Skill para usar con IA → D4Vinci/Scrapling

Si quieres hacer scraping y extracción de datos de webs, usa este framework open source. ✓ Peticiones que imita la huella de Chrome ✓ Navegador con JS y Captcha de Cloudflare ✓ MCP y Agent Skill para usar con IA → D4Vinci/Scrapling t.co

Scrapling — 🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: and follow here for daily... Link: D4Vinci/Scrapling #OpenSource #GitHub #DevTools

Trick or Treat Scorecard: 4 Free GitHub Repos We Actually Tested 🎃 Trick or treat? This week we tested four free, open-source GitHub repos on an ordinary laptop (Ryzen 7, no NVIDIA card). Here's the scorecard, with each one's catch: • Scrapling (web scraping, BSD-3-Clause): our own site in 0.32 s. Trick: the plain install can't fetch until you add the [fetchers] extra. D4Vinci/Scrapling • VoiceStudio (voice cloning, AGPL-3.0): cloned my voice in 8.8 s, but a 7-second line took 6+ minutes on the processor. Trick: it wants a graphics card. debpalash/VoiceStudio • Paperclip (AI agent teams, MIT): refused twice, then delivered a small task in about 2.5 minutes. Trick: it runs on your own AI subscription. paperclipai/paperclip • HyperFrames (HTML to video, Apache-2.0): our web page to a finished video in under 5 minutes. Trick: you, or your AI agent, write the HTML. heygen-com/hyperframes The full tests are in our Shorts this week and in "Kill Your Zombie Subscriptions" on our channel. Join the ZigNet AI Builders Community (free): Telegram t.me · Discord discord.gg F4th3rHux + Ziggie are AI-animated (HeyGen) with an authorised synthetic voice (ElevenLabs). #Halloween #OpenSource #AITools

Scrapling: Your New Secret Weapon for Effortless Web Scraping! --- title: Scrapling: Your New Secret Weapon for Effortless Web Scraping! published: True tags: web, python, data, automation --- Quick Summary: 📝 Scrapling is a Python web scraping framework designed for efficient data extraction from single requests to large-scale crawls. It leverages modern web technologies and offers robust features for handling complex scraping tasks. Key Takeaways: 💡 - ✅ Scrapling simplifies web scraping, especially for modern, dynamic websites. - ✅ It saves developers significant time by abstracting complex setup and parsing logic. - ✅ The project effortlessly handles JavaScript-heavy pages and dynamic content. - ✅ Scrapling lowers the entry barrier, making web scraping accessible to more developers. - ✅ It's a powerful tool for efficient data extraction, analysis, and integration. Project Statistics: 📊 - ⭐ Stars: 90130 - 🍴 Forks: 9317 - ❗ Open Issues: 4 Tech Stack: 💻 - ✅ Python Hey fellow developers! Ever found yourself needing to pull data from a website, only to be met with the daunting task of setting up complex scraping tools, dealing with tricky JavaScript, or getting blocked? Web scraping can be a real headache, right? Well, I just stumbled upon a GitHub project that's about to change all that, and I'm super excited to share it with you: meet Scrapling!Scrapling, as its tagline perfectly puts it, is all about 'Effortless Web Scraping for the Modern Web.' This isn't just another scraping library; it's designed to make the entire process incredibly smooth and intuitive. Imagine being able to extract the data you need from almost any website without getting bogged down in boilerplate code or fighting against anti-scraping measures. Scrapling aims to abstract away much of that complexity, letting you focus on what truly matters: getting your data.Think of it like this: instead of manually navigating through a website's structure, identifying elements, and writing intricate parsing logic, Scrapling provides a streamlined way to define your scraping tasks. It's built to handle the nuances of modern web pages, which often rely heavily on JavaScript to load content. This means it's not easily fooled by dynamic content or pages that require interaction. It's smart enough to interact with the page as a user would, fetching the actual content you see in your browser.So, what does this mean for you? First off, massive time savings. Less time spent debugging selectors or figuring out how to render dynamic content means more time building awesome applications with the data you've collected. Secondly, it lowers the barrier to entry for web scraping. Even if you're not a seasoned scraping expert, Scrapling's intuitive approach makes it accessible. It's about empowering developers to easily gather valuable information from the web, whether for data analysis, building custom integrations, or monitoring changes on a site. Plus, the fact that it's actively maintained and has a growing community means you're not alone if you run into questions.Honestly, Scrapling feels like a breath of fresh air in the web scraping world. If you've ever shied away from scraping due to its perceived difficulty, or if you're looking for a more efficient way to handle your current scraping tasks, you absolutely have to check out Scrapling on GitHub. It's poised to become an essential tool in many developers' arsenals. Go give it a star and see for yourself! Learn More: 🔗 View the Project on GitHub --- 🌟 Stay Connected with GitHub Open Source! > 📱 Join us on Telegram > Get daily updates on the best open-source projects > GitHub Open Source > 👥 Follow us on Facebook > Connect with our community and never miss a discovery > GitHub Open Source

Kill Your Zombie Subscriptions: 6 Free GitHub Repos (ElevenLabs, Wispr Flow & More) #ad The ElevenLabs link below is an affiliate link: we may earn a commission, at no extra cost to you. It's Halloween, and some of your subscriptions are zombies. Ziggie and I went hunting on GitHub for free, open-source replacements. We installed and tested three on our laptop, read up on the other three, and show exactly where each one falls short. The 6 repos (licence · tested?): 1. VoiceStudio · AGPL-3.0 · tested: debpalash/VoiceStudio 2. FluidVoice · GPL-3.0 · Mac only for now, not tested: altic-dev/FluidVoice 3. Scrapling · BSD-3-Clause · tested: D4Vinci/Scrapling 4. FreeLLMAPI · MIT · "personal experimentation only", not tested: tashfeenahmed/freellmapi 5. Paperclip · MIT · tested: paperclipai/paperclip 6. Hindsight · MIT · not tested: vectorize-io/hindsight Our results (Oct 2026, a Ryzen 7 laptop with no NVIDIA card, so everything ran on the CPU): • VoiceStudio: the first install failed (fixed by moving its environment folder); about 12 GB of disk; cloned a voice from 14 seconds of audio in 8.8 s; a 7-second line took over 6 minutes to generate. VoiceStudio also sells a closed-source Pro build ($99/yr or $299 lifetime, voicestudio.sh/pro); everything we tested is in the free, open-source version. • Scrapling: our own site in 0.32 s; 3 pages and 30 quotes in 0.70 s; the plain install can't fetch until you add the [fetchers] extra. • Paperclip: refused twice (a settings conflict, and the project folder must be a git repo), then planned, waited for approval and delivered a small task in about 2.5 minutes on a Claude subscription. Chapters 0:00 Kill your zombie subscriptions 0:29 What is VoiceStudio? 1:12 How to clone your voice locally 2:35 FluidVoice: free dictation (Mac only) 3:15 How to scrape a website for free 3:59 FreeLLMAPI: 635 free model endpoints 4:40 How to manage a team of AI agents 5:41 Hindsight: long-term memory for AI agents 6:17 The honest bit: what free really costs 6:41 Which one first? 6:59 Links + community The voice we use: ElevenLabs (affiliate link) try.elevenlabs.io Join the ZigNet AI Builders Community (free): Telegram t.me · Discord discord.gg F4th3rHux + Ziggie are AI-animated (HeyGen) with an authorised synthetic voice (ElevenLabs). The VoiceStudio clip is a local clone of my own voice, made with my consent. Stars and licences checked 22 Oct 2026. #Halloween #OpenSource #AITools