How to Build an Automated B2B Lead Generation Pipeline: Scraping Google Maps, Company Websites & GitHub Contributors with Node.js & Python
Building a reliable B2B lead generation pipeline in 2026 requires more than scraping a single directory. Local business directories and Google Maps listings give you business names, categories, street addresses, ratings, and phone numbers — but they almost never expose verified decision-maker email addresses or LinkedIn / X (Twitter) profiles directly on the listing card. Similarly, when selling developer tools (DevTools), SaaS infrastructure, or technical recruiting services, your highest-converting prospects aren't sitting on generic business directories — they are actively shipping code on GitHub.
In this deep-dive engineering guide, we will build a complete, multi-source B2B Lead Generation & Contact Enrichment Pipeline using Node.js (@crawlee/cheerio) and Python (requests, pandas). We will cover:
1. Stage 1 — Local Business & Google Maps Lead Extraction: Parsing application/ld+json (LocalBusiness, Organization, ProfessionalService), microdata, phone numbers, and aggregate ratings while automatically following official company domains (/contact, /about, /locations) to uncover hidden business emails.
2. Stage 2 — Technical & DevTools Lead Extraction from GitHub: Extracting open-source contributors, repository owners, public emails, personal blogs, X/Twitter handles, and company affiliations without hitting GitHub's strict 60-requests-per-hour unauthenticated REST rate limit.
3. Stage 3 — Lead Deduplication, Email Verification & CRM Export: Filtering out false-positive emails (sentry.io, wixpress.com, asset filenames), normalizing phone numbers to E.164-compatible formats, and exporting clean CSV/Excel datasets.
If you want to skip hosting your own crawlers and call these pipelines immediately via REST API, you can run our production Actors directly on the Apify Store:
- 🌍 Google Maps & Local Business Lead, Email, Phone & Social Extractor
- 💻 GitHub Developer, Contributor & Stargazer B2B Lead Scraper
- 🏢 B2B Website Email, Phone & Social Media Contact Scraper
---
Architecture Overview: Why Multi-Hop Enrichment Wins
Most off-the-shelf scrapers fail on B2B lead generation for one of two reasons:
1. Single-page blindness: They only scrape the landing URL you pass in. If a company puts their hello@company.com or sales@company.com address on /contact-us or /impressum, a single-page scraper returns emails: [].
2. Regex noise: Naive regular expressions match CSS bundles (bundle@2x.png), error-tracking DSNs (12345@o1.ingest.sentry.io), or placeholder strings (user@example.com), polluting your cold outreach sequence and hurting domain deliverability.
Here is the multi-hop enrichment architecture we use in production:
[Input URLs: Google Maps / Directories / Company Domains / GitHub Repos]
├──> Layer 1: JSON-LD Parser (LocalBusiness, Organization, AggregateRating)
├──> Layer 2: Same-Domain Subpage Crawler (/contact, /about, /locations)
└──> Layer 3: Sanitization & Lead Scoring (Filter telemetry, export CSV/JSON)
---
Stage 1: Extracting Local Business Leads, Emails & Phones in Node.js
When crawling local business websites discovered from Google Maps or regional directories, roughly 68% of SMB websites embed rich application/ld+json blocks generated by WordPress (Yoast / RankMath), Squarespace, Shopify, or Wix. These JSON-LD blocks often contain the exact street address, postal code, direct telephone number, and aggregate customer rating in clean machine-readable JSON.
Below is the production Node.js handler using @crawlee/cheerio that combines JSON-LD schema extraction with automatic /contact subpage crawling:
import { CheerioCrawler, createCheerioRouter } from '@crawlee/cheerio';
const EMAIL_REGEX = /[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}/g;
const PHONE_REGEX = /(?:\+?\d{1,3}[\s.-]?)?\(?\d{3}\)?[\s.-]?\d{3}[\s.-]?\d{4}/g;
const BAD_EMAIL_EXT = /\.(png|jpg|jpeg|gif|svg|webp|ico|css|js|woff|woff2|ttf|map)$/i;
const BLOCKED_DOMAINS = ['example.com', 'sentry.io', 'wixpress.com', 'schema.org', 'domain.com'];
export function parseJsonLdBusinesses($) {
const businesses = [];
$('script[type="application/ld+json"]').each((_, el) => {
try {
const raw = $(el).contents().text().trim();
if (!raw) return;
const parsed = JSON.parse(raw);
const items = Array.isArray(parsed) ? parsed : (parsed['@graph'] || [parsed]);
for (const item of items) {
if (item && typeof item === 'object') {
businesses.push(item);
}
}
} catch {
// Ignore malformed JSON-LD blocks on broken templates
}
});
return businesses;
}
export const router = createCheerioRouter();
router.addDefaultHandler(async ({ enqueueLinks, request, $, body, log }) => {
const loadedUrl = request.loadedUrl || request.url;
const domain = new URL(loadedUrl).hostname.replace(/^www\./, '');
const html = typeof body === 'string' ? body : body.toString('utf-8');
// Automatically follow high-value contact & location pages on the same domain
if (!request.userData.isSubPage) {
await enqueueLinks({
strategy: 'same-hostname',
globs: ['**/contact*', '**/about*', '**/locations*', '**/impressum*'],
userData: { isSubPage: true },
});
}
const emailSet = new Set();
const phoneSet = new Set();
// 1. Extract explicit mailto: and tel: links
$('a[href^="mailto:"]').each((_, el) => {
const email = ($(el).attr('href') || '')
.replace(/^mailto:/i, '')
.split('?')[0]
.trim()
.toLowerCase();
if (email.includes('@') && !BAD_EMAIL_EXT.test(email)) {
emailSet.add(email);
}
});
$('a[href^="tel:"]').each((_, el) => {
const phone = ($(el).attr('href') || '').replace(/^tel:/i, '').trim();
if (phone.length >= 7) phoneSet.add(phone);
});
// 2. Scan raw HTML for inline emails with strict false-positive filtering
for (const match of html.matchAll(EMAIL_REGEX)) {
const candidate = match[0].toLowerCase();
const isBlocked = BLOCKED_DOMAINS.some((d) => candidate.includes(d));
if (!BAD_EMAIL_EXT.test(candidate) && !isBlocked) {
emailSet.add(candidate);
}
}
// 3. Scan visible body text for phone numbers (10 to 15 digits)
for (const match of ($('body').text() || '').matchAll(PHONE_REGEX)) {
const rawPhone = match[0].trim();
const digitCount = rawPhone.replace(/\D/g, '').length;
if (digitCount >= 10 && digitCount <= 15) {
phoneSet.add(rawPhone);
}
}
// 4. Enrich from Schema.org LocalBusiness / Organization JSON-LD
const ldItems = parseJsonLdBusinesses($);
let businessName = $('meta[property="og:site_name"]').attr('content') ||
$('h1').first().text().trim() ||
$('title').first().text().trim() ||
domain;
let address = $('[itemprop="address"], address').first().text().replace(/\s+/g, ' ').trim() || null;
let rating = null;
let reviewCount = null;
let category = null;
for (const ld of ldItems) {
if (ld.name && businessName === domain) businessName = ld.name;
if (ld.telephone) phoneSet.add(String(ld.telephone));
if (ld.email) emailSet.add(String(ld.email).replace(/^mailto:/i, '').toLowerCase());
if (ld['@type'] && !category) {
category = Array.isArray(ld['@type']) ? ld['@type'].join(', ') : String(ld['@type']);
}
if (ld.address && typeof ld.address === 'object') {
const addrParts = [
ld.address.streetAddress,
ld.address.addressLocality,
ld.address.addressRegion,
ld.address.postalCode,
ld.address.addressCountry,
].filter(Boolean);
if (addrParts.length > 0) address = addrParts.join(', ');
}
if (ld.aggregateRating && typeof ld.aggregateRating === 'object') {
rating = ld.aggregateRating.ratingValue ?? rating;
reviewCount = ld.aggregateRating.reviewCount ?? ld.aggregateRating.ratingCount ?? reviewCount;
}
}
// 5. Extract Social Media Profiles
const findSocial = (regex) => {
let matched = null;
$('a[href]').each((_, el) => {
const href = $(el).attr('href') || '';
if (!matched && regex.test(href)) matched = href;
});
return matched;
};
const record = {
businessName: String(businessName).replace(/\s+/g, ' ').trim(),
category: category || 'LocalBusiness',
domain,
sourceUrl: loadedUrl,
address,
rating,
reviewCount,
emails: Array.from(emailSet),
phones: Array.from(phoneSet).slice(0, 10),
linkedin: findSocial(/linkedin\.com\/(company|in)\//i),
facebook: findSocial(/facebook\.com\/(?!sharer)/i),
instagram: findSocial(/instagram\.com\//i),
twitter: findSocial(/(twitter|x)\.com\/(?!intent)/i),
tiktok: findSocial(/tiktok\.com\/@/i),
scrapedAt: new Date().toISOString(),
};
log.info(`[Lead Extracted] ${record.businessName} -> ${record.emails.length} emails, ${record.phones.length} phones`);
});
Calling the Deployed Google Maps & Local Business Lead Extractor via Python
Instead of maintaining Node.js infrastructure and proxy pools yourself, you can invoke the cloud Actor tungnguyen84/google-maps-lead-email-phone-extractor directly from Python:
import requests
import pandas as pd
APIFY_TOKEN = "YOUR_APIFY_API_TOKEN"
ACTOR_URL = (
"https://api.apify.com/v2/acts/"
"tungnguyen84~google-maps-lead-email-phone-extractor/"
f"run-sync-get-dataset-items?token={APIFY_TOKEN}"
)
payload = {
"startUrls": [
{"url": "https://apify.com/contact"},
{"url": "https://crawlee.dev"}
],
"enrichCompanyWebsites": True,
"maxBusinesses": 50
}
response = requests.post(ACTOR_URL, json=payload, timeout=180)
response.raise_for_status()
leads = response.json()
df = pd.DataFrame(leads)
print(df[["businessName", "domain", "emails", "phones", "linkedin"]])
df.to_csv("enriched_b2b_leads.csv", index=False)
---
Stage 2: Scraping GitHub Developers & Contributors for DevTools B2B Outreach
If you are recruiting senior engineers or selling developer-focused SaaS (CI/CD, observability, AI coding assistants, cloud databases, security scanners), scraping generic business directories won't help. You need active open-source contributors and maintainers from relevant GitHub repositories.
However, GitHub's official REST API limits unauthenticated requests to just 60 requests per hour, which breaks almost immediately when enriching a repository with 200 contributors.
By parsing GitHub's semantic HTML microformats (span.p-name, div.p-note, span.p-org, span.p-label, li[itemprop="url"]) with @crawlee/cheerio, we can extract contributor profiles, company affiliations, personal websites, X/Twitter handles, and public emails at high speed without burning API quota:
// Extracting developer lead data from a GitHub profile DOM
const username = new URL(loadedUrl).pathname.split('/').filter(Boolean)[0];
const fullName = $('span.p-name, [itemprop="name"]').first().text().trim() || username;
const bio = $('div.p-note, [data-bio-text]').first().text().trim() || '';
const company = $('span.p-org, [itemprop="worksFor"]').first().text().trim() || null;
const location = $('span.p-label, [itemprop="homeLocation"]').first().text().trim() || null;
const website = $('li[itemprop="url"] a, a.Link--primary[rel*="me"]').first().attr('href') || null;
const followers = $('a[href$="?tab=followers"] span.text-bold').first().text().trim() || null;
Running the GitHub Developer & Contributor Scraper via cURL or Python
Our live Actor GitHub Developer, Contributor & Stargazer B2B Lead Scraper accepts both GitHub repository URLs (automatically discovering and enqueuing contributor profiles) and direct developer profile URLs:
curl -X POST "https://api.apify.com/v2/acts/tungnguyen84~github-developer-contributor-lead-scraper/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"targetUrls": [
{ "url": "https://github.com/tungnguyen84" },
{ "url": "https://github.com/apify/crawlee" }
],
"maxProfiles": 25
}'
---
Stage 3: Deduplicating & Scoring Multi-Page Leads in Python
When your crawler visits both https://company.com and https://company.com/contact, it outputs one record per page. Before importing into HubSpot, Pipedrive, or Apollo, you should merge records by domain so each company occupies a single row with all discovered emails, phones, and social links combined:
from collections import defaultdict
import json
def merge_leads_by_domain(raw_records: list[dict]) -> list[dict]:
grouped = {}
for rec in raw_records:
domain = rec.get("domain") or rec.get("username") or rec.get("sourceUrl")
if not domain:
continue
if domain not in grouped:
grouped[domain] = {
"businessName": rec.get("businessName") or rec.get("fullName"),
"domain": domain,
"website": rec.get("website") or rec.get("sourceUrl"),
"address": rec.get("address") or rec.get("location"),
"rating": rec.get("rating"),
"emails": set(rec.get("emails") or []),
"phones": set(rec.get("phones") or []),
"linkedin": rec.get("linkedin"),
"twitter": rec.get("twitter"),
"facebook": rec.get("facebook"),
"instagram": rec.get("instagram"),
}
else:
entry = grouped[domain]
entry["emails"].update(rec.get("emails") or [])
entry["phones"].update(rec.get("phones") or [])
for field in ("address", "rating", "linkedin", "twitter", "facebook", "instagram"):
if not entry.get(field) and rec.get(field):
entry[field] = rec[field]
merged = []
for domain, data in grouped.items():
data["emails"] = sorted(data["emails"])
data["phones"] = sorted(data["phones"])
data["leadScore"] = (
(50 if data["emails"] else 0)
+ (25 if data["phones"] else 0)
+ (15 if data["linkedin"] else 0)
+ (10 if data["address"] else 0)
)
merged.append(data)
return sorted(merged, key=lambda x: x["leadScore"], reverse=True)
---
🌐 Try Our Live Apify Actors, RapidAPI Endpoints & Standalone Tools
Whether you prefer serverless cloud APIs or standalone offline Python + Excel VBA tools, all of our scrapers are live and ready to test:
1. Apify Store Actors (Pay-Per-Result, No Monthly Commitment)
- 🌍 Google Maps & Local Business Lead, Email, Phone & Social Extractor
- 💻 GitHub Developer, Contributor & Stargazer B2B Lead Scraper
- 🏢 B2B Website Email, Phone & Social Media Contact Scraper
- 🎬 YouTube Video Transcript, Metadata & Channel Email Scraper
- 💬 Reddit Subreddit, Search & Thread to Clean JSON & AI Markdown Scraper
- 📄 Website to Clean Markdown & Text Scraper for AI, LLM & RAG
- 🛍️ Shopify Store Products, Prices & B2B Contact Email Scraper
- 🛒 WooCommerce & WordPress Store Product, Price & Contact Scraper
- 🔍 Website SEO Audit, Meta Tags, Headings & Contact Email Scraper
2. RapidAPI Hub REST APIs (Free Tier BASIC $0.00/mo)
- B2B Website Email, Phone & Social Scraper API
- Website to Clean Markdown AI, LLM & RAG Converter API
- Shopify Store Product, Price & Contact Scraper API
- WooCommerce & WordPress Product, Price & Contact Spy API
- Website Sitemap, SEO Audit & Contact Scraper API
3. Instant Download Full Source Code & Excel VBA Suites (Payhip / Gumroad)
- 📦 Python B2B Lead Scraper & Excel VBA Suite Pro ($14.99)
- 📦 Shopify Competitor Price & Product Spy Pro ($12.99)
- 📦 Ultimate Excel VBA Automation Pack: 10 One-Click Macros ($9.99)
Need a custom web scraper, anti-bot bypass pipeline, or automated CRM integration built for your team? Message me on Fiverr (@tungnguyen84)!
The new Apify CLI is more secure, especially in sandboxes. Run apify login, approve a one-time code in Apify Console, and that’s it. Your token stays out of chat, and your shell history, so there is nothing to leak.
Full changelog: github.com/apify/apify-cli/releases/tag/v1.11.0
2 likes · 62 views
7 kostenlose Repos, die Claude unglaublich mächtig machen | GitHub | KI-Skills | Konnektoren
Claude hat mittlerweile tausende Repos – aber diese 7 Claude Skills, Tools und Konnektoren können deinen kompletten Workflow verändern.
Von echter Marktrecherche über Kundenfindung und SEO bis hin zu Videos, Werbekampagnen und einem Team aus 282 KI-Agenten: Ich teste alle sieben an einem einzigen Beispiel.
🔗 REPOS:
last30days: mvanhorn/last30days-skill
gstack: garrytan/gstack
Apify MCP Server: apify/apify-mcp-server
OpenSEO: every-app/open-seo
HyperFrames: heygen-com/hyperframes
Claude Ads: AgriciDaniel/claude-ads
agency agents: msitarzewski/agency-agents
Das erwartet dich im Video:
✅ last30days – findet heraus, worüber echte Menschen gerade sprechen
✅ gstack – prüft deine Geschäftsidee wie ein Startup-Investor
✅ Apify – findet potenzielle Kunden und relevante Unternehmen
✅ OpenSEO – analysiert Keywords, Rankings und KI-Sichtbarkeit
✅ HyperFrames – erstellt Videos direkt aus Claude
✅ Claude Ads – entwickelt und prüft Werbekampagnen
✅ agency agents – gibt dir Zugriff auf 282 spezialisierte KI-Rollen
Das Spannende daran: Ich teste die Tools nicht einzeln, sondern verbinde sie zu einem kompletten Workflow – von der ersten Idee über Recherche und Validierung bis zur fertigen Marketingkampagne.
Kapitel:
00:00 Intro – 7 kostenlose Claude Repos, Skills & Konnektoren
00:23 last30days – Marktrecherche mit echten Meinungen
01:50 gstack – Geschäftsideen kritisch prüfen
03:33 Apify – Potenzielle Kunden finden
04:51 OpenSEO – SEO, Keywords & KI-Sichtbarkeit
05:59 HyperFrames – KI-Videos erstellen
06:52 Claude Ads – Werbekampagnen planen & optimieren
08:03 agency agents – 282 KI-Spezialisten für Claude
09:26 Fazit – Welche Tools lohnen sich wirklich?
**LASS UNS VERNETZEN**
✔️ Website: shakeelazq.de
✔️ Instagram: instagram.com
✔️ TikTok: tiktok.com
✔️ LinkedIn: linkedin.com
✔️ Newsletter: 8g2tqd.subscribepage.io
**✉️ BUSINESS-ANFRAGEN**
Für Kooperationen, Projekte und geschäftliche Anfragen erreichst du mich hier: kontakt.shakeelazq@gmail.com
#Claude #ClaudeAI #GitHub #ClaudeSkills #KI
255 likes · 12 comments · 8,302 views
How do I find a company's contact email, phone and social profiles from its domain?
> Short answer: fetch the company's homepage plus its contact, about and imprint pages (respecting robots.txt), extract mailto:/tel: links, JSON-LD Organization data, obfuscated emails and social links, then validate the emails' domain for MX records. The Fairquery Website Contact & Company Enricher does all of that per domain and returns one clean row (role emails, E.164 phones, socials, company name, description, address) for $3.00 per 1,000 domains (plus a start fee of $0.05 per 1,000 runs); unreachable, blocked or invalid domains are free.
What you get per domain
- Business emails, validated with the same code as the Fairquery Email Validator: syntax, DNS MX / null MX, disposable list, free-provider and typo checks (no SMTP probing). By default only role/generic inboxes (info@, sales@, hello@) are returned.
- Phone numbers normalised to E.164, with the page they came from.
- Social profiles: company LinkedIn, X/Twitter, GitHub, Facebook, Instagram, YouTube, TikTok and so on. Personal LinkedIn profiles (/in/...) are never returned.
- Company name, description and postal address from JSON-LD, og:site_name, <address> blocks and imprint pages.
- Mail domain facts: MX, null MX, mail provider (for example Google Workspace or Microsoft 365), optional SPF/DMARC.
How it works
1. Normalises each input (https://www.Acme.com/contact, jane@acme.com, acme.com → acme.com) and removes duplicates.
2. Reads robots.txt (RFC 9309) and honours Disallow and Crawl-delay for user agent FairqueryBot.
3. Fetches the homepage, then the best-scoring same-site pages (contact, Kontakt, contacto, about, imprint/Impressum, legal notice, team) up to maxPagesPerDomain (default 5, max 15), one request at a time per site with requestDelayMs (default 1,000 ms) between them.
4. Extracts JSON-LD, mailto:/tel: links, Cloudflare-protected emails, name [at] domain [dot] com obfuscation, social links and addresses, then validates the emails.
It is plain HTTP, with no headless browser and no proxy, so sites that render contact details only with JavaScript or block datacenter IPs return fewer fields.
How to run it with curl
curl -X POST \
"https://api.apify.com/v2/acts/fairquery~website-contact-company-enricher/run-sync-get-dataset-items" \
-H "Authorization: Bearer $APIFY_TOKEN" \
-H "Content-Type: application/json" \
-d '{"domains": ["apify.com"], "maxPagesPerDomain": 4}'
Real output from that request
We ran exactly this on October 17, 2026 (row shown in full, minus the timestamp and the duplicate addresses array):
{
"type": "domain",
"input": "apify.com",
"domain": "apify.com",
"final_url": "https://apify.com/",
"company_name": "Apify Technologies s.r.o.",
"description": "Thousands of tools to automate your business. Get real-time web data, track competitors, generate leads, and integrate your apps and AI agents.",
"emails": [
{
"email": "hello@apify.com",
"source_url": "https://apify.com/contact",
"is_role_account": true,
"on_company_domain": true,
"is_free_provider": false,
"is_disposable": false,
"verdict": "valid",
"reasons": [
"ROLE_ACCOUNT"
],
"has_mx": true,
"mail_provider": "Google Workspace / Gmail"
}
],
"primary_email": "hello@apify.com",
"phones": [],
"social_links": {
"github": [
"https://github.com/apify"
],
"linkedin": [
"https://linkedin.com/company/apify"
],
"twitter": [
"https://x.com/apify"
],
"tiktok": [
"https://tiktok.com/@apifytech",
"https://tiktok.com/@apifyoffice"
]
},
"address": "Na Příkopě 959/27, 11000, Prague, CZ",
"mail_domain": {
"has_mx": true,
"null_mx": false,
"mail_provider": "Google Workspace / Gmail",
"has_spf": null,
"has_dmarc": null,
"dmarc_policy": null
},
"pages_crawled": [
"https://apify.com/",
"https://apify.com/contact",
"https://apify.com/contact-sales"
],
"pages_blocked_by_robots": 0,
"robots_txt": "ok",
"email_scope": "role"
}
The run crawled 3 pages, took 5.1 seconds and would cost $0.003 at the Free-plan rate of $3.00 per 1,000 domains, plus a start fee of $0.05 per 1,000 runs (not folded into the domain charge). The README documents about 3–8 seconds per domain with the default politeness settings and 5 domains in parallel, so 100 domains take roughly 1–3 minutes. No phone number was published on the crawled pages here, so phones is empty rather than guessed.
Python: enrich a CSV of company websites
import csv, os
from apify_client import ApifyClient
client = ApifyClient(os.environ["APIFY_TOKEN"])
domains = [row["website"] for row in csv.DictReader(open("companies.csv"))]
run = client.actor("fairquery/website-contact-company-enricher").call(
run_input={"domains": domains, "emailScope": "role", "defaultPhoneCountry": "US"}
)
for r in client.dataset(run["defaultDatasetId"]).iterate_items():
if r["type"] == "domain":
print(r["domain"], r["primary_email"], [p["e164"] for p in r["phones"]],
r["social_links"].get("linkedin"))
else:
print("skipped:", r.get("code"), r.get("message"))
Input options
| Field | Default | Meaning |
|---|---|---|
| domains | (required) | Domains, URLs or emails |
| maxPagesPerDomain | 5 | 1–15, homepage included |
| emailScope | role | business adds named addresses on the company's own domain; all returns everything found |
| validateEmails | true | MX / disposable / typo checks |
| checkSpfDmarc | false | SPF/DMARC for the company mail domain |
| defaultPhoneCountry | – | e.g. US; also parses national-format numbers in page text |
| domainConcurrency / requestDelayMs / requestTimeoutSecs | 5 / 1000 / 15 | Politeness and timeouts |
| maxDomains | 1000 | Safety cap per run |
Error rows (never charged): INPUT_EMPTY, INVALID_DOMAIN, MAX_DOMAINS_EXCEEDED, SITE_UNREACHABLE, BLOCKED_BY_ROBOTS, ENRICH_FAILED.
From an AI agent
Call Actor fairquery/website-contact-company-enricher through Apify MCP, or call it through Apify MCP. It has enrich_domains (paid, your Apify token) and a free local normalize_domains tool for cleaning a messy list first. Real output of normalize_domains for ["https://www.Apify.com/contact", "jane@stripe.com", "basecamp.com", "apify.com", "not a domain"]:
{
"domains": ["apify.com", "stripe.com", "basecamp.com"],
"invalid": ["not a domain"],
"duplicates_removed": 1
}
Pricing
$3.00 per 1,000 domains enriched ($30.00 per 10,000) on Apify Free-plan prices, pay per event, plus a start fee of $0.05 per 1,000 runs (Apify Actor-start). Only domains that return a result row are charged; unreachable, robots-blocked and invalid domains are free.
How it compares with alternatives
| Option | Good for | Trade-offs |
|---|---|---|
| Doing it by hand | A handful of companies | Slow; easy to miss imprint pages and obfuscated emails |
| Writing your own scraper | Full control | robots.txt handling, page discovery, email de-obfuscation, phone parsing and email validation to build and maintain |
| B2B contact databases (for example Hunter, Apollo, ZoomInfo) | Finding named people, job titles and personal work emails from large pre-built databases | Subscription pricing; data comes from their database rather than a live read of the company's site |
| Fairquery Contact Enricher | Live, per-domain company contact data with source URLs and validated emails, pay per domain | Only what the company publishes on its own site; role inboxes by default, no person search; JavaScript-only sites yield less |
FAQ
How can I get a company's contact email from its website domain?
Crawl the homepage and its contact, about and imprint pages, extract mailto links, JSON-LD and obfuscated emails, and validate the domain's MX records. The Fairquery Website Contact & Company Enricher does this for $3.00 per 1,000 domains and returns validated role emails with the page they were found on.
Does it find personal email addresses of employees?
No. By default it returns only role/generic inboxes such as info@, sales@ or hello@ (emailScope 'role'). 'business' adds named addresses published on the company's own domain. It never returns personal LinkedIn profiles and does not search for people.
Does it respect robots.txt?
Yes. It reads robots.txt (RFC 9309), honours Disallow and Crawl-delay for user agent FairqueryBot, makes one request at a time per site with a 1 second default delay, and reports pages_blocked_by_robots.
How fast is it?
About 3-8 seconds per domain with the default settings and 5 domains in parallel, so 100 domains take roughly 1-3 minutes. Our single-domain test run took 5.1 seconds.
What does it cost?
$3.00 per 1,000 domains enriched ($30.00 per 10,000) on Apify Free-plan prices, plus a start fee of $0.05 per 1,000 runs. Unreachable, robots-blocked and invalid domains are free.
Why are some fields empty?
It only returns what the company publishes on the crawled pages. Sites that render contacts with JavaScript only, or block datacenter IPs, return fewer fields, because it uses plain HTTP without a browser or proxy.
> Use responsibly: business contact data can still be personal data under GDPR/UK GDPR. Have a lawful basis, honour opt-outs, and follow anti-spam law (CAN-SPAM, ePrivacy/PECR) when you contact anyone. DNS checks cannot prove a mailbox exists.
Source and docs: Apify Store listing · Fairquery tools · original post
---
Originally published on the Fairquery blog.
5. apify/crawlee
Framework profesional con todo incluido: rotación de proxies, reintentos automáticos, browser fingerprinting, colas de trabajo.
Lo que cobran las empresas de scraping por miles al mes.
Aquí gratis.
4 likes · 1 repost · 159 views
A Threads Username That Doesn't Exist Returns 200 OK and 351 KB of Login Page
I run a visa agency in Bali. The other half of my week goes into Apify Actors, and one of them is deliberately boring: one Threads username in, one row out — followers, full name, bio, the links and contacts published in it, verification, the linked Instagram, and the latest post with its engagement. No batch job, no thousand posts I did not ask for. That shape exists because of how it gets called: an agent asks about one account, an n8n node fires once per row, a CRM enrichment runs per contact.
For a shape like that, latency is the product. So on 24 October 2026 I timed it, and the real accounts were fine — and the typos were not.
| Input | Requests to Threads | Bytes pulled | Run time |
|---|---|---|---|
| zuck | 1 | 1,190,211 | 3.9 s |
| natgeo | 1 | 858,228 | 4.6 s |
| a username nobody owns | 3 | 1,057,628 | 14.3 s |
| the same username again | 3 | 1,057,629 | 14.6 s |
| the same username, earlier | 3 | 936,153 | 21.2 s |
Three requests and up to twenty-one seconds to learn that an account does not exist, against one request and four seconds to return a profile that does. Nothing was billed for the misses — a username that is missing, private or renamed has always been a free error row here — so this never showed up as money. It showed up as an agent waiting.
What Threads actually serves for a username nobody owns
There is no 404. There is no redirect to an error page. There is a 200 and a document:
| | @zuck | @malinavisa-no-such-acct-9q7x |
|---|---|---|
| HTTP status | 200 | 200 |
| Bytes | 1,201,337 | 351,710 |
| RelayPrefetchedStreamCache blocks | 2 | 0 |
| "Sorry, this page isn't available" | absent | absent |
| LSD token present | yes | yes |
| <title> | Mark Zuckerberg (@zuck) • Threads, Say more | Threads • Log in |
| og:url | https://www.threads.com/@zuck | https://www.threads.com/login |
A logged-out visitor asking for an account that exists gets the profile as JSON embedded in the HTML — Meta's Relay payloads, the same ones the React app would hydrate from. A visitor asking for an account that does not exist gets the login page, rendered under the profile URL, with a perfectly valid LSD token and no payloads at all.
Two details make this harder than it sounds.
First, threads.net now 301s to threads.com (0.16 s per hop, measured), and somewhere in that migration the old sentinel went away. My parser had been looking for Sorry, this page isn't available since September; today's HTML does not contain that string for a missing account, or any other apology.
Second, you cannot see any of this with curl. Following redirects from a datacenter IP, curl -L returned 231,125 bytes for both the real and the fake account — the same stripped shell, zero Relay payloads, differing only in their tokens. The embedded profile JSON only appears when the request carries a browser TLS fingerprint; this Actor uses impit with Chrome's. So the naive existence check — "did the page have data in it?" — comes back false for a real account too if your client looks wrong, which is exactly why my code did what it did next.
The retry budget landed on the wrong branch
Here is the old loop, and it is not stupid — it is just answering a different question than the one in front of it:
for (let attempt = 1; attempt <= 3; attempt++) {
const html = await request(url, { headers: pageHeaders }, 'page');
parsed = parsePage(html, url);
if (Object.keys(parsed.payloads).length || /Sorry, this page isn't available/i.test(html)) return parsed;
// A page without Relay payloads is Threads serving a stripped shell — rotate the session and retry.
session += 1; stats.retries += 1;
await sleep(1200 * attempt);
}
A page with no payloads genuinely is, most of the time, Threads throttling an exit address: you rotate the proxy session and try again, and the second or third attempt comes back full. With the sentinel gone, a missing username enters the same branch and runs the whole budget — three fetches of 351 KB each, plus 1.2 s and 2.4 s of backoff — before anyone concludes anything. Twelve to nineteen seconds of work phase in the logs, all of it spent proving the same negative three times.
The block detector one layer down had quietly rotted too. It looked for "login_form" or <title>Log in in the first 4,000 characters of the response. The string login_form does not occur anywhere in today's HTML, and <title> sits 98,841 bytes into the document, so that slice never reached it. Blocks were still caught — by their 429 and 403 status codes — but the content-based check had been dead for a while without failing loudly. That is the quiet kind of rot: a condition that cannot fire looks identical to a condition that never needed to.
The marker that survived
Everything cheap about that document had moved, except one thing: the canonical URL is in the first kilobyte of head. Byte 2,141 for the login shell, byte 2,253 for the profile.
/** Threads serves a username that does not exist as its own login page with HTTP 200. The page's
* <title> sits ~99 KB into the HTML, but its canonical URL is in the first kilobyte and points at
* /login instead of /@username — the one crisp marker left after the threads.com migration. */
const LOGIN_WALL = /<meta property="og:url" content="https:\/\/www\.threads\.(?:net|com)\/login"/i;
export const isLoginWall = (html = '') => LOGIN_WALL.test(String(html).slice(0, 8000));
And in the page loop, the login wall gets its own branch:
if (parsed.loginWall) {
if (attempt >= 2) return parsed; // two exit addresses agreed: no such account
session += 1; stats.retries += 1;
continue; // no backoff — the answer is not going to change
}
Why two attempts instead of one, when I have a crisp marker? Because I cannot prove that a block never looks like this. A 429 and a 403 are caught above this code, but if Threads ever decides to answer a throttled address with its login page under a 200, a one-shot check would convert a temporary block into a confident "this account does not exist" — a wrong error row is worse than a slow correct one. Two independent proxy sessions have to agree, and the agreement costs one extra fetch and no sleeping.
Before and after, same username, same afternoon
| Build | Requests | Retries | Bytes | Work phase | Run time |
|---|---|---|---|---|---|
| 0.0.26, old | 3 | 3 | 1,057,628 | 11.8 s | 14.3 s |
| 0.0.26, old | 3 | 3 | 1,057,629 | 12.3 s | 14.6 s |
| 0.0.28, fixed | 2 | 1 | 705,057 | 3.4 s | 4.6 s |
| 0.0.28, fixed | 2 | 1 | 705,088 | 4.0 s | 5.1 s |
| 0.0.28, fixed | 2 | 1 | 705,115 | 5.0 s | 7.5 s |
The work phase — container boot excluded, the part that is my code talking to Threads — went from 11.8–19.1 s down to 3.1–5.0 s. Real profiles are untouched: one request, 1.19 MB, same row, same four seconds. The error row itself is byte-identical to before, which was a requirement, not a bonus: anyone matching on that string in production should not have to care that the plumbing changed.
{
"type": "error",
"username": "malinavisa-no-such-acct-9q7x",
"profileUrl": "https://www.threads.net/@malinavisa-no-such-acct-9q7x",
"error": "account not found, private, or renamed — check the username",
"blocked": false,
"scrapedAt": "2026-10-24T13:15:26.018Z"
}
A profile that does exist comes back as one flat row, trimmed here to the fields people actually use:
{
"type": "profile",
"username": "mkbhd",
"fullName": "Marques Brownlee",
"followerCount": 1710708,
"biography": "I promise I won't overdo the (text) filters.",
"bioLinks": ["https://mkbhd.com/"],
"emails": [], "phones": [],
"isVerified": true, "isPrivate": false,
"instagramUrl": "https://www.instagram.com/mkbhd/",
"latestPostAt": "2026-10-19T16:01:41.000Z",
"latestPostUrl": "https://www.threads.net/@mkbhd/post/Dero8yTm4Og",
"latestPostLikes": 4208,
"latestPostReplies": 118,
"scrapedAt": "2026-10-24T13:10:07.137Z"
}
Calling it is one request, and the free summary row at the end tells you what was charged before the billing tab does:
curl -X POST "https://api.apify.com/v2/acts/lergassy~threads-profile-api/run-sync-get-dataset-items?token=<TOKEN>" \
-H "Content-Type: application/json" \
-d '{"username":"natgeo","latestPosts":0}'
Filter type != "notice" and you have your data. The listing is Threads Profile API.
When this is the wrong tool
If you have ten thousand usernames in a spreadsheet, the per-call shape is the wrong shape — one run with a list is the same money here (profiles bill per row, no start fee) but a bulk Actor is built for that queue. If you want an account's posts rather than its profile, this returns the latest one plus up to 25 on request, and nothing deeper. And private accounts are not solvable by any amount of retrying: Threads shows a logged-out visitor the same login page it shows for a name nobody owns, which is precisely why both come back under one honest error message instead of an invented profile.
The general lesson is cheaper to learn from someone else's logs than your own: when a site has no 404, your "I am blocked" branch and your "it does not exist" branch receive the identical response, and whichever one you wrote first will absorb the other. Mine absorbed it silently, for free, in seconds nobody was billed for — and seconds are the whole value of a one-row lookup.
1 reaction · 2 comments
6 Free Viral Github Repos that actually help in Making Money
Free Skool community (all prompts + links from this video): skool.com
Learn AI properly in 3 months (Aaghaz AI program): aaghaz.ai
Most people use Claude Code without these 6 free GitHub repos, which means they're only using half of it. In this video I show you what each repo does, how to install it in one step, and how to turn them into paid client work.
The 6 repos:
1. Humanizer: blader/humanizer
2. Apify MCP: apify/apify-mcp-server
3. Google Workspace CLI: googleworkspace/cli
4. HyperFrames: heygen-com/hyperframes
5. Agent Skills: addyosmani/agent-skills
6. Playwright CLI: microsoft/playwright-cli
Comment "REPOS" and I'll send you the full guide (every link, step-by-step install, all the prompts).
Chapters
0:00 Using only half of Claude?
0:37 Who I am
0:46 GitHub repos explained (Skill, MCP, CLI)
1:45 #1 Humanizer
2:47 #2 Apify MCP
3:34 #3 Google Workspace CLI
4:20 The next 3 repos
4:36 #4 HyperFrames
5:28 #5 Agent Skills
6:31 #6 Playwright CLI
7:13 3 rules before you install
7:38 How to make money with these repos
8:06 Free guide
#ClaudeCode #GitHub #AITools
1,121 likes · 74 comments · 69,627 views
Yüzlerce GitHub Reposu Denedim ! Claude İçin En İyi 6 Repo Bu !
Claude Code için yüzlerce GitHub reposu denedim; gerçekten işe yarayan 6 tanesini, ne işe yaradıklarını ve nasıl paraya çevrileceklerini anlatıyorum.
HyperFrames ile yapay zekana video kurgulat, Apify ile Claude'u Instagram ve TikTok'a sok, ScrollCraft ile 10.000$'lık siteler yap, Humanizer ile "AI slop" görüntüsünü sil, Agent Skills ve Awesome listeleriyle Claude skill setini büyüt. Sonunda da kendi işine yarayan GitHub repolarını yapay zekana buldurmanın altın kuralını veriyorum.
🔗 Videodaki repolar:
1. HyperFrames — heygen-com/hyperframes
2. Apify MCP — apify/apify-mcp-server
3. ScrollCraft — nateherkai/scroll-craft
4. Humanizer — blader/humanizer
5. Agent Skills — addyosmani/agent-skills
6. Awesome — sindresorhus/awesome
⏱️ Bölümler
0:00 Giriş
2:19 1. HyperFrames — yapay zekan video kurgular
4:40 HyperFrames ile para kazanmak
5:40 2. Apify — Claude Instagram ve TikTok'a girsin
8:15 3. ScrollCraft — 10.000$'lık havalı siteler
11:11 4. Humanizer — "AI slop" görüntüsünü sil
12:25 5. Agent Skills — uzmanın hazır skill seti
14:00 Taktik: skill'leri kendisi kullansın
14:53 6. Awesome — işe yarayan her şeyin listesi
16:39 Videoyu neden uzatmıyorum
17:40 Altın sır 1: Paketleme
19:31 Altın sır 2: Repoları yapay zekana buldur
🎓 YAPAY ZEKA TOPLULUĞU (Skool)
Yapay zeka eğitimi bulabileceğin topluluk.
👉 skool.com
Bu videodaki hiçbir araç sponsor değildir; sadece kendi kullandıklarımı anlattım.
#claudecode #github #yapayzeka
#yapayzeka #claude #chatgpt
claude code, github, github repo, claude code github, claude, claude code kullanımı, claude skill, claude skills, hyperframes, apify, apify mcp, scrollcraft, humanizer, agent skills, awesome list, claude mcp, claude video edit, yapay zeka ile video kurgu, yapay zeka, yapay zeka araçları, ai slop, yapay zeka ile para kazanma, muhammed ogulcan
1,553 likes · 79 comments · 44,916 views
写爬虫，总被链接队列和浏览器操作折腾？🕸️
Crawlee 提供爬取队列、浏览器自动化和数据存储。
适合采集获准访问的公开数据。🧰
apify/crawlee
#WebScraping #Automation #浇朋友 #蓝朋友 #互关 #fo
3 likes · 80 views
MCP + CLI > MCP vs. CLI
That tweet got 1.6k likes in March:
{% embed x.com %}
Perplexity’s CTO reportedly told his team they’re moving from MCP to APIs and CLIs:
{% embed x.com %}
OpenClaw left MCP out on purpose:
{% embed x.com %}
Garry Tan’s version was shorter: “MCP sucks honestly.”"
{% embed x.com %}
They’re half right: the problem is real, but the diagnosis isn’t. What’s broken is how most agents use MCP, and the fix is not to throw the protocol away but to stop using it badly.
> This piece builds on my talk at AI Engineer World’s Fair “MCP doesn’t suck. Your agent does.” (San Francisco, July 2026). Watch it on YouTube.
{% embed youtube.com %}
The problem is the harness, not the protocol
Most agents treat MCP tools as prompt-time function calls. Connect a server and every tool definition lands in the context window. Call a tool and the full result lands there too. Ten servers with five tools each, and you pay thousands of tokens on every turn before the agent has done anything. Results pile up, context rots, the agent gets slower and less accurate, and data the model never needed to see is now inside the model.
None of that is in the spec. MCP already has the fixes: progressive tool discovery (Anthropic’s tool search tool, Cursor’s dynamic context discovery), code mode (Cloudflare and Anthropic both wrote it up), and sub-agents with split context. But most MCP clients don’t implement them. The protocol moves faster than the ecosystem.
Now look at what CLIs get right, mostly by accident. Agents never load a CLI’s docs all at once; they read --help when they need it. They run CLI tools as code by default, chaining them with pipes and shell scripts. And every model knows the shell by heart, because there’s 50 years of it in the training data. The catch here: a CLI is a local black box. No standard transport, no standard auth, no discovery, no way to scope or intercept what it does to a remote service.
That’s why agents have MCP connectors, not CLI connectors.
MCP for remote, CLI for local
The fix is to use each where it’s strong. MCP stays the standard remote interface: server discovery, authentication, and access control. The CLI becomes the local interface between the agent and MCP. That’s what we built in mcpc, an open-source universal CLI client for MCP:
┌──────────┐ Bash() ┌────────┐ MCP ┌────────────┐
│ AI agent │ ─────────► │ mcpc │ ────────► │ MCP server │
└──────────┘ └────────┘ └────────────┘
Sessions, OAuth, Tools,
Resources, Prompts,
Tasks, x402, ...
The agent only needs one tool, Bash(), to call mcpc, learn how it works through --help, and interact with MCP servers. And it can leverage the full MCP protocol out of the box: persistent sessions, OAuth 2.1 (including enterprise SSO), tools, resources, prompts, async tasks, skills, x402 payments, and a code mode using --json. mcpc speaks MCP 2026-07-28 and falls back to older versions. No LLM inside, fast, and minimal dependencies. As a bonus, one login and one set of live sessions can be shared between many agents on the same machine.
What it looks like in the shell
brew install apify/tap/mcpc # or: npm install -g @apify/mcpc
mcpc login mcp.apify.com # OAuth once; tokens go to the OS keychain
mcpc connect mcp.apify.com @apify # persistent session
mcpc @apify # server info, capabilities, instructions
mcpc @apify grep "crawler" # progressive tool discovery
mcpc @apify tools-get search-actors # one tool's schema, only when needed
mcpc @apify tools-call search-actors keywords:="tiktok scraper"
And code mode, where the agent writes a script instead of making a round trip through its context for every call:
mcpc @apify tools-call search-actors keywords:="tiktok scraper" limit:=1 --json \
| jq -r '.content[0].text | capture("`(?<actor>[^`]+)`").actor' \
| xargs -I {} mcpc @apify tools-call fetch-actor-details actor:="{}" --json
That’s progressive discovery and code mode on any MCP server, whether or not the agent’s own MCP client supports either.
What the evals say
We built connector-bench to measure the efficiency of mcpc against CLIs and naive MCP implementations. Each run gives an agent the same cross-server task under three harnesses: raw MCP with every tool loaded into context, the native CLIs for the same services, and mcpc. We track cumulative cost against wall-clock time.
Here’s an example comparing Apify MCP server with Apify CLI on an Actor discovery scenario:
<!-- IMAGE: upload mcpc-vs-cli-vs-raw-mcp-cost.png (in Downloads/Article) here -->
In many of our tests, mcpc finishes the job faster than native CLI, albeit sometimes with higher cost. That’s understandable, as native optimized CLIs are usually less verbose and more token-efficient than a generic CLI wrapper for MCP. But we believe there is plenty of room for optimization and the performance of mcpc will converge towards native CLIs.
So stop saying CLI beats MCP
MCP + CLI beats both. Even OpenClaw uses MCP through mcporter, an MCP-to-CLI converter, which proves the point. The problem was never the protocol. It was that agents implemented MCP poorly.
MCP has ten official SDKs for languages like TypeScript, Python, or Go, but none for the shell, the most universal programming language. mcpc is our attempt to fill that gap, so any agent with a code runtime can use all of MCP.
Try it, break it, star it:
{% embed apify/mcpc %}
1 reaction · 2 comments
Mentions
How to Build an Automated B2B Lead Generation Pipeline: Scraping Google Maps, Company Websites & GitHub Contributors with Node.js & Python Building a reliable B2B lead generation pipeline in 2026 requires more than scraping a single directory. Local business directories and Google Maps listings give you business names, categories, street addresses, ratings, and phone numbers — but they almost never expose verified decision-maker email addresses or LinkedIn / X (Twitter) profiles directly on the listing card. Similarly, when selling developer tools (DevTools), SaaS infrastructure, or technical recruiting services, your highest-converting prospects aren't sitting on generic business directories — they are actively shipping code on GitHub. In this deep-dive engineering guide, we will build a complete, multi-source B2B Lead Generation & Contact Enrichment Pipeline using Node.js (@crawlee/cheerio) and Python (requests, pandas). We will cover: 1. Stage 1 — Local Business & Google Maps Lead Extraction: Parsing application/ld+json (LocalBusiness, Organization, ProfessionalService), microdata, phone numbers, and aggregate ratings while automatically following official company domains (/contact, /about, /locations) to uncover hidden business emails. 2. Stage 2 — Technical & DevTools Lead Extraction from GitHub: Extracting open-source contributors, repository owners, public emails, personal blogs, X/Twitter handles, and company affiliations without hitting GitHub's strict 60-requests-per-hour unauthenticated REST rate limit. 3. Stage 3 — Lead Deduplication, Email Verification & CRM Export: Filtering out false-positive emails (sentry.io, wixpress.com, asset filenames), normalizing phone numbers to E.164-compatible formats, and exporting clean CSV/Excel datasets. If you want to skip hosting your own crawlers and call these pipelines immediately via REST API, you can run our production Actors directly on the Apify Store: - 🌍 Google Maps & Local Business Lead, Email, Phone & Social Extractor - 💻 GitHub Developer, Contributor & Stargazer B2B Lead Scraper - 🏢 B2B Website Email, Phone & Social Media Contact Scraper --- Architecture Overview: Why Multi-Hop Enrichment Wins Most off-the-shelf scrapers fail on B2B lead generation for one of two reasons: 1. Single-page blindness: They only scrape the landing URL you pass in. If a company puts their hello@company.com or sales@company.com address on /contact-us or /impressum, a single-page scraper returns emails: []. 2. Regex noise: Naive regular expressions match CSS bundles (bundle@2x.png), error-tracking DSNs (12345@o1.ingest.sentry.io), or placeholder strings (user@example.com), polluting your cold outreach sequence and hurting domain deliverability. Here is the multi-hop enrichment architecture we use in production: [Input URLs: Google Maps / Directories / Company Domains / GitHub Repos] ├──> Layer 1: JSON-LD Parser (LocalBusiness, Organization, AggregateRating) ├──> Layer 2: Same-Domain Subpage Crawler (/contact, /about, /locations) └──> Layer 3: Sanitization & Lead Scoring (Filter telemetry, export CSV/JSON) --- Stage 1: Extracting Local Business Leads, Emails & Phones in Node.js When crawling local business websites discovered from Google Maps or regional directories, roughly 68% of SMB websites embed rich application/ld+json blocks generated by WordPress (Yoast / RankMath), Squarespace, Shopify, or Wix. These JSON-LD blocks often contain the exact street address, postal code, direct telephone number, and aggregate customer rating in clean machine-readable JSON. Below is the production Node.js handler using @crawlee/cheerio that combines JSON-LD schema extraction with automatic /contact subpage crawling: import { CheerioCrawler, createCheerioRouter } from '@crawlee/cheerio'; const EMAIL_REGEX = /[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}/g; const PHONE_REGEX = /(?:\+?\d{1,3}[\s.-]?)?\(?\d{3}\)?[\s.-]?\d{3}[\s.-]?\d{4}/g; const BAD_EMAIL_EXT = /\.(png|jpg|jpeg|gif|svg|webp|ico|css|js|woff|woff2|ttf|map)$/i; const BLOCKED_DOMAINS = ['example.com', 'sentry.io', 'wixpress.com', 'schema.org', 'domain.com']; export function parseJsonLdBusinesses($) { const businesses = []; $('script[type="application/ld+json"]').each((_, el) => { try { const raw = $(el).contents().text().trim(); if (!raw) return; const parsed = JSON.parse(raw); const items = Array.isArray(parsed) ? parsed : (parsed['@graph'] || [parsed]); for (const item of items) { if (item && typeof item === 'object') { businesses.push(item); } } } catch { // Ignore malformed JSON-LD blocks on broken templates } }); return businesses; } export const router = createCheerioRouter(); router.addDefaultHandler(async ({ enqueueLinks, request, $, body, log }) => { const loadedUrl = request.loadedUrl || request.url; const domain = new URL(loadedUrl).hostname.replace(/^www\./, ''); const html = typeof body === 'string' ? body : body.toString('utf-8'); // Automatically follow high-value contact & location pages on the same domain if (!request.userData.isSubPage) { await enqueueLinks({ strategy: 'same-hostname', globs: ['**/contact*', '**/about*', '**/locations*', '**/impressum*'], userData: { isSubPage: true }, }); } const emailSet = new Set(); const phoneSet = new Set(); // 1. Extract explicit mailto: and tel: links $('a[href^="mailto:"]').each((_, el) => { const email = ($(el).attr('href') || '') .replace(/^mailto:/i, '') .split('?')[0] .trim() .toLowerCase(); if (email.includes('@') && !BAD_EMAIL_EXT.test(email)) { emailSet.add(email); } }); $('a[href^="tel:"]').each((_, el) => { const phone = ($(el).attr('href') || '').replace(/^tel:/i, '').trim(); if (phone.length >= 7) phoneSet.add(phone); }); // 2. Scan raw HTML for inline emails with strict false-positive filtering for (const match of html.matchAll(EMAIL_REGEX)) { const candidate = match[0].toLowerCase(); const isBlocked = BLOCKED_DOMAINS.some((d) => candidate.includes(d)); if (!BAD_EMAIL_EXT.test(candidate) && !isBlocked) { emailSet.add(candidate); } } // 3. Scan visible body text for phone numbers (10 to 15 digits) for (const match of ($('body').text() || '').matchAll(PHONE_REGEX)) { const rawPhone = match[0].trim(); const digitCount = rawPhone.replace(/\D/g, '').length; if (digitCount >= 10 && digitCount <= 15) { phoneSet.add(rawPhone); } } // 4. Enrich from Schema.org LocalBusiness / Organization JSON-LD const ldItems = parseJsonLdBusinesses($); let businessName = $('meta[property="og:site_name"]').attr('content') || $('h1').first().text().trim() || $('title').first().text().trim() || domain; let address = $('[itemprop="address"], address').first().text().replace(/\s+/g, ' ').trim() || null; let rating = null; let reviewCount = null; let category = null; for (const ld of ldItems) { if (ld.name && businessName === domain) businessName = ld.name; if (ld.telephone) phoneSet.add(String(ld.telephone)); if (ld.email) emailSet.add(String(ld.email).replace(/^mailto:/i, '').toLowerCase()); if (ld['@type'] && !category) { category = Array.isArray(ld['@type']) ? ld['@type'].join(', ') : String(ld['@type']); } if (ld.address && typeof ld.address === 'object') { const addrParts = [ ld.address.streetAddress, ld.address.addressLocality, ld.address.addressRegion, ld.address.postalCode, ld.address.addressCountry, ].filter(Boolean); if (addrParts.length > 0) address = addrParts.join(', '); } if (ld.aggregateRating && typeof ld.aggregateRating === 'object') { rating = ld.aggregateRating.ratingValue ?? rating; reviewCount = ld.aggregateRating.reviewCount ?? ld.aggregateRating.ratingCount ?? reviewCount; } } // 5. Extract Social Media Profiles const findSocial = (regex) => { let matched = null; $('a[href]').each((_, el) => { const href = $(el).attr('href') || ''; if (!matched && regex.test(href)) matched = href; }); return matched; }; const record = { businessName: String(businessName).replace(/\s+/g, ' ').trim(), category: category || 'LocalBusiness', domain, sourceUrl: loadedUrl, address, rating, reviewCount, emails: Array.from(emailSet), phones: Array.from(phoneSet).slice(0, 10), linkedin: findSocial(/linkedin\.com\/(company|in)\//i), facebook: findSocial(/facebook\.com\/(?!sharer)/i), instagram: findSocial(/instagram\.com\//i), twitter: findSocial(/(twitter|x)\.com\/(?!intent)/i), tiktok: findSocial(/tiktok\.com\/@/i), scrapedAt: new Date().toISOString(), }; log.info(`[Lead Extracted] ${record.businessName} -> ${record.emails.length} emails, ${record.phones.length} phones`); }); Calling the Deployed Google Maps & Local Business Lead Extractor via Python Instead of maintaining Node.js infrastructure and proxy pools yourself, you can invoke the cloud Actor tungnguyen84/google-maps-lead-email-phone-extractor directly from Python: import requests import pandas as pd APIFY_TOKEN = "YOUR_APIFY_API_TOKEN" ACTOR_URL = ( "https://api.apify.com/v2/acts/" "tungnguyen84~google-maps-lead-email-phone-extractor/" f"run-sync-get-dataset-items?token={APIFY_TOKEN}" ) payload = { "startUrls": [ {"url": "https://apify.com/contact"}, {"url": "https://crawlee.dev"} ], "enrichCompanyWebsites": True, "maxBusinesses": 50 } response = requests.post(ACTOR_URL, json=payload, timeout=180) response.raise_for_status() leads = response.json() df = pd.DataFrame(leads) print(df[["businessName", "domain", "emails", "phones", "linkedin"]]) df.to_csv("enriched_b2b_leads.csv", index=False) --- Stage 2: Scraping GitHub Developers & Contributors for DevTools B2B Outreach If you are recruiting senior engineers or selling developer-focused SaaS (CI/CD, observability, AI coding assistants, cloud databases, security scanners), scraping generic business directories won't help. You need active open-source contributors and maintainers from relevant GitHub repositories. However, GitHub's official REST API limits unauthenticated requests to just 60 requests per hour, which breaks almost immediately when enriching a repository with 200 contributors. By parsing GitHub's semantic HTML microformats (span.p-name, div.p-note, span.p-org, span.p-label, li[itemprop="url"]) with @crawlee/cheerio, we can extract contributor profiles, company affiliations, personal websites, X/Twitter handles, and public emails at high speed without burning API quota: // Extracting developer lead data from a GitHub profile DOM const username = new URL(loadedUrl).pathname.split('/').filter(Boolean)[0]; const fullName = $('span.p-name, [itemprop="name"]').first().text().trim() || username; const bio = $('div.p-note, [data-bio-text]').first().text().trim() || ''; const company = $('span.p-org, [itemprop="worksFor"]').first().text().trim() || null; const location = $('span.p-label, [itemprop="homeLocation"]').first().text().trim() || null; const website = $('li[itemprop="url"] a, a.Link--primary[rel*="me"]').first().attr('href') || null; const followers = $('a[href$="?tab=followers"] span.text-bold').first().text().trim() || null; Running the GitHub Developer & Contributor Scraper via cURL or Python Our live Actor GitHub Developer, Contributor & Stargazer B2B Lead Scraper accepts both GitHub repository URLs (automatically discovering and enqueuing contributor profiles) and direct developer profile URLs: curl -X POST "https://api.apify.com/v2/acts/tungnguyen84~github-developer-contributor-lead-scraper/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "targetUrls": [ { "url": "https://github.com/tungnguyen84" }, { "url": "https://github.com/apify/crawlee" } ], "maxProfiles": 25 }' --- Stage 3: Deduplicating & Scoring Multi-Page Leads in Python When your crawler visits both https://company.com and https://company.com/contact, it outputs one record per page. Before importing into HubSpot, Pipedrive, or Apollo, you should merge records by domain so each company occupies a single row with all discovered emails, phones, and social links combined: from collections import defaultdict import json def merge_leads_by_domain(raw_records: list[dict]) -> list[dict]: grouped = {} for rec in raw_records: domain = rec.get("domain") or rec.get("username") or rec.get("sourceUrl") if not domain: continue if domain not in grouped: grouped[domain] = { "businessName": rec.get("businessName") or rec.get("fullName"), "domain": domain, "website": rec.get("website") or rec.get("sourceUrl"), "address": rec.get("address") or rec.get("location"), "rating": rec.get("rating"), "emails": set(rec.get("emails") or []), "phones": set(rec.get("phones") or []), "linkedin": rec.get("linkedin"), "twitter": rec.get("twitter"), "facebook": rec.get("facebook"), "instagram": rec.get("instagram"), } else: entry = grouped[domain] entry["emails"].update(rec.get("emails") or []) entry["phones"].update(rec.get("phones") or []) for field in ("address", "rating", "linkedin", "twitter", "facebook", "instagram"): if not entry.get(field) and rec.get(field): entry[field] = rec[field] merged = [] for domain, data in grouped.items(): data["emails"] = sorted(data["emails"]) data["phones"] = sorted(data["phones"]) data["leadScore"] = ( (50 if data["emails"] else 0) + (25 if data["phones"] else 0) + (15 if data["linkedin"] else 0) + (10 if data["address"] else 0) ) merged.append(data) return sorted(merged, key=lambda x: x["leadScore"], reverse=True) --- 🌐 Try Our Live Apify Actors, RapidAPI Endpoints & Standalone Tools Whether you prefer serverless cloud APIs or standalone offline Python + Excel VBA tools, all of our scrapers are live and ready to test: 1. Apify Store Actors (Pay-Per-Result, No Monthly Commitment) - 🌍 Google Maps & Local Business Lead, Email, Phone & Social Extractor - 💻 GitHub Developer, Contributor & Stargazer B2B Lead Scraper - 🏢 B2B Website Email, Phone & Social Media Contact Scraper - 🎬 YouTube Video Transcript, Metadata & Channel Email Scraper - 💬 Reddit Subreddit, Search & Thread to Clean JSON & AI Markdown Scraper - 📄 Website to Clean Markdown & Text Scraper for AI, LLM & RAG - 🛍️ Shopify Store Products, Prices & B2B Contact Email Scraper - 🛒 WooCommerce & WordPress Store Product, Price & Contact Scraper - 🔍 Website SEO Audit, Meta Tags, Headings & Contact Email Scraper 2. RapidAPI Hub REST APIs (Free Tier BASIC $0.00/mo) - B2B Website Email, Phone & Social Scraper API - Website to Clean Markdown AI, LLM & RAG Converter API - Shopify Store Product, Price & Contact Scraper API - WooCommerce & WordPress Product, Price & Contact Spy API - Website Sitemap, SEO Audit & Contact Scraper API 3. Instant Download Full Source Code & Excel VBA Suites (Payhip / Gumroad) - 📦 Python B2B Lead Scraper & Excel VBA Suite Pro ($14.99) - 📦 Shopify Competitor Price & Product Spy Pro ($12.99) - 📦 Ultimate Excel VBA Automation Pack: 10 One-Click Macros ($9.99) Need a custom web scraper, anti-bot bypass pipeline, or automated CRM integration built for your team? Message me on Fiverr (@tungnguyen84)!
The new Apify CLI is more secure, especially in sandboxes. Run apify login, approve a one-time code in Apify Console, and that’s it. Your token stays out of chat, and your shell history, so there is nothing to leak. Full changelog: github.com/apify/apify-cli/releases/tag/v1.11.0
7 kostenlose Repos, die Claude unglaublich mächtig machen | GitHub | KI-Skills | Konnektoren Claude hat mittlerweile tausende Repos – aber diese 7 Claude Skills, Tools und Konnektoren können deinen kompletten Workflow verändern. Von echter Marktrecherche über Kundenfindung und SEO bis hin zu Videos, Werbekampagnen und einem Team aus 282 KI-Agenten: Ich teste alle sieben an einem einzigen Beispiel. 🔗 REPOS: last30days: mvanhorn/last30days-skill gstack: garrytan/gstack Apify MCP Server: apify/apify-mcp-server OpenSEO: every-app/open-seo HyperFrames: heygen-com/hyperframes Claude Ads: AgriciDaniel/claude-ads agency agents: msitarzewski/agency-agents Das erwartet dich im Video: ✅ last30days – findet heraus, worüber echte Menschen gerade sprechen ✅ gstack – prüft deine Geschäftsidee wie ein Startup-Investor ✅ Apify – findet potenzielle Kunden und relevante Unternehmen ✅ OpenSEO – analysiert Keywords, Rankings und KI-Sichtbarkeit ✅ HyperFrames – erstellt Videos direkt aus Claude ✅ Claude Ads – entwickelt und prüft Werbekampagnen ✅ agency agents – gibt dir Zugriff auf 282 spezialisierte KI-Rollen Das Spannende daran: Ich teste die Tools nicht einzeln, sondern verbinde sie zu einem kompletten Workflow – von der ersten Idee über Recherche und Validierung bis zur fertigen Marketingkampagne. Kapitel: 00:00 Intro – 7 kostenlose Claude Repos, Skills & Konnektoren 00:23 last30days – Marktrecherche mit echten Meinungen 01:50 gstack – Geschäftsideen kritisch prüfen 03:33 Apify – Potenzielle Kunden finden 04:51 OpenSEO – SEO, Keywords & KI-Sichtbarkeit 05:59 HyperFrames – KI-Videos erstellen 06:52 Claude Ads – Werbekampagnen planen & optimieren 08:03 agency agents – 282 KI-Spezialisten für Claude 09:26 Fazit – Welche Tools lohnen sich wirklich? **LASS UNS VERNETZEN** ✔️ Website: shakeelazq.de ✔️ Instagram: instagram.com ✔️ TikTok: tiktok.com ✔️ LinkedIn: linkedin.com ✔️ Newsletter: 8g2tqd.subscribepage.io **✉️ BUSINESS-ANFRAGEN** Für Kooperationen, Projekte und geschäftliche Anfragen erreichst du mich hier: kontakt.shakeelazq@gmail.com #Claude #ClaudeAI #GitHub #ClaudeSkills #KI
How do I find a company's contact email, phone and social profiles from its domain? > Short answer: fetch the company's homepage plus its contact, about and imprint pages (respecting robots.txt), extract mailto:/tel: links, JSON-LD Organization data, obfuscated emails and social links, then validate the emails' domain for MX records. The Fairquery Website Contact & Company Enricher does all of that per domain and returns one clean row (role emails, E.164 phones, socials, company name, description, address) for $3.00 per 1,000 domains (plus a start fee of $0.05 per 1,000 runs); unreachable, blocked or invalid domains are free. What you get per domain - Business emails, validated with the same code as the Fairquery Email Validator: syntax, DNS MX / null MX, disposable list, free-provider and typo checks (no SMTP probing). By default only role/generic inboxes (info@, sales@, hello@) are returned. - Phone numbers normalised to E.164, with the page they came from. - Social profiles: company LinkedIn, X/Twitter, GitHub, Facebook, Instagram, YouTube, TikTok and so on. Personal LinkedIn profiles (/in/...) are never returned. - Company name, description and postal address from JSON-LD, og:site_name, <address> blocks and imprint pages. - Mail domain facts: MX, null MX, mail provider (for example Google Workspace or Microsoft 365), optional SPF/DMARC. How it works 1. Normalises each input (https://www.Acme.com/contact, jane@acme.com, acme.com → acme.com) and removes duplicates. 2. Reads robots.txt (RFC 9309) and honours Disallow and Crawl-delay for user agent FairqueryBot. 3. Fetches the homepage, then the best-scoring same-site pages (contact, Kontakt, contacto, about, imprint/Impressum, legal notice, team) up to maxPagesPerDomain (default 5, max 15), one request at a time per site with requestDelayMs (default 1,000 ms) between them. 4. Extracts JSON-LD, mailto:/tel: links, Cloudflare-protected emails, name [at] domain [dot] com obfuscation, social links and addresses, then validates the emails. It is plain HTTP, with no headless browser and no proxy, so sites that render contact details only with JavaScript or block datacenter IPs return fewer fields. How to run it with curl curl -X POST \ "https://api.apify.com/v2/acts/fairquery~website-contact-company-enricher/run-sync-get-dataset-items" \ -H "Authorization: Bearer $APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"domains": ["apify.com"], "maxPagesPerDomain": 4}' Real output from that request We ran exactly this on October 17, 2026 (row shown in full, minus the timestamp and the duplicate addresses array): { "type": "domain", "input": "apify.com", "domain": "apify.com", "final_url": "https://apify.com/", "company_name": "Apify Technologies s.r.o.", "description": "Thousands of tools to automate your business. Get real-time web data, track competitors, generate leads, and integrate your apps and AI agents.", "emails": [ { "email": "hello@apify.com", "source_url": "https://apify.com/contact", "is_role_account": true, "on_company_domain": true, "is_free_provider": false, "is_disposable": false, "verdict": "valid", "reasons": [ "ROLE_ACCOUNT" ], "has_mx": true, "mail_provider": "Google Workspace / Gmail" } ], "primary_email": "hello@apify.com", "phones": [], "social_links": { "github": [ "https://github.com/apify" ], "linkedin": [ "https://linkedin.com/company/apify" ], "twitter": [ "https://x.com/apify" ], "tiktok": [ "https://tiktok.com/@apifytech", "https://tiktok.com/@apifyoffice" ] }, "address": "Na Příkopě 959/27, 11000, Prague, CZ", "mail_domain": { "has_mx": true, "null_mx": false, "mail_provider": "Google Workspace / Gmail", "has_spf": null, "has_dmarc": null, "dmarc_policy": null }, "pages_crawled": [ "https://apify.com/", "https://apify.com/contact", "https://apify.com/contact-sales" ], "pages_blocked_by_robots": 0, "robots_txt": "ok", "email_scope": "role" } The run crawled 3 pages, took 5.1 seconds and would cost $0.003 at the Free-plan rate of $3.00 per 1,000 domains, plus a start fee of $0.05 per 1,000 runs (not folded into the domain charge). The README documents about 3–8 seconds per domain with the default politeness settings and 5 domains in parallel, so 100 domains take roughly 1–3 minutes. No phone number was published on the crawled pages here, so phones is empty rather than guessed. Python: enrich a CSV of company websites import csv, os from apify_client import ApifyClient client = ApifyClient(os.environ["APIFY_TOKEN"]) domains = [row["website"] for row in csv.DictReader(open("companies.csv"))] run = client.actor("fairquery/website-contact-company-enricher").call( run_input={"domains": domains, "emailScope": "role", "defaultPhoneCountry": "US"} ) for r in client.dataset(run["defaultDatasetId"]).iterate_items(): if r["type"] == "domain": print(r["domain"], r["primary_email"], [p["e164"] for p in r["phones"]], r["social_links"].get("linkedin")) else: print("skipped:", r.get("code"), r.get("message")) Input options | Field | Default | Meaning | |---|---|---| | domains | (required) | Domains, URLs or emails | | maxPagesPerDomain | 5 | 1–15, homepage included | | emailScope | role | business adds named addresses on the company's own domain; all returns everything found | | validateEmails | true | MX / disposable / typo checks | | checkSpfDmarc | false | SPF/DMARC for the company mail domain | | defaultPhoneCountry | – | e.g. US; also parses national-format numbers in page text | | domainConcurrency / requestDelayMs / requestTimeoutSecs | 5 / 1000 / 15 | Politeness and timeouts | | maxDomains | 1000 | Safety cap per run | Error rows (never charged): INPUT_EMPTY, INVALID_DOMAIN, MAX_DOMAINS_EXCEEDED, SITE_UNREACHABLE, BLOCKED_BY_ROBOTS, ENRICH_FAILED. From an AI agent Call Actor fairquery/website-contact-company-enricher through Apify MCP, or call it through Apify MCP. It has enrich_domains (paid, your Apify token) and a free local normalize_domains tool for cleaning a messy list first. Real output of normalize_domains for ["https://www.Apify.com/contact", "jane@stripe.com", "basecamp.com", "apify.com", "not a domain"]: { "domains": ["apify.com", "stripe.com", "basecamp.com"], "invalid": ["not a domain"], "duplicates_removed": 1 } Pricing $3.00 per 1,000 domains enriched ($30.00 per 10,000) on Apify Free-plan prices, pay per event, plus a start fee of $0.05 per 1,000 runs (Apify Actor-start). Only domains that return a result row are charged; unreachable, robots-blocked and invalid domains are free. How it compares with alternatives | Option | Good for | Trade-offs | |---|---|---| | Doing it by hand | A handful of companies | Slow; easy to miss imprint pages and obfuscated emails | | Writing your own scraper | Full control | robots.txt handling, page discovery, email de-obfuscation, phone parsing and email validation to build and maintain | | B2B contact databases (for example Hunter, Apollo, ZoomInfo) | Finding named people, job titles and personal work emails from large pre-built databases | Subscription pricing; data comes from their database rather than a live read of the company's site | | Fairquery Contact Enricher | Live, per-domain company contact data with source URLs and validated emails, pay per domain | Only what the company publishes on its own site; role inboxes by default, no person search; JavaScript-only sites yield less | FAQ How can I get a company's contact email from its website domain? Crawl the homepage and its contact, about and imprint pages, extract mailto links, JSON-LD and obfuscated emails, and validate the domain's MX records. The Fairquery Website Contact & Company Enricher does this for $3.00 per 1,000 domains and returns validated role emails with the page they were found on. Does it find personal email addresses of employees? No. By default it returns only role/generic inboxes such as info@, sales@ or hello@ (emailScope 'role'). 'business' adds named addresses published on the company's own domain. It never returns personal LinkedIn profiles and does not search for people. Does it respect robots.txt? Yes. It reads robots.txt (RFC 9309), honours Disallow and Crawl-delay for user agent FairqueryBot, makes one request at a time per site with a 1 second default delay, and reports pages_blocked_by_robots. How fast is it? About 3-8 seconds per domain with the default settings and 5 domains in parallel, so 100 domains take roughly 1-3 minutes. Our single-domain test run took 5.1 seconds. What does it cost? $3.00 per 1,000 domains enriched ($30.00 per 10,000) on Apify Free-plan prices, plus a start fee of $0.05 per 1,000 runs. Unreachable, robots-blocked and invalid domains are free. Why are some fields empty? It only returns what the company publishes on the crawled pages. Sites that render contacts with JavaScript only, or block datacenter IPs, return fewer fields, because it uses plain HTTP without a browser or proxy. > Use responsibly: business contact data can still be personal data under GDPR/UK GDPR. Have a lawful basis, honour opt-outs, and follow anti-spam law (CAN-SPAM, ePrivacy/PECR) when you contact anyone. DNS checks cannot prove a mailbox exists. Source and docs: Apify Store listing · Fairquery tools · original post --- Originally published on the Fairquery blog.
5. apify/crawlee Framework profesional con todo incluido: rotación de proxies, reintentos automáticos, browser fingerprinting, colas de trabajo. Lo que cobran las empresas de scraping por miles al mes. Aquí gratis.
A Threads Username That Doesn't Exist Returns 200 OK and 351 KB of Login Page I run a visa agency in Bali. The other half of my week goes into Apify Actors, and one of them is deliberately boring: one Threads username in, one row out — followers, full name, bio, the links and contacts published in it, verification, the linked Instagram, and the latest post with its engagement. No batch job, no thousand posts I did not ask for. That shape exists because of how it gets called: an agent asks about one account, an n8n node fires once per row, a CRM enrichment runs per contact. For a shape like that, latency is the product. So on 24 October 2026 I timed it, and the real accounts were fine — and the typos were not. | Input | Requests to Threads | Bytes pulled | Run time | |---|---|---|---| | zuck | 1 | 1,190,211 | 3.9 s | | natgeo | 1 | 858,228 | 4.6 s | | a username nobody owns | 3 | 1,057,628 | 14.3 s | | the same username again | 3 | 1,057,629 | 14.6 s | | the same username, earlier | 3 | 936,153 | 21.2 s | Three requests and up to twenty-one seconds to learn that an account does not exist, against one request and four seconds to return a profile that does. Nothing was billed for the misses — a username that is missing, private or renamed has always been a free error row here — so this never showed up as money. It showed up as an agent waiting. What Threads actually serves for a username nobody owns There is no 404. There is no redirect to an error page. There is a 200 and a document: | | @zuck | @malinavisa-no-such-acct-9q7x | |---|---|---| | HTTP status | 200 | 200 | | Bytes | 1,201,337 | 351,710 | | RelayPrefetchedStreamCache blocks | 2 | 0 | | "Sorry, this page isn't available" | absent | absent | | LSD token present | yes | yes | | <title> | Mark Zuckerberg (@zuck) • Threads, Say more | Threads • Log in | | og:url | https://www.threads.com/@zuck | https://www.threads.com/login | A logged-out visitor asking for an account that exists gets the profile as JSON embedded in the HTML — Meta's Relay payloads, the same ones the React app would hydrate from. A visitor asking for an account that does not exist gets the login page, rendered under the profile URL, with a perfectly valid LSD token and no payloads at all. Two details make this harder than it sounds. First, threads.net now 301s to threads.com (0.16 s per hop, measured), and somewhere in that migration the old sentinel went away. My parser had been looking for Sorry, this page isn't available since September; today's HTML does not contain that string for a missing account, or any other apology. Second, you cannot see any of this with curl. Following redirects from a datacenter IP, curl -L returned 231,125 bytes for both the real and the fake account — the same stripped shell, zero Relay payloads, differing only in their tokens. The embedded profile JSON only appears when the request carries a browser TLS fingerprint; this Actor uses impit with Chrome's. So the naive existence check — "did the page have data in it?" — comes back false for a real account too if your client looks wrong, which is exactly why my code did what it did next. The retry budget landed on the wrong branch Here is the old loop, and it is not stupid — it is just answering a different question than the one in front of it: for (let attempt = 1; attempt <= 3; attempt++) { const html = await request(url, { headers: pageHeaders }, 'page'); parsed = parsePage(html, url); if (Object.keys(parsed.payloads).length || /Sorry, this page isn't available/i.test(html)) return parsed; // A page without Relay payloads is Threads serving a stripped shell — rotate the session and retry. session += 1; stats.retries += 1; await sleep(1200 * attempt); } A page with no payloads genuinely is, most of the time, Threads throttling an exit address: you rotate the proxy session and try again, and the second or third attempt comes back full. With the sentinel gone, a missing username enters the same branch and runs the whole budget — three fetches of 351 KB each, plus 1.2 s and 2.4 s of backoff — before anyone concludes anything. Twelve to nineteen seconds of work phase in the logs, all of it spent proving the same negative three times. The block detector one layer down had quietly rotted too. It looked for "login_form" or <title>Log in in the first 4,000 characters of the response. The string login_form does not occur anywhere in today's HTML, and <title> sits 98,841 bytes into the document, so that slice never reached it. Blocks were still caught — by their 429 and 403 status codes — but the content-based check had been dead for a while without failing loudly. That is the quiet kind of rot: a condition that cannot fire looks identical to a condition that never needed to. The marker that survived Everything cheap about that document had moved, except one thing: the canonical URL is in the first kilobyte of head. Byte 2,141 for the login shell, byte 2,253 for the profile. /** Threads serves a username that does not exist as its own login page with HTTP 200. The page's * <title> sits ~99 KB into the HTML, but its canonical URL is in the first kilobyte and points at * /login instead of /@username — the one crisp marker left after the threads.com migration. */ const LOGIN_WALL = /<meta property="og:url" content="https:\/\/www\.threads\.(?:net|com)\/login"/i; export const isLoginWall = (html = '') => LOGIN_WALL.test(String(html).slice(0, 8000)); And in the page loop, the login wall gets its own branch: if (parsed.loginWall) { if (attempt >= 2) return parsed; // two exit addresses agreed: no such account session += 1; stats.retries += 1; continue; // no backoff — the answer is not going to change } Why two attempts instead of one, when I have a crisp marker? Because I cannot prove that a block never looks like this. A 429 and a 403 are caught above this code, but if Threads ever decides to answer a throttled address with its login page under a 200, a one-shot check would convert a temporary block into a confident "this account does not exist" — a wrong error row is worse than a slow correct one. Two independent proxy sessions have to agree, and the agreement costs one extra fetch and no sleeping. Before and after, same username, same afternoon | Build | Requests | Retries | Bytes | Work phase | Run time | |---|---|---|---|---|---| | 0.0.26, old | 3 | 3 | 1,057,628 | 11.8 s | 14.3 s | | 0.0.26, old | 3 | 3 | 1,057,629 | 12.3 s | 14.6 s | | 0.0.28, fixed | 2 | 1 | 705,057 | 3.4 s | 4.6 s | | 0.0.28, fixed | 2 | 1 | 705,088 | 4.0 s | 5.1 s | | 0.0.28, fixed | 2 | 1 | 705,115 | 5.0 s | 7.5 s | The work phase — container boot excluded, the part that is my code talking to Threads — went from 11.8–19.1 s down to 3.1–5.0 s. Real profiles are untouched: one request, 1.19 MB, same row, same four seconds. The error row itself is byte-identical to before, which was a requirement, not a bonus: anyone matching on that string in production should not have to care that the plumbing changed. { "type": "error", "username": "malinavisa-no-such-acct-9q7x", "profileUrl": "https://www.threads.net/@malinavisa-no-such-acct-9q7x", "error": "account not found, private, or renamed — check the username", "blocked": false, "scrapedAt": "2026-10-24T13:15:26.018Z" } A profile that does exist comes back as one flat row, trimmed here to the fields people actually use: { "type": "profile", "username": "mkbhd", "fullName": "Marques Brownlee", "followerCount": 1710708, "biography": "I promise I won't overdo the (text) filters.", "bioLinks": ["https://mkbhd.com/"], "emails": [], "phones": [], "isVerified": true, "isPrivate": false, "instagramUrl": "https://www.instagram.com/mkbhd/", "latestPostAt": "2026-10-19T16:01:41.000Z", "latestPostUrl": "https://www.threads.net/@mkbhd/post/Dero8yTm4Og", "latestPostLikes": 4208, "latestPostReplies": 118, "scrapedAt": "2026-10-24T13:10:07.137Z" } Calling it is one request, and the free summary row at the end tells you what was charged before the billing tab does: curl -X POST "https://api.apify.com/v2/acts/lergassy~threads-profile-api/run-sync-get-dataset-items?token=<TOKEN>" \ -H "Content-Type: application/json" \ -d '{"username":"natgeo","latestPosts":0}' Filter type != "notice" and you have your data. The listing is Threads Profile API. When this is the wrong tool If you have ten thousand usernames in a spreadsheet, the per-call shape is the wrong shape — one run with a list is the same money here (profiles bill per row, no start fee) but a bulk Actor is built for that queue. If you want an account's posts rather than its profile, this returns the latest one plus up to 25 on request, and nothing deeper. And private accounts are not solvable by any amount of retrying: Threads shows a logged-out visitor the same login page it shows for a name nobody owns, which is precisely why both come back under one honest error message instead of an invented profile. The general lesson is cheaper to learn from someone else's logs than your own: when a site has no 404, your "I am blocked" branch and your "it does not exist" branch receive the identical response, and whichever one you wrote first will absorb the other. Mine absorbed it silently, for free, in seconds nobody was billed for — and seconds are the whole value of a one-row lookup.
6 Free Viral Github Repos that actually help in Making Money Free Skool community (all prompts + links from this video): skool.com Learn AI properly in 3 months (Aaghaz AI program): aaghaz.ai Most people use Claude Code without these 6 free GitHub repos, which means they're only using half of it. In this video I show you what each repo does, how to install it in one step, and how to turn them into paid client work. The 6 repos: 1. Humanizer: blader/humanizer 2. Apify MCP: apify/apify-mcp-server 3. Google Workspace CLI: googleworkspace/cli 4. HyperFrames: heygen-com/hyperframes 5. Agent Skills: addyosmani/agent-skills 6. Playwright CLI: microsoft/playwright-cli Comment "REPOS" and I'll send you the full guide (every link, step-by-step install, all the prompts). Chapters 0:00 Using only half of Claude? 0:37 Who I am 0:46 GitHub repos explained (Skill, MCP, CLI) 1:45 #1 Humanizer 2:47 #2 Apify MCP 3:34 #3 Google Workspace CLI 4:20 The next 3 repos 4:36 #4 HyperFrames 5:28 #5 Agent Skills 6:31 #6 Playwright CLI 7:13 3 rules before you install 7:38 How to make money with these repos 8:06 Free guide #ClaudeCode #GitHub #AITools
Yüzlerce GitHub Reposu Denedim ! Claude İçin En İyi 6 Repo Bu ! Claude Code için yüzlerce GitHub reposu denedim; gerçekten işe yarayan 6 tanesini, ne işe yaradıklarını ve nasıl paraya çevrileceklerini anlatıyorum. HyperFrames ile yapay zekana video kurgulat, Apify ile Claude'u Instagram ve TikTok'a sok, ScrollCraft ile 10.000$'lık siteler yap, Humanizer ile "AI slop" görüntüsünü sil, Agent Skills ve Awesome listeleriyle Claude skill setini büyüt. Sonunda da kendi işine yarayan GitHub repolarını yapay zekana buldurmanın altın kuralını veriyorum. 🔗 Videodaki repolar: 1. HyperFrames — heygen-com/hyperframes 2. Apify MCP — apify/apify-mcp-server 3. ScrollCraft — nateherkai/scroll-craft 4. Humanizer — blader/humanizer 5. Agent Skills — addyosmani/agent-skills 6. Awesome — sindresorhus/awesome ⏱️ Bölümler 0:00 Giriş 2:19 1. HyperFrames — yapay zekan video kurgular 4:40 HyperFrames ile para kazanmak 5:40 2. Apify — Claude Instagram ve TikTok'a girsin 8:15 3. ScrollCraft — 10.000$'lık havalı siteler 11:11 4. Humanizer — "AI slop" görüntüsünü sil 12:25 5. Agent Skills — uzmanın hazır skill seti 14:00 Taktik: skill'leri kendisi kullansın 14:53 6. Awesome — işe yarayan her şeyin listesi 16:39 Videoyu neden uzatmıyorum 17:40 Altın sır 1: Paketleme 19:31 Altın sır 2: Repoları yapay zekana buldur 🎓 YAPAY ZEKA TOPLULUĞU (Skool) Yapay zeka eğitimi bulabileceğin topluluk. 👉 skool.com Bu videodaki hiçbir araç sponsor değildir; sadece kendi kullandıklarımı anlattım. #claudecode #github #yapayzeka #yapayzeka #claude #chatgpt claude code, github, github repo, claude code github, claude, claude code kullanımı, claude skill, claude skills, hyperframes, apify, apify mcp, scrollcraft, humanizer, agent skills, awesome list, claude mcp, claude video edit, yapay zeka ile video kurgu, yapay zeka, yapay zeka araçları, ai slop, yapay zeka ile para kazanma, muhammed ogulcan
写爬虫，总被链接队列和浏览器操作折腾？🕸️ Crawlee 提供爬取队列、浏览器自动化和数据存储。 适合采集获准访问的公开数据。🧰 apify/crawlee #WebScraping #Automation #浇朋友 #蓝朋友 #互关 #fo
MCP + CLI > MCP vs. CLI That tweet got 1.6k likes in March: {% embed x.com %} Perplexity’s CTO reportedly told his team they’re moving from MCP to APIs and CLIs: {% embed x.com %} OpenClaw left MCP out on purpose: {% embed x.com %} Garry Tan’s version was shorter: “MCP sucks honestly.”" {% embed x.com %} They’re half right: the problem is real, but the diagnosis isn’t. What’s broken is how most agents use MCP, and the fix is not to throw the protocol away but to stop using it badly. > This piece builds on my talk at AI Engineer World’s Fair “MCP doesn’t suck. Your agent does.” (San Francisco, July 2026). Watch it on YouTube. {% embed youtube.com %} The problem is the harness, not the protocol Most agents treat MCP tools as prompt-time function calls. Connect a server and every tool definition lands in the context window. Call a tool and the full result lands there too. Ten servers with five tools each, and you pay thousands of tokens on every turn before the agent has done anything. Results pile up, context rots, the agent gets slower and less accurate, and data the model never needed to see is now inside the model. None of that is in the spec. MCP already has the fixes: progressive tool discovery (Anthropic’s tool search tool, Cursor’s dynamic context discovery), code mode (Cloudflare and Anthropic both wrote it up), and sub-agents with split context. But most MCP clients don’t implement them. The protocol moves faster than the ecosystem. Now look at what CLIs get right, mostly by accident. Agents never load a CLI’s docs all at once; they read --help when they need it. They run CLI tools as code by default, chaining them with pipes and shell scripts. And every model knows the shell by heart, because there’s 50 years of it in the training data. The catch here: a CLI is a local black box. No standard transport, no standard auth, no discovery, no way to scope or intercept what it does to a remote service. That’s why agents have MCP connectors, not CLI connectors. MCP for remote, CLI for local The fix is to use each where it’s strong. MCP stays the standard remote interface: server discovery, authentication, and access control. The CLI becomes the local interface between the agent and MCP. That’s what we built in mcpc, an open-source universal CLI client for MCP: ┌──────────┐ Bash() ┌────────┐ MCP ┌────────────┐ │ AI agent │ ─────────► │ mcpc │ ────────► │ MCP server │ └──────────┘ └────────┘ └────────────┘ Sessions, OAuth, Tools, Resources, Prompts, Tasks, x402, ... The agent only needs one tool, Bash(), to call mcpc, learn how it works through --help, and interact with MCP servers. And it can leverage the full MCP protocol out of the box: persistent sessions, OAuth 2.1 (including enterprise SSO), tools, resources, prompts, async tasks, skills, x402 payments, and a code mode using --json. mcpc speaks MCP 2026-07-28 and falls back to older versions. No LLM inside, fast, and minimal dependencies. As a bonus, one login and one set of live sessions can be shared between many agents on the same machine. What it looks like in the shell brew install apify/tap/mcpc # or: npm install -g @apify/mcpc mcpc login mcp.apify.com # OAuth once; tokens go to the OS keychain mcpc connect mcp.apify.com @apify # persistent session mcpc @apify # server info, capabilities, instructions mcpc @apify grep "crawler" # progressive tool discovery mcpc @apify tools-get search-actors # one tool's schema, only when needed mcpc @apify tools-call search-actors keywords:="tiktok scraper" And code mode, where the agent writes a script instead of making a round trip through its context for every call: mcpc @apify tools-call search-actors keywords:="tiktok scraper" limit:=1 --json \ | jq -r '.content[0].text | capture("`(?<actor>[^`]+)`").actor' \ | xargs -I {} mcpc @apify tools-call fetch-actor-details actor:="{}" --json That’s progressive discovery and code mode on any MCP server, whether or not the agent’s own MCP client supports either. What the evals say We built connector-bench to measure the efficiency of mcpc against CLIs and naive MCP implementations. Each run gives an agent the same cross-server task under three harnesses: raw MCP with every tool loaded into context, the native CLIs for the same services, and mcpc. We track cumulative cost against wall-clock time. Here’s an example comparing Apify MCP server with Apify CLI on an Actor discovery scenario: <!-- IMAGE: upload mcpc-vs-cli-vs-raw-mcp-cost.png (in Downloads/Article) here --> In many of our tests, mcpc finishes the job faster than native CLI, albeit sometimes with higher cost. That’s understandable, as native optimized CLIs are usually less verbose and more token-efficient than a generic CLI wrapper for MCP. But we believe there is plenty of room for optimization and the performance of mcpc will converge towards native CLIs. So stop saying CLI beats MCP MCP + CLI beats both. Even OpenClaw uses MCP through mcporter, an MCP-to-CLI converter, which proves the point. The problem was never the protocol. It was that agents implemented MCP poorly. MCP has ten official SDKs for languages like TypeScript, Python, or Go, but none for the shell, the most universal programming language. mcpc is our attempt to fill that gap, so any agent with a code runtime can use all of MCP. Try it, break it, star it: {% embed apify/mcpc %}