TILs on scraping

Atom feed for scraping

scraping Proxy Strategies for Web Scraping - 2026-04-09

The proxy model is dictated by what the target's anti-bot layer binds trust to: the IP, a session, or a clearance cookie. Pick the model from the target's behaviour, not from the proxy provider's marketing. This article covers the three models, how to tell which one a target needs, and the operational rules that keep a residential pool healthy and cheap. …

scraping Bypassing Cloudflare with curl_cffi and Cookie Management - 2026-04-09

How to scrape Cloudflare Enterprise-protected pages using browser TLS impersonation, a pool of solved cookie sets, and a stealth browser for challenge solving -- without running a headless browser per request. …

scraping Bypassing Akamai Bot Manager with curl_cffi - 2026-04-09

How to scrape Akamai-protected pages using browser TLS impersonation and a pool of pre-warmed sessions — without a headless browser. …

scraping Bypassing AWS WAF with Proof-of-Work Token Generation - 2026-04-09

How to solve AWS WAF challenges programmatically using curl_cffi, a proof-of-work solver, and browser fingerprint emulation -- without a headless browser. …