scraping

The context API to search, scrape, and interact with the web at scale. 🔥

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

Scrapy, a fast high-level web crawling & scraping framework for Python.

🕵️‍♂️ Collect a dossier on a person by username from 3000+ sites

Open source AI job application bot in Python: browser automation and web scraping to read job postings, then auto-apply with a tailored resume and cover letter for each posting.

Python scraper based on AI

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Elegant Scraper and Crawler Framework for Golang

Pythonic HTML Parsing for Humans™

Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)

A scalable web crawler framework for Java.

🦊 Anti-detect browser

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

Stealth headless browser for AI agents — bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement.

List of libraries, tools and APIs for web scraping and data processing.

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Tabula is a tool for liberating data tables trapped inside PDF files

Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

🚀 Free HTTP, SOCKS4, & SOCKS5 proxy list * Updated every 5 minutes * and rotating proxy API (100+ countries)

Swiss-army tool for scraping and extracting data from online assets, made for hackers

Twitter API Scraper | Without an API key | Twitter Internal API | Free | Twitter scraper | Twitter Bot

Mechanize is a ruby library that makes automated web interaction easy.

Snoop — инструмент разведки на основе открытых данных (OSINT world)

AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.

Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons & enroll you for PAID UDEMY COURSES, ABSOLUTELY FREE!

Scrape Facebook public pages without an API key

A browser testing and web crawling library for PHP and Symfony

An open source fingerprint browser based on Ungoogled Chromium. 指纹浏览器 隐私浏览器

Learn step-by-step how to scrape Google Trends data and make a result comparison using Python and Oxylabs SERP API. Extract keywords, their popularity, breakdown by region, related queries, and more.