scraper

📰 Newspaper4k a fork of the beloved Newspaper3k. Extraction of articles, titles, and metadata from news websites.

Crawly, a high-level web crawling & scraping framework for Elixir.

Write web scrapers in Ruby using a clean, AI-assisted DSL. Kimurai uses AI to figure out where the data lives, then caches the selectors and scrapes with pure Ruby. Get the intelligence of an LLM without the per-request latency or token costs.

Scalable Python web scraping scripts for +40 popular domains

Unofficial MyAnimeList PHP+REST API which provides functions other than the official API

Universal Reddit Scraper - A comprehensive Reddit scraping/archival command-line tool.

一个管理电脑内资源，并提供游戏化的综合管理器

An ergonomic, privacy-aware Rust HTTP Client

Google play scraper for Python inspired by <facundoolano/google-play-scraper>

First ever tool to view "Instagram private posts" anonymously

A high performance web crawler / scraper in Elixir.

Fetch X/Twitter tweets, replies, timelines, and articles without login or API keys — field tool for AI agents.

HTTP API for Scrapy spiders

A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion.

Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.

Make a ZIM file from any Web site and surf offline!

A general-purpose doujinshi and image gallery downloader

Converts some webnovels to epub format

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

Java API For Chrome and Firefox

An open database of international sanctions data, persons of interest and politically exposed persons

Self-hosted scraping engine — bypasses any JS challenge & captcha: Cloudflare, Turnstile, reCAPTCHA, hCaptcha, GeeTest. FlareSolverr & Byparr alternative and drop-in replacement for your *arr stack.

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON.

operative framework is a rust investigation OSINT framework, you can interact with multiple targets, execute multiple modules, create links with target, export rapport to PDF file, add note to target or results, interact with RESTFul API, write your own modules.

A Multi-User Selfhosted price tracker for Amazon, Aliexpress, ebay and many more along with custom stores. Get notified when price matches one or more criterias set by the user

Python package for scraping real estate property data

The Slash OSINT Tool

A Scala library for scraping content from HTML pages

A list of scrapers from around the web.

Twitter Internal API Document