crawler

浏览过的精彩逆向文章汇总，值得一看

🤖 Scrape data from HTML websites automatically by just providing examples

K 哥爬虫代码分享，JS 逆向，爬虫进阶。关注公众号：K哥爬虫

Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.

The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.

📝 quickly crawl the information (e.g. followers, tags etc...) of an instagram profile.

The fastest dork scanner written in Go.

Boss直聘爬虫 / BOSS直聘职位数据抓取工具，基于 Chrome CDP 协议复用真实登录态，绕过字体反爬，输出明文薪资 JSON/CSV + 薪资技能分析。A Chrome-CDP-based BOSS Zhipin job scraper/crawler.

Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性，基于 Workerman。

一个好用的哔哩哔哩漫画下载器，拥有图形界面，支持关键词搜索漫画和二维码登入，黑科技下载未解锁章节，多线程下载，多种保存格式，本地漫画管理，一键检查更新！

基于appium的app自动遍历工具

Easily download all the photos/videos from tumblr blogs. 下载指定的 Tumblr 博客中的图片，视频

📰 Newspaper4k a fork of the beloved Newspaper3k. Extraction of articles, titles, and metadata from news websites.

Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.

Run a high-fidelity browser-based web archiving crawler in a single Docker container

Crawly, a high-level web crawling & scraping framework for Elixir.

Write web scrapers in Ruby using a clean, AI-assisted DSL. Kimurai uses AI to figure out where the data lives, then caches the selectors and scrapes with pure Ruby. Get the intelligence of an LLM without the per-request latency or token costs.

Scalable Python web scraping scripts for +40 popular domains

Golang短视频去水印：抖音,皮皮虾,火山,微视,最右,快手,全民小视频,皮皮搞笑,西瓜视频,虎牙,梨视频,acfun,好看视频...

An ergonomic, privacy-aware Rust HTTP Client

小说下载|小说爬取|起点|笔趣阁|导出Markdown|导出txt|转换epub|广告过滤|自动校对

A tool for pixiv.net. 人人可用的P站爬虫

Google play scraper for Python inspired by <facundoolano/google-play-scraper>

A scalable, mature and versatile web crawler based on Apache Storm

SpiderSuite (web security crawler) releases, wiki and roadmap

A high performance web crawler / scraper in Elixir.

Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.

Pure-Python protocol tools and research notes for X.com, with VibeLoft Twitter account datasets

A multi-thread crawler framework with many builtin image crawlers provided.

ChatWeb can crawl web pages, read PDF, DOCX, TXT, and extract the main content, then answer your questions based on the content, or summarize the key points.