crawler

爬虫逆向案例，已完成：TLS指纹｜瑞数｜震坤行 | 网易易盾 | 微信小程序反编译逆向(百达星系) | 同花顺 | rpc解密 | 加速乐 | 极验滑块验证码 | 巨量算数 | Boss直聘 | 企查查 | 中国五矿 | qq音乐 | 产业政策大数据平台 | 企知道 | 雪球网(acw_sc__v2) | 1688 | 七麦数据 | whggzy | 企名科技 | mohurd | 艺恩数据 | 欧科云链

SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA engineers, and consultants. Supports Windows, macOS, and Linux (x64 and arm64).

HTTP API for Scrapy spiders

High-performance asynchronous Douyin(抖音) TikTok Xiaohongshu(小红书) Kuaishou(快手) Weibo(微博) Instagram YouTube(油管) Twitter(X) Captcha Solver(验证码解决器) Temp Mail(临时邮箱) API(接口).

A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion.

ArrowDL (Arrow Downloader) is a download manager for Windows, MacOS and Linux

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

ai reverse 一把梭

Simple but useful Python web scraping tutorial code.

Java API For Chrome and Firefox

Doujinshi downloader 绅士漫画下载

js cookie逆向利器：js cookie变动监控可视化工具 & js cookie hook打条件断点

Open source SEO audit tool.

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON.

《爬虫逆向进阶实战》书籍代码库

A lightweight web crawler framework.（Java爬虫框架）

The Big Brother V6.0 is a weaponized OSINT platform featuring username enumeration (473+ platforms), quad-vector visual intelligence, Sky Radar tracking, crypto wallet analysis, SSL intelligence, digital footprint reconstruction, EXIF extraction, advanced dorking, and network reconnaissance.

:newspaper: Let ChatGPT Summarize Hacker News for You

A Tumblr and Twitter Blog Backup Application

The best PTT library

Wscan is a web security scanner that focuses on web security, dedicated to making web security accessible to everyone.

Search google, bing, yahoo, and other search engines with python

Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"

NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。

Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabilities using realistic, randomly generated decoy data and AI-generated HTML templates.

Locally saves webpages to your hard disk with images, css, js & links as is.

Moodle-DL downloads course content fast from Moodle (eg. lecture pdfs)

Jie stands out as a comprehensive security assessment and exploitation tool meticulously crafted for web applications. Its robust suite of features encompasses vulnerability scanning, information gathering, and exploitation, elevating it to an indispensable toolkit for both security professionals and penetration testers. 挖洞辅助工具(漏洞扫描、信息收集)

a reliable high-level web crawling & scraping framework for Node.js.

Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents.