crawler

Leaked GPTs Prompts Bypass the 25 message limit or to try out GPTs without a Plus subscription.

Web Scraping Framework

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.

🏳️‍🌈 Media downloader from any sites, including Twitter, Reddit, Instagram, BlueSky, TikTok, Threads, Facebook, OnlyFans, YouTube, Pinterest, PornHub, XHamster, XVIDEOS, ThisVid etc.

Transform Web Content into LLM-Ready Data

蓝天采集器是一款开源免费的爬虫系统，仅需点选编辑规则即可采集数据，可运行在本地、虚拟主机或云服务器中，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统

Web crawling framework based on asyncio.

爬虫案例合集。包括但不限于《淘宝、京东、天猫、豆瓣、抖音、快手、微博、微信、阿里、头条、pdd、优酷、爱奇艺、携程、12306、58、搜狐、各种指数、维普万方、Zlibraty、Oalib、小说、招标网、采购网、小红书、大众点评、推特、脉脉、知乎》

Free antidetect browser stealth for Playwright: undetected headless Firefox fingerprint. Python scraping, recaptcha and bot detection bypass. Open source

NewPipe's core library for extracting data from streaming sites

浏览器内存漫游解决方案（探索中...）

磁力網站U3C3介紹以及域名更新

Flexible Node.js AI-assisted crawler library

多平台内容监控·采集·搬运 一个 Web 面板管起抖音 / 小红书 / 快手

Diskover Community Edition - Open source file indexer, file search engine and data management and analytics powered by Elasticsearch

Google, Naver multiprocess image web crawler (Selenium)

Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.

python爬虫，目前库存：网易云音乐歌曲爬取，B站视频爬取，知乎问答爬取，壁纸爬取，xvideos视频爬取，有声书爬取，微博爬虫，安居客信息爬取+数据可视化，哔哩哔哩视频封面提取器，ip代理池封装，知乎百万级用户爬虫+数据分析，github用户爬虫

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

抖音爬虫——采集账号主页、喜欢、收藏、音乐原声、话题、搜索、合集、作品、关注、粉丝等公开数据。

ScopeSentry-Cyberspace mapping, subdomain enumeration, port scanning, sensitive information discovery, vulnerability scanning, distributed nodes

CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)

❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)

Elasticsearch File System Crawler (FS Crawler)

An ergonomic, privacy-aware Python HTTP Client

小红书数据采集、网站图片、视频资源批量下载工具，颜值超高的数据采集工具（批量下载，视频提取，图片）Telegram:https://t.me/+ZtLSwuIKTo44MDY1

Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to patch/unpatch, can be enabled/disabled on demand.

A web privacy measurement framework