Topic · crawlers

crawlers

Tracked open-source repos tagged crawlers, sorted by stars.

Repos

2

Total stars

5,257

Avg. stars

2,629

Share

0.00%

Topics that frequently appear alongside crawlers on the same repo.

Recent risers

Repos created in the last 90 days, tagged crawlers.

No new repos tagged with this topic in the last 90 days.

#1
ai.robots.txt ↗@ai-robots-txtPython
A list of AI agents and robots to block.
★ 4,097+12Star change over the last 7 days
#2
isbot ↗@omrilotanTypeScript
🤖/👨‍🦰 Detect bots/crawlers/spiders using the user agent string
★ 1,160+0Star change over the last 7 days

← Back to topics