cxcscmu/Craw4LLM

Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"

↗ Open on GitHub

Stars

664

Forks

60

Language

Python

License

MIT

Last push

2 years ago

Python llm large-language-models crawler pretraining pre-training web-crawler crawling web-crawling

No related intel yet

This repo has not appeared in any of the sources the radar tracks. The collector runs on a schedule — check back once it covers this repo.