Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

The All in One Framework to Build Undefeatable Scrapers

Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"

A simple web scraper to extract Product Data and Pricing from Amazon

Omnisci3nt is an open-source web reconnaissance and intelligence tool for extracting deep technical insights from domains, including subdomains, SSL certificates, exposed services, archived content, and configuration data. — Omnisci3nt gives you the full picture in seconds.

Scrapy Training companion code

Parser and database to index the terpene profile of different strains of Cannabis from online databases

💵 💰 🇧🇷 Informações sobre taxas oficiais diárias de Inflação, Selic, Poupança, Dólar, Dólar PTAX, Euro e Euro PTAX pelo site do Banco Central do Brasil

Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO 👉

Command Line Tool to download torrents

PhantomCrawler is a Python-based web testing and research tool that simulates website interactions from multiple proxy IP addresses to analyze traffic behavior, access controls, and response patterns under different network conditions.

Compares price of the product entered by the user from e-commerce sites Amazon and Flipkart 💰 📊

Web Scraping Craigslist's Engineering Jobs in NY with Scrapy

Scraping and Web Crawling Framework For Zhihu Live

implementing an end-to-end tweets ETL/Analysis pipeline.

🕷️ A lightweight Model Context Protocol (MCP) server that exposes Crawl4AI web scraping and crawling capabilities as tools for AI agents. Similar to Firecrawl's API but self-hosted and free. Perfect for integrating web scraping into your AI workflows with OpenAI Agents SDK, Cursor, Claude Code, and other MCP-compatible tools.

Web scraping API for building AI applications.

Zoominfo scraper with using of rotating proxies and headless Chrome from ScrapingAnt

A TensorFlow (Deep Learning - CNN) based solution for tackling captcha when collecting data from Amazon.

It contain various script on web crawling/ data mining of social web(RSS,facebook,twitter,Linkedin)