web-crawlers

YiraBot: Simplifying Web Scraping for All. A user-friendly tool for developers and enthusiasts, offering command-line ease and Python integration. Ideal for research, SEO, and data collection.

open-source machine-learning data-mining scraping python3 text-extraction web-scraping html-parser robots-txt data-extraction seotools command-line-tool beginner-friendly contributions-welcome big-data-analytics seo-analysis good-first-issue sitemap-parser web-crawlers

Updated Jul 24, 2026
Python

ancatmara / learnpython2017

Star

Python course for 2nd year NLP students at NRU HSE, 2017-2018

Updated Nov 17, 2023
Jupyter Notebook

romis2012 / is-bot

Star

Detect bots/crawlers/spiders via user-agent string

python user-agent bots user-agent-parser crawlers bot-detection web-crawlers

Updated Aug 4, 2026
Python

mihaicodes / web-crawler

Star

A Web Crawler developed in Python.

python crawler web script scripting web-crawler scripts python-script scripting-language python3 python-3 crawlers webcrawler web-crawling web-crawler-python webcrawling webcrawl crawler-python web-crawlers

Updated Aug 8, 2022
Python

ilovedevs / awesome-web-crawler

Star

List of best web crawlers to extract data from the web. Find web crawling tools for different needs.

Updated Sep 19, 2025

AAlkiyumi / Senior-Design-Project

Star

Web scraper for collecting product and review data from e-commerce sites using Scraping Bee, AWS, Selenium, and Pandas. Focuses on cost-effective solutions, user-friendly interfaces, and efficient data extraction and analysis.

aws data-storage selenium pandas web-scraping data-extraction uc data-analysis data-processing product-reviews web-crawlers university-of-cincinnati e-commerce-analytics scraping-bee senior-design-project e-commerce-data review-sentiment-analysis uc2026 cs5001

Updated Apr 4, 2025
Python

henu-wang / ai-robots-txt-generator

Star

🤖 Generate optimized robots.txt files for AI search engine crawlers (GPTBot, PerplexityBot, ClaudeBot, and more)

python seo geo gemini web-scraping robots-txt web-optimization ai-search perplexity web-crawlers chatgpt gptbot ai-seo llms-txt generative-engine-optimization ai-crawlers geoscore claudebot

Updated Mar 9, 2026
Python

MiracleWc / Web-crawlers-travel-information

Star

Web crawlers based on travel information, crawler data from Ctrip, tuniu, qunar.

web-crawlers travel-information crawler-data

Updated Sep 8, 2019
Python

dudgrayman / agent-ready-kit-template

Star

Template statique pour rendre un petit site plus lisible par les agents IA, crawlers et outils d’extraction.

mcp seo static-site json-ld seo-optimization ai-agents seo-tools web-crawlers llms-txt agent-ready

Updated Jun 3, 2026
HTML

codassassin / web-crawler-v2.0

Star

This is an advanced version of the previously released version of web-crawler

python web-crawler python3 subdomain web-crawling web-crawler-python subdomain-scanner web-crawls subdomain-enumeration subdomain-scan web-crawlers subdomain-finder subdomain-scanners

Updated Aug 5, 2021
Python

elmohq / awesome-answer-engine-optimization

Sponsor

Star

Official documentation, crawler user agents, specifications, research, and tools for Answer Engine Optimization (AEO) and Generative Engine Optimization (GEO).