List of Bots, Web Crawlers and User Agents

List of web crawlers. Default order is alphabetical by crawler name.

Name & LinkPlatformUsage
80legs-
Audisto CrawlerSEO
Beam Us Up SEO CrawlerSEO
BotifySEO
ContentKingSEO
CrawlMonsterSEO
DeepCrawlSEO
Gnutella crawler-
Heritrixweb archiving
HextraktSEO
JetOctopusSEO
Norconex Collectors -
OnCrawlSEO
Screaming Frog SEO Spider SEO
SEO MacroscopeSEO
Sitebulb Website Crawler SEO
SpotiboSEO
TubeKitYouTube
Visual SEO Studio SEO
Webbee SEO Spider SEO
WildShark SEO SpiderSEO
Wolf Spider-
Xenu’s Link SleuthSEO
YaCy P2P search engine
READ. SHARE. REPEAT.
  •  
  •  
  •  
  •  
  •  
  •