List of all Crawlers

008

008 is the user-agent used by 80legs, a web crawling service provider. 80legs allows its users to design and run custom web crawls.

Click on any string to get more details

008 0.83

ABACHOBotABACHOBot

Abacho's spider. German based portal and search engine. Has localized versions in the following countries: Austria, Switzerland, France, UK, Spain, Italy, Sweden and Turkey.

Click on any string to get more details

ABACHOBot

Accoona-AI-AgentAccoona-AI-Agent

Accoona's webcrawler

Click on any string to get more details

Accoona-AI-Agent 1.1.2

Accoona-AI-Agent 1.1.1

AddSugarSpiderBot


Click on any string to get more details

AddSugarSpiderBot

AnyApexBot

Crawler for the web directory AnyApex

Click on any string to get more details

AnyApexBot 1.0

Arachmo

Japanese Crawler. Seems to be a download tool. Here's some information in japanese. If you can translate than, please let me know

Click on any string to get more details

Arachmo

B-l-i-t-z-B-O-T

Crawler for the German search engine tricus. Spiders German, Dutch, Swiss and Austrian websites. Same as BlitzBOT

Click on any string to get more details

B-l-i-t-z-B-O-T

BaiduspiderBaiduspider

Crawler for the chinese search engine Baidu

Click on any string to get more details

Baiduspider 2.0

Baiduspider

BecomeBotBecomeBot

Become crawler. Shopping related portal

Click on any string to get more details

BecomeBot 3.0

BecomeBot 2.3

BeslistBot

Dutch shopping portal

Click on any string to get more details

BeslistBot 1.0

BillyBobBot


Click on any string to get more details

BillyBobBot 1.0

Bimbot

Unknown crawler, gives no information. IP address belongs to Backbone Communications Inc. (BBCOM). Provides converged data and voice services

Click on any string to get more details

Bimbot 1.0

Bingbot

Bot for Microsofts Bing search engine

Click on any string to get more details

Bingbot 2.0

BlitzBOT

Crawler for the German search engine tricus. Spiders German, Dutch, Swiss and Austrian websites. Same as B-l-i-t-z-B-O-T

Click on any string to get more details

BlitzBOT

boitho.com-dc

Boitho's Web Crawler, a distributed crawler that downloads web pages to build the database used by Boitho.com to search in. To allow volunteers to donate their superfluous bandwidth and idle CPU time, they have developed a distributed crawler, like [email protected] and Grub. That way people can install a program on their computers and help them with the crawling.

Click on any string to get more details

boitho.com-dc 0.85

boitho.com-dc 0.83

boitho.com-dc 0.82

boitho.com-dc 0.81

boitho.com-dc 0.79

boitho.com-robot

This is an old version of Boitho's boitho.com-dc. It was a more traditional webrobot, run on computers controlled by Boitho, while boitho.com-dc is a distributed crawler run on the computers of volunteers.
The boitho.com-robot isn抰 in use any more.

Click on any string to get more details

boitho.com-robot 1.1

boitho.com-robot 1.0

btbot

btbot's search engine for bittorrents, ringtones for cell phones, friends and extraterrestrial intelligence

Click on any string to get more details

btbot 0.4

CatchBot

Web crawler for Catch, the online division of Reed Business Information Australia

Click on any string to get more details

CatchBot 2.0

CatchBot 1.0

Cerberian Drtrs


Click on any string to get more details

Cerberian Drtrs 3.2

Charlotte

Charlotte is a spider created by Searchme, Inc. in Mountain View, CA

Click on any string to get more details

Charlotte 1.1

Charlotte 1.0t

Charlotte 1.0b

Charlotte 0.9t

ConveraCrawler

ConveraCrawler is an experimental web crawler under development since April 2004. ConveraCrawler is owned and operated by Convera Corporation

Click on any string to get more details

ConveraCrawler 0.9e

ConveraCrawler 0.9d

ConveraCrawler 0.9

cosmos

Crawler from xyleme which indexes XML content on the web.

Click on any string to get more details

cosmos 0.9

Covario IDS

Proprietary crawler used as part of Covario's Organic Search Insight solution

Click on any string to get more details

Covario IDS 1.0

DataparkSearchDataparkSearch

Open source web-based search engine released under the GNU General Public License and designed to organize search within a website, group of websites, intranet or local system. DataparkSearch consists of two parts. The first part is indexing mechanism (indexer). Indexer walks over html hypertext references and stores found words and new references into database. The second part is web CGI front-end to provide search using data collected by indexer.

Click on any string to get more details

DataparkSearch 4.37

DataparkSearch 4.36

DataparkSearch 4.35

DiamondBot

Crawler for Claria (formerly Gator). Adware company

Click on any string to get more details

DiamondBot

Discobot

Discobot is the experimental web crawler for Discovery Engine

Click on any string to get more details

Discobot 1.0

Dotbot


Click on any string to get more details

Dotbot 1.1

Dotbot 1.0.1

EmeraldShield.com WebBot

Crawls domains as part of a spam and web filtration services. If a site is determined to contain questionable, or objectionable content it will be added to a blocklist. Ignores the robots.txt file

Click on any string to get more details

EmeraldShield.com WebBot

envolk[ITS]spider

envolk search engine spider [ITS] Internet Tracking Spider(TM)

Click on any string to get more details

envolk[ITS]spider 1.6

EsperanzaBot

Web Crawler of Esperanza Consulting LTD

Click on any string to get more details

EsperanzaBot

ExabotExabot

Exava shopping search engine, belongs now to Become

Click on any string to get more details

Exabot 2.0

FAST Enterprise Crawler

Product of the norvegian company Fast. Part of their FAST ProPublish solution for gathering, processing and delivering reference material to online and offline users.

Click on any string to get more details

FAST Enterprise Crawler 6

FAST-WebCrawler

Crawler for the Fast search engine

Click on any string to get more details

FAST-WebCrawler 3.8

FAST-WebCrawler 3.7

FAST-WebCrawler 3.6

FAST-WebCrawler 3.x

FDSE robotFDSE robot

Search engine of Fluid Dynamics Software Corporation

Click on any string to get more details

FDSE robot

FindLinks

A project of the Automated Speech Processing Group at the Institute of Computer Science at Universit盲t Leipzig.

Click on any string to get more details

FindLinks 2.0.1

FindLinks 1.1.6-beta6

FindLinks 1.1.6-beta4

FindLinks 1.1.6-beta1

FindLinks 1.1.5-beta7

FindLinks 1.1.4-beta1

FindLinks 1.1.3-beta9

FindLinks 1.1.3-beta8

FindLinks 1.1.3-beta6

FindLinks 1.1.3-beta4

FindLinks 1.1.3-beta2

FindLinks 1.1.3-beta1

FindLinks 1.1.2-a5

FindLinks 1.1.1-a5

FindLinks 1.1.1-a1

FindLinks 1.1.1

FindLinks 1.1-a9

FindLinks 1.1-a8

FindLinks 1.1-a7

FindLinks 1.1-a5

FindLinks 1.1-a4

FindLinks 1.1-a3

FindLinks 1.1

FindLinks 1.06

FindLinks 1.0.9

FindLinks 1.0.8

FindLinks 1.0

FurlBot

Furl's crawler. Furl is a social bookmark service from LookSmart

Click on any string to get more details

FurlBot Furl Search 2.0

FyberSpiderFyberSpider

FyberSearch web crawler

Click on any string to get more details

FyberSpider

g2crawler

g2crawler : Gnutella2Crawler codename Aenea. Not in use anymore.

Click on any string to get more details

g2crawler

GaisbotGaisbot

Gais - Global Area Information Servers - Search enginge crawler of the National Chung Cheng University Taiwan

Click on any string to get more details

Gaisbot 3.0+

Gaisbot 3.0

GalaxyBot

Browser for Galaxy Classifieds, a searchable directory.

Click on any string to get more details

GalaxyBot 1.0

genieBot

Web-indexing robot of GenieKnows Local Search Engine

Click on any string to get more details

genieBot

GigabotGigabot

Gigablast's indexing agent

Click on any string to get more details

Gigabot 3.0

Gigabot 2.0

Gigabot 1.0

GirafabotGirafabot


Click on any string to get more details

Girafabot

GooglebotGooglebot


Click on any string to get more details

Googlebot 2.1

Googlebot-ImageGooglebot-Image

Google's image crawler

Click on any string to get more details

Googlebot-Image 1.0

GurujiBot

Indian search engine

Click on any string to get more details

GurujiBot 1.0

HappyFunBot

Crawler for Happy Fun Search

Click on any string to get more details

HappyFunBot 1.1

hl_ftien_spider

Web Crawler from China. IP addresses belong to Qipusi Technology Ltd and Rongzhengwuye-ltd from Tjanjin city

Click on any string to get more details

hl_ftien_spider 1.1

hl_ftien_spider

Holmes

Sherlock Holmes is a open source universal search engine. The URL can be added by the user. Often used to spam your logfiles

Click on any string to get more details

Holmes 3.9

Holmes 3.12.4

Holmes 3.12.3

Holmes 3.12.2

Holmes 3.12.1

htdig

Crawler of the ht://Dig Group's software package, a system for indexing and searching a finite (not necessarily small) set of sites or intranet. It is not meant to replace any of the many internet-wide search engines. htdig retrieves HTML documents using the HTTP protocol.

Click on any string to get more details

htdig 3.1.6

htdig 3.1.5

iaskspider

Bot for iAsk , chinese search engine from Sina.com

Click on any string to get more details

iaskspider 2.0

iaskspider

ia_archiveria_archiver

Alexa Web crawler

Click on any string to get more details

ia_archiver 8.9

ia_archiver 8.8

ia_archiver 8.2

ia_archiver 8.1

ia_archiver 8.0

ia_archiver

iCCrawler

ICCrawler is ICCenter's specialized web-crawling robot. Currently they are collecting only job offers from company sites. Those job offers are getting listed at ICjobs

Click on any string to get more details

iCCrawler

ichiro

Japanese Webcrawler for Goo

Click on any string to get more details

ichiro 4.0

ichiro 3.0

ichiro 2.0

igdeSpyder

Crawler for the russian IGDE commercial search engine

Click on any string to get more details

igdeSpyder

IRLbot

IRL-crawler is a Texas A&M University research project sponsored in part by the National Science Foundation that investigates algorithms for mapping the topology of the Internet and discovering the various parts of the web. The crawler downloads random web pages (text only) and follows certain links to find other websites.

Click on any string to get more details

IRLbot 3.0

IRLbot 2.0

IssueCrawlerIssueCrawler

Govcom.org Foundation's web bot. Locates and visualizes networks on the Web. The Issue Crawler is used by NGOs and other researchers to answer questions about specific networks and effective networking more generally. You also may do in-depth research with the software. You need an account to use it.

Click on any string to get more details

IssueCrawler

Jaxified Bot


Click on any string to get more details

Jaxified Bot

JyxobotJyxobot

Czech Webcrawler for Jyxo

Click on any string to get more details

Jyxobot 1

KoepaBot


Click on any string to get more details

KoepaBot

L.webis

Crawler developed at the Institute of Informatics and Telematics (IIT), of the National Research Council (CNR) of Italy, in Pisa

Click on any string to get more details

L.webis 0.87

LapozzBotLapozzBot

Hungarian bot. Spiders for the Lapozz search engine.
躣v鰖l鰉 !?!

Click on any string to get more details

LapozzBot 1.4

Larbin

Multi-purpose web crawler

Click on any string to get more details

Larbin 5.0

Larbin 2.6.3

Larbin 2.6.2

Larbin 2.6.1

Larbin 2.5.0

Larbin xy250

Larbin

LDSpider

LDSpider project aims to build a web crawling framework for the linked data web

Click on any string to get more details

LDSpider

LexxeBot

Bot for Lexxe Search Engine

Click on any string to get more details

LexxeBot 1.0

Linguee Bot

Search engine for bilingual texts. Helps with translating common phrases into another language

Click on any string to get more details

Linguee Bot

LinkWalker

SEVENtwentyfour Inc Link Checker

Click on any string to get more details

LinkWalker 2.0

LinkWalker

lmspider

Collects text from the web as part of a research project at Scansoft (renamed Nuance) ,trying to use web documents to improve the linguistic models used in their speech recognition engine

Click on any string to get more details

lmspider

lwp-trivial

lwp-trivial is the user-agent associated with the Perl code Module LWP::Simple

Click on any string to get more details

lwp-trivial 1.41

lwp-trivial 1.38

lwp-trivial 1.36

lwp-trivial 1.35

lwp-trivial 1.33

mabontland

Crawler for the web directory mabontland

Click on any string to get more details

mabontland

magpie-crawler

Crawler for Brandwatch

Click on any string to get more details

magpie-crawler 1.1

Mediapartners-GoogleMediapartners-Google

Unregistered versions of opera prior to 8.5 contained advertising. To serve up relevant adverts based on what you are browsing Google provided these adverts.
More information

Click on any string to get more details

Mediapartners-Google 2.1

MJ12botMJ12bot

Majestic-12 Web Crawler

Click on any string to get more details

MJ12bot 1.2.4

MJ12bot 1.2.3

MJ12bot 1.0.8

MJ12bot 1.0.7

MJ12bot 1.0.6

MJ12bot 1.0.5

MnogosearchMnogosearch

Web search engine software for intranet and internet servers from Mnogosearch.org (a project of Lavtech)

Click on any string to get more details

Mnogosearch 3.1.21

mogimogi

Unclear. The IP address belongs to Goo but they don't give any information about that bot. Goo itself uses ichiro for their search engine

Click on any string to get more details

mogimogi 1.0

MojeekBotMojeekBot

MojeekBot (formerly Citenikbot) is the web crawler for the Mojeek search engine.

Click on any string to get more details

MojeekBot 2.0

MojeekBot 0.2

Moreoverbot

Rssfeed bot

Click on any string to get more details

Moreoverbot 5.1

Moreoverbot 5.00

Morning Paper

Crawler for Boutell.com.

Click on any string to get more details

Morning Paper 1.0

msnbotmsnbot

MSN (or Microsoft Service Network) Search Web Crawler

Click on any string to get more details

msnbot 2.1

msnbot 2.0b

msnbot 1.1

msnbot 1.0

msnbot 0.9

msnbot 0.11

msnbot 0.1

MSRBot

Microsoft Research web crawler

Click on any string to get more details

MSRBot

MVAClient

I have no information about this one. The ip address belongs to Chunghwa Telecom Co.,Ltd. in Taiwan. It is blacklisted by SORBS. If you know anything about this bot please let me know

Click on any string to get more details

MVAClient

mxbot

Crawler for Chainn

Click on any string to get more details

mxbot 1.0

NetResearchServer

Spider for LOOP Improvements. Crawls the web by using the links found in the DMOZ Open Directory Project.

Click on any string to get more details

NetResearchServer 4.0

NetResearchServer 3.5

NetResearchServer 2.8

NetResearchServer 2.7

NetResearchServer 2.5

NetResearchServer

NetSeer Crawler


Click on any string to get more details

NetSeer Crawler 2.0

NewsGator


Click on any string to get more details

NewsGator 2.5

NewsGator 2.0

NG-SearchNG-Search

NG-Search is experimental searchengine with new semantic trials to list the most relevance words and groups around your query

Click on any string to get more details

NG-Search 0.9.8

NG-Search 0.86

nicebot


Click on any string to get more details

nicebot

noxtrumbot

Spanish search engine for Spanish and Portuguese pages. Belongs to TPI, Telef髇ica Publicidad e Informaci髇, S.A

Click on any string to get more details

noxtrumbot 1.0

Nusearch Spider

Crawls for the Nusearch search engine. Customizable search engine with some additional features like active bookmarks, and alternative result views.

Click on any string to get more details

Nusearch Spider

NutchCVS

Open source robot

Click on any string to get more details

NutchCVS 0.8-dev

NutchCVS 0.7.2

NutchCVS 0.7.1

NutchCVS 0.7

NutchCVS 0.06-dev

NutchCVS 0.05

Nymesis


Click on any string to get more details

Nymesis 1.0

obot

German spider from Cobion, now part of Internet Security Systems. Scans the web for their clients looking for copyright infringement

Click on any string to get more details

obot

oegp

The IP address belongs to the Deutsche Telekom in Germany. They don't give any information about that crawler. IP address is blacklisted

Click on any string to get more details

oegp 1.3.0

omgilibot


Click on any string to get more details

omgilibot 0.4

omgilibot 0.3

OmniExplorer_Bot

New crawler for Omni-Explorer. Site not launched yet (February 06)

Click on any string to get more details

OmniExplorer_Bot 6.70

OmniExplorer_Bot 6.65a

OmniExplorer_Bot 6.63b

OmniExplorer_Bot 6.62

OmniExplorer_Bot 6.60

OmniExplorer_Bot 6.47

OmniExplorer_Bot 5.91c

OmniExplorer_Bot 5.28

OmniExplorer_Bot 5.25

OmniExplorer_Bot 5.20

OmniExplorer_Bot 5.01

OmniExplorer_Bot 4.80

OmniExplorer_Bot 4.32

OOZBOT


Click on any string to get more details

OOZBOT 0.20

OOZBOT 0.17

OrbiterOrbiter

Spider for DailyOrbit search engine. Visits only the homepage of a domain.

Click on any string to get more details

Orbiter

PageBitesHyperBotPageBitesHyperBot

Crawler for PageBites, a search engine for job openings and/or r閟um閟. You can also post your r閟um