Hi guys,
When making full-flash websites, it’s quite effective to deliver html-content to search engines.
To determine whether a visitor is a robot or not, you have to match the visitor’s user agent against a list of known bot user agents.

I parsed a bot user agent list out of this table: http://www.pgts.com.au/pgtsj/pgtsj0208d.html
You can easily match a user agent string against this list.

ADSAComponent (postmaster@cnds.ucd.ie)
Mozilla/2.0 (Compatible; AOL-IWENG 3.0; Win16)
ASPseek/1.2.10
ASPseek/1.2.11
ASPseek/1.2.12
http://www.almaden.ibm.com/cs/crawler [c01]
http://www.almaden.ibm.com/cs/crawler [fc3]
http://www.almaden.ibm.com/cs/crawler [wf224]
http://www.almaden.ibm.com/cs/crawler [wf55]
Amfibibot/0.06 (Amfibi Robot; http://www.amfibi.com; agent@amfibi.com)
Mozilla/4.0 (Search Engine Marketing Tactics Amsterdam 2002 Information Spider)
AnswerBus (http://www.answerbus.com/)
antibot-V1.1.11/i586-linux-2.2
antibot-V1.1.13/i586-linux-2.2
antibot-V1.2.0/redhat-linux-9
appie 1.1 (www.walhello.com)
Argus/1.1 (Nutch; http://www.simpy.com/bot.html; feedback at simpy dot com)
Art-Online.com 0.9(Beta)
Mozilla/2.0 (compatible; Ask Jeeves)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
Mozilla/3.0 (compatible; AvantGo 3.2)
BDFetch
BDNcentral Crawler v2.3 [en] (http://www.bdncentral.com/robot.html)
BDNcentral Crawler v2.3 [en] (http://www.bdncentral.com/robot.html) (X11; I; Linux 2.0.44 i686)
BaiDuSpider
Baiduspider+(+http://www.baidu.com/search/spider.htm)
battlebot
Big Brother (http://pauillac.inria.fr/~fpottier/)
BlogBot/1.2
boitho.com-dc/0.4 ( http://www.boitho.com/dcbot.html )
boitho.com-dc/0.5 ( http://www.boitho.com/dcbot.html )
boitho.com-dc/0.66 ( http://www.boitho.com/dcbot.html )
boitho.com-robot/1.0
boitho.com-robot/1.1
Mozilla/4.0 (compatible; BorderManager 3.0)
BrailleBot 1.0
BruinBot (+http://webarchive.cs.ucla.edu/bruinbot.html)
bumblebee/1.0 (bumblebee@relevare.com; http://www.relevare.com/)
Computer_and_Automation_Research_Institute_Crawler (nospamspidernospam@spider.ilab.sztakinospam.hunospam)
Computer_and_Automation_Research_Institute_Crawler (spider@spider.ilab.sztaki.hu)
cd34/0.1
Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.0-Build-41)
Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.0-Build-43)
CerberianDrtrs/Version-3.0-Release-24
Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.0-Build-40)
Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-11)
Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-12)
Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-13)
Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-16)
Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.1-Build-17)
Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-0)
CipinetBot (http://www.cipinet.com/bot.html)
Clushbot/2.1 (+http://www.clush.com/bot.html)
Clushbot/3.21-BinaryFury (+http://www.clush.com/bot.html)
Clushbot/3.23-BinaryFury (+http://www.clush.com/bot.html)
Clushbot/3.24-BinaryFury (+http://www.clush.com/bot.html)
Clushbot/3.6-BinaryFury (+http://www.clush.com/bot.html)
Clushbot/3.9-BinaryFury (+http://www.clush.com/bot.html)
ComMOOnity LambdaMOO/1.8.1
CrawlConvera0.1 (CrawlConvera@yahoo.com)
CrawlConvera0.1 (www.authoritativeweb.com)
ConveraCrawler/0.2
ConveraCrawler/0.5 (+http://www
cosmos/0.9_(robot@xyleme.com)
Cowbot-0.1 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)
Cowbot-0.1.1 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)
Crawl_Application
CrocCrawler v3.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686)
CrocCrawler v4.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686)
Custo 2.0 (www.netwu.com)
CydralSpider/1.9 (Cydral Web Image Search; http://www.cydral.com)
DeMozulator 1.0 (MacOS, dMoz URL Check Agent, trebor@animeigo.com)
DeepIndex (http://www.deepindex.com)
DoCoMo/1.0/N504i/c10/TB
DoCoMo/1.0/P504iS/c10/TB
Dual Proxy
Dumbot(version 0.1 beta – dumbfind.com)
Dumbot(version 0.1 beta – http://www.dumbfind.com/dumbot.html)
Dumbot(version 0.1 beta)
e-SocietyRobot(http://www.yama.info.waseda.ac.jp/~yamana/es/)
EARTHCOM.info/1.2
EmailSiphon
Enterprise_Search/1.00.136;MSSQL (http://www.innerprise.net/es-spider.asp)
exactseek-crawler-2.63 (crawler@exactseek.com)
exactseek-crawler-2.63 crawler@exactseek.com
exactseek-crawler-2.63-5 (crawler@exactseek.com)
exactseek-crawler-2.63-5 crawler@exactseek.com
Explorer 6
FAST Enterprise Crawler/6 (crawler@fast.no)
FAST Enterprise Crawler/6 (www.fastsearch.com)
FAST FirstPage retriever (compatible; MSIE 5.5; Mozilla/4.0)
FAST-WebCrawler/3.2 test
FAST-WebCrawler/3.4/PartnerSite (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)
FAST-WebCrawler/3.6 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
FAST-WebCrawler/3.6/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
FAST-WebCrawler/3.6/FirstPage (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)
FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://www.alltheweb.com/help/webmaster/crawler)
FAST-WebCrawler/3.8 (crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler)
FAST-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
FAST-WebCrawler/3.x Multimedia
FAST-WebCrawler/3.x Multimedia (mm dash crawler at fast dot no)
Mozilla/4.0 (compatible: FDSE robot)
FastBug http://www.ay-up.com
favicon finder at http://iconsurf.com/
favicon monitor at http://iconsurf.com/
Filangy/0.01-beta (Filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com)
Filangy/1.01 (Filangy; http://www.filangy.com/filangyinfo.jsp?inc=robots.jsp; filangy-agent@filangy.com)
Filangy/1.01 (Filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com)
FindLinks/0.71 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/0.82 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/0.87 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/0.89 (+http://wortschatz.uni-leipzig.de/findlinks/)
Firefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5)
Flickbot 1.1 RPT-HTTPClient/0.3-3
FlickBot 2.0 RPT-HTTPClient/0.3-3
Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info@searchhippo.com)
Mozilla/4.0 (compatible; MSIE 5.0; www.galaxy.com; http://www.pgts.com.au/; +http://www.galaxy.com/info/crawler.html)
FyberSpider (+http://www.fybersearch.com/fyberspider.php)
GAIS Robot/1.1A2
Gaisbot/3.0+(robot@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)
GalaxyBot/1.0 (http://www.galaxy.com/galaxybot.html)
gatherer/0.9
gazz/5.0 (gazz@nttr.co.jp)
Generic
GeonaBot 1.0; http://www.geona.com/
GeonaBot/1.1; http://www.geona.com/
GetRight/4.5e
Gigabot/1.0
Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com)
Goldfire Server
Googlebot/2.1 (+http://www.google.com/bot.html)
Googlebot/2.1 (+http://www.googlebot.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot/Test (+http://www.googlebot.com/bot.html)
Googlebot-Image/1.0
Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html)
Green Research, Inc.
GregBot (compatible; MSIE; Windows; Q312461)
Mozilla/4.0 (compatible; grub-client-0.3.0; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.0.3; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.0.4; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.0.5; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.0.6; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.0.7; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.07; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.1.1; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.2.1; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.3.1; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.3.7; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-1.5.3; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-2.3)
grub-client
Crawler [en] (compatible; Crawler Gulper Web Bot 0.2.4 www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)
Mozilla/5.0 [en] (compatible; Gulper Web Bot 0.2.4 www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)
HTTPConnect
Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)
Harvest-NG/1.0.2
Hatena Antenna/0.4 (http://a.hatena.ne.jp/help#robot)
Hatena Antenna/0.4 (http://a.hatena.ne.jp/help)
hget/0.3
Hitwise Spider v1.0 http://www.hitwise.com
htdig/3.1.5 (admin@ipc-opc.lan)
htdig/3.1.5 (unconfigured@htdig.searchengine.maintainer)
htdig/3.1.6 (http://computerorgs.com)
htdig
Html Link Validator (www.lithopssoft.com)
Httpcheck/1.0 (Perl 5.006001)
httpget-5.2.2
Mozilla/4.0 (compatible; ICS 1.2.105)
IPiumBot laurion(dot)com
IRLbot/1.0 (+http://irl.cs.tamu.edu/crawler)
ia_archiver
lcabotAccept: */*
ichiro/1.0 (ichiro@nttr.co.jp)
IconSurf/2.0 favicon monitor (see http://iconsurf.com/robot.html)
IlTrovatore-Setaccio/0.03-dev (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)
IlTrovatore-Setaccio/1.2 (+http://www.iltrovatore.it/aiuto/faq.html)
IlTrovatore-Setaccio/1.2 (Indexing; http://www.iltrovatore.it/bot.html; bot@iltrovatore.it)
IlTrovatore-Setaccio/1.2 (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)
IlTrovatore-Setaccio/1.2 (It-bot; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)
IlTrovatore-Setaccio/1.2-dev (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)
IlTrovatore-Setaccio (+http://www.iltrovatore.it)
Iltrovatore-Setaccio/0.3-dev (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)
Iltrovatore-Setaccio/1.2 (It-bot; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)
Iltrovatore-Setaccio
imagefetch/0.1 libwww-perl/5.66
Mozilla/3.0 (compatible; Indy Library)
InelaBot/0.2 (+http://inelegant.org/bot)
InfoSeek Sidewinder/1.0A
Infoseek SideWinder/1.45 (Compatible; MSIE 10.0; UNIX)
Infoseek SideWinder/2.0B (Linux 2.4 i686)
Mozilla/3.0 (INGRID/3.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints)
Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Mozilla/5.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Slurp/2.0 (slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Slurp/si-emb (slurp@inktomi.com; http://www.inktomi.com/slurp.html)
InternetLinkAgent/3.1
http://www.istarthere.com (spider@istarthere.com)
Java1.4.0
JoBo/1.3 (http://www.matuschek.net/jobo.html)
k2spider
KMcrawler
Knowledge.com/0.2
Knowledge.com/0.3
Knowledge Engine
kuloko-bot/0.2
LNSpiderguy
Larbin larbin2.6.2@unspecified.mail
larbin_2.6.2 (kalou@kalou.net)
larbin_2.6.2 (larbin2.6.2@unspecified.mail)
larbin_2.6.2 (larbin@correa.org)
larbin_2.6.2 (pimenas@softnet.tuc.gr)
larbin_2.6.2 (pimenas@systems.tuc.gr)
larbin_2.6.2 (sumeet_sobti@yahoo.com)
larbin_2.6.2 (vitalbox1@hotmail.com)
larbin_2.6.2 (vshelk@yahoo.com)
larbin_2.6.2 larbin2.6.2@unspecified.mail
larbin_2.6.2 larbin@correa.org
larbin_2.6.2 pimenas@systems.tuc.gr
larbin_2.6.2 sumeet_sobti@yahoo.com
larbin_2.6.2 vitalbox1@hotmail.com
larbin_2.6.3 (andreas.beder@chello.at)
larbin_2.6.3 (larbin-crawler@un.bewaff.net)
larbin_2.6.3 (larbin2.6.3@unspecified.mail)
larbin_2.6.3 (pimenas@softnet.tuc.gr)
larbin_2.6.3 larbin2.6.3@unspecified.mail
larbin_2.6.3_for_(http://cosco.hiit.fi/search/) (Tomi.Silander@hiit.fi)
larbin_2.6.3_for_(http://cosco.hiit.fi/search/) (tsilande@hiit.fi)
larbin_2.6.3_for_(http://cosco.hiit.fi/search/) Tomi.Silander@hiit.fi
larbin_2.6.3_for_(http://cosco.hiit.fi/search/) tsilande@hiit.fi
eseek-crawler-larbin-2.63 (crawler@exactseek.com)
eseek-crawler-larbin-2.63 crawler@exactseek.com
LARBIN-EXPERIMENTAL (efp@gmx.net)
LARBIN-EXPERIMENTAL efp@gmx.net
Larbin (larbin2.6.2@unspecified.mail)
MSIE-5.13 (larbin@unspecified.mail)
MSIE-5.13 larbin@unspecified.mail
Mozilla (la2@unspecified.mail)
Mozilla la2@unspecified.mail
Mozilla/4.0 (efp@gmx.net)
Mozilla/4.0 efp@gmx.net
SearchGuild_DMOZ_Experiment (chris@searchguild.com)
SearchGuild_DMOZ_Experiment chris@searchguild.com
WinampMPEG/2.00 (larbin@unspecified.mail)
WinampMPEG/2.00 larbin@unspecified.mail
larbin (samualt9@bigfoot.com)
larbin samualt9@bigfoot.com
larbin_extended (larbin@oktie.com)
larbin_test (nobody@airmail.etn)
libwww-MGET/1.0 libwww/5.2.8
/ libwww/5.3.2
/ libwww/5.4.0
libwww-perl/5.48
libwww-perl/5.50
libwww-perl/5.51
libwww-perl/5.52 FP/4.0
libwww-perl/5.53
libwww-perl/5.63
libwww-perl/5.64
MyApp/0.1 libwww-perl/5.65
libwww-perl/5.65
rawiswar/0.1 libwww-perl/5.66
libwww-perl/5.68
VanillaZilla/0.1 libwww-perl/5.69
libwww-perl/5.69
libwww-perl/5.74
libwww-perl/5.75
libwww-perl/5.76
libwww-perl/5.800
libwww-perl/5.801
libwww-perl/5.802
libwww-perl/5.803
Perl-Win32::Internet/0.082
LimeBot/1.0 (+www.cruiselime.com/LimeBot.php)
LinkLint-checkonly/2.3.5
Linkbot 3.0
Linknzbot 2004/(+http://www.linknz.co.nz/robot.php)
Linknzbot/ (+http://www.linknz.co.nz/robot.php)
Links SQL (http://gossamer-threads.com/scripts/links-sql/)
Lite Bot 0616B
Look.com
lwp-trivial/1.29
lwp-trivial/1.35
lwp-trivial/1.36
lwp-request/2.01
LWP::Simple/5.48
LWP::Simple/5.65
Lycos_Spider_(modspider)
Microsoft Data Access Internet Publishing Provider Cache Manager
Microsoft Data Access Internet Publishing Provider DAV
Microsoft Data Access Internet Publishing Provider DAV 1.1
Microsoft Data Access Internet Publishing Provider Protocol Discovery
MSFrontPage/4.0
Mozilla/2.0 (compatible; MS FrontPage 4.0)
MSFrontPage/5.0
Mozilla/2.0 (compatible; MS FrontPage 5.0)
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; Q312461; BTopenworld; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Q312461; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; matlas-2.0.2501; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MSIECrawler)
MSProxy/2.0
MSRBOT/0.1 (http://research.microsoft.com/research/sv/msrbot/)
Mediapartners-Google/2.1
Mediapartners-Google/2.1 (+http://www.googlebot.com/bot.html)
Mercator-2.0
MetaGer-LinkChecker
metacarta (crawler@metacarta.com)
metacarta crawler@metacarta.com
Microsoft URL Control – 5.00.3609
Microsoft URL Control – 5.01.4319
Microsoft URL Control – 6.00.8169
Microsoft URL Control – 6.00.8862
Microsoft-ATL-Native/7.00
MicrosoftPrototypeCrawler (How’s my crawling? mailto:newbiecrawler@hotmail.com)
moget/1.0 (moget@goo.ne.jp)
moget/2.1 (moget@goo.ne.jp)
mozDex/0.04-dev (mozDex; http://www.mozdex.com/bot.html; spider@mozdex.com)
mozDex/0.05-dev (mozDex; http://www.mozdex.com/bot.html; spider@mozdex.com)
Mozilla/4.0 (compatible; Netcraft Web Server Survey)
MSNBOT/0.1 (http://search.msn.com/msnbot.htm)
msnbot/0.11 (+http://search.msn.com/msnbot.htm)
msnbot/0.3 (+http://search.msn.com/msnbot.htm)
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
Mozilla/3.01 (compatible;)
NG/1.0
NPBot
NPBot (http://www.nameprotect.com/botinfo.html)
NPBot-1/2.0
NPBot-1/2.0 (http://www.nameprotect.com/botinfo.html)
NaverBot-1.0 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)
NaverBot_dloader/1.5
dloader(NaverRobot)/1.0
dloader(NaverRobot)/1.5
NetAnts/1.25
NetNoseCrawler/v1.0
Mozilla/4.0 (compatible; MSIE 5.0; NetNose-Crawler 2.0; A New Search Experience: http://www.netnose.com)
NetResearchServer/2.4(loopimprovements.com/robot.html)
NetResearchServer/2.5(loopimprovements.com/robot.html)
NetResearchServer/2.7(loopimprovements.com/robot.html)
NetResearchServer/2.8(loopimprovements.com/robot.html)
NetResearchServer/2.9(loopimprovements.com/robot.html)
NetResearchServer/3.4(loopimprovements.com/robot.html)
NetResearchServer(http://www.look.com)
NextGenSearchBot 1 (for information visit http://www.eliyon.com/NextGenSearchBot)
none
NuSearch Spider www.nusearch.com
NutchCVS/0.05 (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)
NutchCVS/0.05-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)
CreativeCommons/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)
Robot: NutchCrawler, Owner: wdavies@acm.org
NutchOrg/0.03-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net)
OWR_Crawler 0.1
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot)
oBot
Ocelli/1.3 (http://www.globalspec.com/Ocelli)
OmniExplorer_Bot/1.07 (+http://www.omni-explorer.com) Internet Categorizer
OmniExplorer_Bot/1.07 (+http://www.omni-explorer.com) Job Crawler
Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html)
Openbot/3.0+(robot@monkia.com.tw;+http://gais.cs.ccu.edu.tw/robot.php)
Openfind data gatherer, Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html)
OrangeBot
Mozilla/4.0 (compatible; Advanced Email Extractor v2.24)
Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
PEERbot www.peerbot.com
PWS.Kiosk – Content Filtering
parabot (paracite@ecs.soton.ac.uk)
Patwebbot (http://www.herz-power.de/technik.html)
pavuk/0.9pl28 i586-pc-cygwin
pavuk/0.9pl29b i686-pc-linux-gnu
pipeLiner/0.3a (PipeLine Spider; http://www.pipeline-search.com/webmaster.html; webmaster@pipeline-search.com)

http://www.planethosting.com

polybot 1.0 (http://cis.poly.edu/polybot/)
Pompos/1.1 http://pompos.iliad.fr
Pompos/1.2 http://pompos.iliad.fr
Pompos/1.3 http://dir.com/pompos.html
Portal Manager 0.7
potbot 1.0
ProWebGuide Link Checker (http://www.prowebguide.com)
Program Shareware 1.0.3
psbot/0.1 (+http://www.picsearch.com/bot.html)
pverify/1.2
QPCreep Test Rig ( We are not indexing, just testing )
QuepasaCreep v0.9.14
QuepasaCreep ( crawler@quepasacorp.com )
QuepasaCreep v0.9.13
RPT-HTTPClient/0.3-3
reifier.org (admin@reifier.org)
reifier.org admin@reifier.org
rico/0.1
RixBot (http://www.oops-as.no/rix/)
RoboPal (http://www.findpal.com/)
RobotMidareru/0.7libwww-perl/5.65
Search Engine World Robots.txt Validator at http://www.searchengineworld.com/cgi-bin/robotcheck.cgi
Robozilla/1.0
Mozilla/5.0 (compatible; SYCLIKControl/LinkChecker;)
SafariBookmarkChecker/1.25 (+http://www.coriolis.ch/)
SafariBookmarkChecker/1.26 (+http://www.coriolis.ch/)
Scooter/1.0
Scooter-ARS-1.1
Scooter-3.0.FS – Altavista.com
Scooter-3.2
Scooter-3.2.BT
Scooter-3.2.EX
Scooter-3.2.FNR
Scooter-3.2.PDF
Scooter-3.2.SF0
Scooter-3.2.TX.FNR
Scooter-3.2.XX0
Scooter/3.2
Scooter/3.2.SF0
Scooter_x0-3.2.EX
Scooter/3.3
Scooter/3.3.QA
Scooter/3.3.QA.pczukor
Scooter/3.3.vscooter
Scooter/3.3_SF
Scrubby/2.1 (http://www.scrubtheweb.com/abs/meta-check.html)
Scrubby/2.2 (http://www.scrubtheweb.com/)
Search Agent 1.0
SearchSpider.com/1.1
Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/0.3
Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.0 (XDF)
semanticdiscovery/0.1
Sensis.com.au Web Crawler (search_commentsatsensisdotcomdotau)
sherlock/1.3 httpget/1.3
sherlock_spider (jimfan@163.com)
SiteXpert
InternetSeer.com
sitecheck.internetseer.com (For more info see: http://sitecheck.internetseer.com)
sitescooper/3.1.2 (http://sitescooper.org) libwww-perl/5.51
SlySearch/1.3 (http://www.slysearch.com)
SlySearch/1.3 http://www.slysearch.com
sohu-search
Speedy Spider (http://www.entireweb.com)
Mozilla/4.0 (compatible; SpeedySpider; www.entireweb.com)
Speedy_Spider_(http://www.entireweb.com)
SpiderKU/0.9
SpiderMonkey/7.04 (SpiderMonkey.ca info at http://spidermonkey.ca/sm.shtml)
Spider_Monkey/7.06 (SpiderMonkey.ca info at http://SpiderMonkey.ca /sm.shtml)
Spider_Monkey/7.06 (SpiderMonkey.ca info at http://www.spidermonkey.ca/sm.shtml)
Mozilla/5.0 (compatible; SpurlBot/0.2)
Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux-gnu)
Star Downloader
Steeler/1.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
Mozilla/4.0 (compatible; SuperCleaner 2.56; Windows NT 5.1)
Szukacz/1.5
Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html; info@szukacz.pl)
Tarantula Experimental Crawler
Tcl http client package 1.0
Tcl http client package 2.3
(Teradex Mapper; mapper@teradex.com; http://www.teradex.com)
Teradex_Crawler (crawler@teradex.com; http://crawler.teradex.com)
TheSuBot/0.1 (www.thesubot.de)
thesubot-beta-www.thesubot.de
thumbshots-de-Bot (Version: 1.02, powered by www.thumbshots.de)
timboBot/0.9 http://www.breakingblogs.com/timbo_bot.html
Tkensaku/0.9 (http://www.tkensaku.com/q.html)
TranSGeniKBot (http://www.tsgk.net)
TranSGeniKBot http://www.tsgk.net
TulipChain/5.7 (http://ostermiller.org/tulipchain/) Java/1.4.0_02 (http://java.sun.com/) Windows_Me/4.90
TulipChain/5.94 (http://ostermiller.org/tulipchain/) Java/1.4.1_01 (http://apple.com/) Mac_OS_X/10.2.8
TulipChain/6.01 (http://ostermiller.org/tulipchain/) Java/1.4.2_03 (http://java.sun.com/) Windows_XP/5.1 RPT-HTTPClient/0.3-3
TulipChain/6.02 (http://ostermiller.org/tulipchain/) Java/1.4.2_03 (http://apple.com/) Mac_OS_X/10.3.3 RPT-HTTPClient/0.3-3
TulipChain/6.03 (http://ostermiller.org/tulipchain/) Java/1.4.2_05 (http://java.sun.com/) Windows_XP/5.1 RPT-HTTPClient/0.3-3
TurnitinBot/1.4 (http://www.turnitin.com/robot/crawlerinfo.html)
TurnitinBot/1.4 http://www.turnitin.com/robot/crawlerinfo.html
TurnitinBot/1.5 (http://www.turnitin.com/robot/crawlerinfo.html)
TurnitinBot/1.5 http://www.turnitin.com/robot/crawlerinfo.html
TurnitinBot/2.0 (http://www.turnitin.com/robot/crawlerinfo.html)
TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html
TutorGigBot/1.5 ( +http://www.tutorgig.info )
Tutorial Crawler 1.4 (http://www.tutorgig.com/crawler)
UIowaCrawler/1.0
UIowaCrawler/2.0
USyd-NLP-Spider (http://www.it.usyd.edu.au/~vinci/bot.html)
UdmSearch/3.1.20
unchaos_crawler_2.0.2 (search.engine@unchaos.com)
updated/0.1beta (updated.com; http://www.updated.com; crawler@updated.om)
VSE/1.0 (vsecrawler@hotmail.com)
Vagabondo/2.0 MT (webagent at wise-guys dot nl)
Vagabondo/2.0 MT (webagent@NOSPAMwise-guys.nl)
Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)
Vivante Link Checker (http://www.vivante.com)
void-bot/0.1 (bot@void.be; http://www.void.be/)
Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot; 1.6
Mozilla/4.0_(compatible;_MSIE_5.0;_Windows_95)_VoilaBot/1.6 libwww/5.3.2
vspider
W3C-checklink/2.90 libwww-perl/5.64
W3C-checklink/3.6.2.3 libwww-perl/5.64
W3C-checklink/3.9.2 [3.17] libwww-perl/5.79
W3C-checklink/4.0 [4.4] libwww-perl/5.800
W3C-checklink/4.1 [4.14] libwww-perl/5.800
W3C_Validator/1.183 libwww-perl/5.64
W3C_Validator/1.305.2.109 libwww-perl/5.79
W3C_Validator/1.305.2.12 libwww-perl/5.64
W3C_Validator/1.305.2.137 libwww-perl/5.79
W3C_Validator/1.305.2.148 libwww-perl/5.800
W3C_Validator/1.305.2.148 libwww-perl/5.803
WWWeasel Robot v1.00 (http://wwweasel.de)
WebFilter Robot 1.0
WebRACE/1.1 (University of Cyprus, Distributed Crawler)
WebSauger 1.20b
WebSearch/2.0.1 (Dez@Blanchfield.COM.AU, http://www.WebSearch.com.au/)
WebSearch.COM.AU/3.0.1 (The Australian Search Engine; http://WebSearch.COM.AU; Search@WebSearch.COM.AU)
http://www.WebSearch.com.au/ – Australian Search Engine/3.1.3 (sites@websearch.com.au)
http://www.WebSearch.com.au/ – Australian Search Engine/3.1.6 (sites@websearch.com.au)
http://www.WebSearch.com.au/ (larbin2.6.2@unspecified.mail)
http://www.WebSearch.com.au/ larbin2.6.2@unspecified.mail
http://www.websearch.com.au (larbin2.6.2@unspecified.mail)
http://www.websearch.com.au larbin2.6.2@unspecified.mail
www.WebSearch.com.au (search@websearch.com.au)
www.WebSearch.com.au search@websearch.com.au
webbot
Webclipping.com
webcollage/1.102
webcollage/1.104
webcollage/1.87
webcollage/1.93
webcollage/1.94
Fri Nov 15 04:51:18 EST 2002WebcraftBoot Java/1.4.1_01
Sun Apr 20 22:00:01 EDT 2003WebcraftBoot Java/1.4.2-beta
Tue Apr 15 22:00:03 EDT 2003WebcraftBoot Java/1.4.2-beta
Thu Mar 27 18:20:34 CET 2003WebcraftBoot
Mozilla/3.0 (compatible; Webinator-indexer.cyberalert.com/2.56)
www.webwombat.com.au
webyield robot (http://www.webyield.net/search/search.pl)
Wget/1.5.2
Wget/1.5.3
Wget/1.5.3.1
Wget/1.6
Wget/1.7
Wget/1.8
Wget/1.8.1
Wget/1.8.1+cvs
Wget/1.8.2
Wget/1.9
Wget/1.9-beta
Wget/1.9.1
Willow Internet Crawler by Twotrees V2.1
Wotbox/alpha0.5.1 (bot@wotbox.com; http://www.wotbox.com) Java/1.4.1_02

http://www.ciml.co.uk

Xenu’s Link Sleuth 1.1a
Xenu Link Sleuth 1.2b
Xenu Link Sleuth 1.2d
Xenu Link Sleuth 1.2e
Xenu Link Sleuth 1.2f
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com)
YottaCars_Bot/4.12 (+http://www.yottacars.com) Car Search Engine
Zao/0.1 (http://www.kototoi.org/zao/)
Zao/0.2 (http://www.kototoi.org/zao/)
Zao-Crawler
Zeus 3140 Webster Pro V2.9 Win32
Zeus 57657 Webster Pro V2.9 Win32
ZipppBot/0.11 (ZipppBot; http://www.zippp.net; webmaster@zippp.net)
ZoomSpider – wrensoft.com
Mozilla/4.0 compatible ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com)
Mozilla/4.0 compatible ZyBorg/1.0 (wn-1.zyborg@looksmart.net; http://www.WISEnutbot.com)
Mozilla/4.0 compatible ZyBorg/1.0 (wn-12.zyborg@looksmart.net; http://www.WISEnutbot.com)
Mozilla/4.0 compatible ZyBorg/1.0 (wn-2.zyborg@looksmart.net; http://www.WISEnutbot.com)
Mozilla/4.0 compatible ZyBorg/1.0 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)
Mozilla/4.0 compatible ZyBorg/1.0 DLC (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)
Mozilla/4.0 compatible ZyBorg/1.0 Daily Refresh Beta-d03 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)
Mozilla/4.0 compatible ZyBorg/1.0 Daily Refresh Beta-d05 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)
Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.dlc@looksmart.net; http://www.WISEnutbot.com)
Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)
Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker Beta-d01 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)
AnsearchBot
AnyBrowser.com Search Engine
LeechGet 2002 (www.leechget.de)
LeechGet 2004 (www.leechget.net)
NationalDirectory-WebSpider/1.3
arianna.libero.it Linux/2.4.9-34smp (linux)