// archives

agent

This tag is associated with 2 posts

Detect bots via user agent string

By ⋅ 20. Januar 2009 ⋅ Post a comment

Hi guys, When making full-flash websites, it’s quite effective to deliver html-content to search engines. To determine whether a visitor is a robot or not, you have to match the visitor’s user agent against a list of known bot user agents. I parsed a bot user agent list out of this table: http://www.pgts.com.au/pgtsj/pgtsj0208d.html You can [...]

Blocking web crawlers on lighttpd

By ⋅ 19. September 2008 ⋅ 5 comments

Nutch did ignore my robots.txt (for whatever reason, I was unable to figure out why), so I had to find another way to forbid those directories for the crawler. I finally came up with this neat piece of config for lighty: - throws an HTTP 403 when matching our defined User Agent and URL.

—

Letzte Posts

Extbase Plugin Settings in Backend Modul auslesen
Januar 13, 2012
By Alex
Extbase Backend Modul als Hauptmodul integrieren
Januar 12, 2012
By Alex
Extbase Plugins mit TypoScript einbinden
Juli 1, 2011
By Alex
Typoscript XML Sitemap
Juni 1, 2010
By Alex
TYPO3 Extension: ad_rtepasteplain
Februar 6, 2010
By Alex