Posted on:
19. September 2008
Blocking web crawlers on lighttpd
Note: The information contained in this post may be outdated! […]
Note: The information contained in this post may be outdated! […]
In order to get our meta descriptions displayed in the results we need to write a plugin that extends 2 different extension points.
I thought an ideal solution would be telling nutch to ignore specific sections. A good and common practice doing this kind of stuff is creating HTML comment tags.