You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@roller.apache.org by Anil Gangolli <an...@busybuddha.org> on 2006/05/22 16:34:11 UTC
Re: Spider trap in Roller's calendar
I've filed an issue based on this.
http://opensource.atlassian.com/projects/roller/browse/ROL-1145
You can watch it if interested. It's best just to file these if you can.
--a.
p.s. A short term workaround might be to add a rel="nofollow" in the macro
that generates the links; this may work for savvy crawlers.
----- Original Message -----
From: "Trygve Lie" <tr...@hotmail.com>
To: <ro...@incubator.apache.org>
Sent: Monday, May 22, 2006 3:55 AM
Subject: Spider trap in Roller's calendar
> Hi
>
> The calendar in Roller can cause a small problem for search engine spiders
> since it's possible to page backwards in dates by the calendar. It's
> actually possible to page pack to the year zero...
> Ex: http://rollerweblogger.org/page/roller/000104
>
> A spider hitting such a "trap" will just continue to page backwards until
> it "gets tired".
>
> There are two dangerous problems with this:
> - This can cause unnecessary stress on the server running Roller (ex;
> Yahoos spider make big slurps and do actually not consider if the server
> can handle it or not).
> - At some point the spider will "get tired" because such paging will
> generate a lot of similar pages (when there are no content all pages will
> be similar) and the spider will then mark the site as "possible spam" due
> to all the similar pages.
>
> I would like to suggest that there might be added a small check which
> makes the backward paging in the calendar only go back to when the first
> post was made in the blog. This would cause the calendar to page back to
> the month when the first post was added to the blog. To be able to page
> beyond that month does not have any actual interest.
>
> Kind regards
> Trygve Lie
>
> _________________________________________________________________
> MSN Spaces http://spaces.msn.com/?mkt=nb-no Vis hvem du er og hva du vil
>