You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "yoursoft@freemail.hu" <yo...@freemail.hu> on 2005/04/19 12:25:18 UTC

NUTCH-7 - analyze tool takes up all the disk space when there are circular links

Dear Guys,

I have the same problem with latest nutch from svn. The reported 
solution doesn't work for me. I would like to fix this problem.
Have anyone to any idea how to start with it?

Best Regards,
    Ferenc

Re: [Nutch-dev] Re: NUTCH-7 - analyze tool takes up all the disk space when there are circular links

Posted by Massimo Miccoli <mm...@iltrovatore.it>.
Thanks Doug,
I will try. But no other way to detect crawler loop? How we can discover 
similar case with billion of pages?

Thanks
Massimo

Doug Cutting wrote:

> Massimo Miccoli wrote:
>
>> The general problem is urls like: 
>> http://www.agriturismo.pg.it/storia-citta-umbria/index.html
>> a custom not found pages that generate infinite crawler loop on site.  
>
>
> You're referring to error pages that do not return 404?
>
> In another thread I just suggested a way to handle these:
>
> http://www.mail-archive.com/nutch-user%40incubator.apache.org/msg00286.html 
>
>
> The url you mention is amenable to this solution.  It's title contains 
> the string "pagina di errore", but it does not return a 404.
>
> Doug
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: New Crystal Reports XI.
> Version 11 adds new functionality designed to reduce time involved in
> creating, integrating, and deploying reporting solutions. Free runtime 
> info,
> new features, or free trial, at: http://www.businessobjects.com/devxi/728
> _______________________________________________
> Nutch-developers mailing list
> Nutch-developers@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nutch-developers
>

Re: [Nutch-dev] Re: NUTCH-7 - analyze tool takes up all the disk space when there are circular links

Posted by Doug Cutting <cu...@nutch.org>.
Massimo Miccoli wrote:
> The general problem is urls like: 
> http://www.agriturismo.pg.it/storia-citta-umbria/index.html
> a custom not found pages that generate infinite crawler loop on site.  

You're referring to error pages that do not return 404?

In another thread I just suggested a way to handle these:

http://www.mail-archive.com/nutch-user%40incubator.apache.org/msg00286.html

The url you mention is amenable to this solution.  It's title contains 
the string "pagina di errore", but it does not return a 404.

Doug

Re: [Nutch-dev] Re: NUTCH-7 - analyze tool takes up all the disk space when there are circular links

Posted by Massimo Miccoli <mm...@iltrovatore.it>.
The general problem is urls like: 
http://www.agriturismo.pg.it/storia-citta-umbria/index.html
a custom not found pages that generate infinite crawler loop on site.  
It's not a rare casse, whe one (like me) try to fetch
whole web.
BTW if you wanto you can test a Nutch search on 50.000.000 pages (not 
urls) at http://crawlers.iltrovatore.it:8088/search.jsp

massimo



Doug Cutting wrote:

> Massimo Miccoli wrote:
>
>> In any way the circular links is a big problem for Nutch. Not only 
>> for analyze tool, but also for fetcher speed and Wedb size.  Any 
>> solution?
>
>
> What is the general problem with Nutch's handling of circular links? 
> Nearly every site has them.  I am able to crawl, index and search 
> sites with circular links.
>
> Doug
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: New Crystal Reports XI.
> Version 11 adds new functionality designed to reduce time involved in
> creating, integrating, and deploying reporting solutions. Free runtime 
> info,
> new features, or free trial, at: http://www.businessobjects.com/devxi/728
> _______________________________________________
> Nutch-developers mailing list
> Nutch-developers@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nutch-developers
>

Re: [Nutch-dev] Re: NUTCH-7 - analyze tool takes up all the disk space when there are circular links

Posted by Doug Cutting <cu...@nutch.org>.
Massimo Miccoli wrote:
> In any way the circular links is a big problem for Nutch. Not only for 
> analyze tool, but also for fetcher speed and Wedb size.  Any solution?

What is the general problem with Nutch's handling of circular links? 
Nearly every site has them.  I am able to crawl, index and search sites 
with circular links.

Doug

Re: [Nutch-dev] Re: NUTCH-7 - analyze tool takes up all the disk space when there are circular links

Posted by Massimo Miccoli <mm...@iltrovatore.it>.
In any way the circular links is a big problem for Nutch. Not only for 
analyze tool, but also for fetcher speed and Wedb size.  Any solution?

Thx Massimo

Doug Cutting wrote:

> yoursoft@freemail.hu wrote:
>
>> I have the same problem with latest nutch from svn. The reported 
>> solution doesn't work for me. I would like to fix this problem.
>> Have anyone to any idea how to start with it?
>
>
> Sorry, but I have not used the analyze tool in a while.  It is not 
> actively maintained.
>
> As a workaround, I recommend setting both 
> fetchlist.score.by.link.count and indexer.boost.by.link.count to 
> true.  This works well, provided you're not encountering a lot of link 
> spam.
>
> Doug
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: New Crystal Reports XI.
> Version 11 adds new functionality designed to reduce time involved in
> creating, integrating, and deploying reporting solutions. Free runtime 
> info,
> new features, or free trial, at: http://www.businessobjects.com/devxi/728
> _______________________________________________
> Nutch-developers mailing list
> Nutch-developers@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nutch-developers
>

Re: [Nutch-dev] Re: NUTCH-7 - analyze tool takes up all the disk space when there are circular links

Posted by Doug Cutting <cu...@nutch.org>.
yoursoft@freemail.hu wrote:
> In my config both parameter are true, and I have problem with www.nb1.hu/*.

Sorry if I wasn't clear.  Once you set these parameters, you should no 
longer use the 'analyze' command at all.

Doug

Re: [Nutch-dev] Re: NUTCH-7 - analyze tool takes up all the disk space when there are circular links

Posted by "yoursoft@freemail.hu" <yo...@freemail.hu>.
In my config both parameter are true, and I have problem with www.nb1.hu/*.
You will replace the analysis with map-reduce?

Doug Cutting wrotte:

> yoursoft@freemail.hu wrote:
>
>> I have the same problem with latest nutch from svn. The reported 
>> solution doesn't work for me. I would like to fix this problem.
>> Have anyone to any idea how to start with it?
>
>
> Sorry, but I have not used the analyze tool in a while.  It is not 
> actively maintained.
>
> As a workaround, I recommend setting both 
> fetchlist.score.by.link.count and indexer.boost.by.link.count to 
> true.  This works well, provided you're not encountering a lot of link 
> spam.
>
> Doug
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: New Crystal Reports XI.
> Version 11 adds new functionality designed to reduce time involved in
> creating, integrating, and deploying reporting solutions. Free runtime 
> info,
> new features, or free trial, at: http://www.businessobjects.com/devxi/728
> _______________________________________________
> Nutch-developers mailing list
> Nutch-developers@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nutch-developers
>
>


Re: NUTCH-7 - analyze tool takes up all the disk space when there are circular links

Posted by Doug Cutting <cu...@nutch.org>.
yoursoft@freemail.hu wrote:
> I have the same problem with latest nutch from svn. The reported 
> solution doesn't work for me. I would like to fix this problem.
> Have anyone to any idea how to start with it?

Sorry, but I have not used the analyze tool in a while.  It is not 
actively maintained.

As a workaround, I recommend setting both fetchlist.score.by.link.count 
and indexer.boost.by.link.count to true.  This works well, provided 
you're not encountering a lot of link spam.

Doug