Re: .8 svn - fetcher performance..

[TDLN <di...@gmail.com>]

+1 for a solution to this pressing issue!

I am seeing the same problem, in my case two symptoms:

1) low fetch speeds
2) crawls end "before their time" with "aborting with xxx hung
threads" error message

I am doing a focussed crawl on about 70.000 domains.
crawl.ignore.external.links is set to true.

In previous discussions on the list these issues have mainly been
attributed to crawls on such a limited set of domains.

See if I understand this correcly. FetchLists are hostwise disjoint,
thus all urls from the same domain are in the same FetchList. Folks
*not* on MapReduce are by definition always working with one Fetcher.
Otherwise could be many, in which case this mechanism prevents the
politeness rules from being disobeyed.

Could somebody confirm these assumptions are correct?

I have tried to work around the issues by changing the configuration.
I tried increasing fetcher.threads.fetch, http.timeout and
http.max.delays.

I also changed generate.max.per.host setting, following Doug's advice
of setting this value to TopN / Fetcher Threads, all to no lasting
avail.

So far, I haven't tried increasing the fetcher.threads.per.host to
more than 4 with 100 threads, though. I will do that now.

I really think we should gather some more data regarding fetch speed
problems. Maybe some of you who are seeing decent fetch speeds in a
focussed crawl setup could share some of their tips in tuning the
installation.

Thanks a lot for your time if you read so far :)

Rgrds, Thomas Delnoij




On 6/28/06, Sami Siren <ss...@gmail.com> wrote:
> Ken Krugler wrote:
>
> > Hi Doug,
> >
> >> Did you ever resolve your 0.8 vs 0.7 crawling performance question? I'm
> >> running into a similar problem.
> >
> >
> > We wound up dramatically increasing the number of threads, which
> > seemed to help solve the bandwidth utilization problem. With Nutch 0.7
> > we were running about 200 threads per crawler, and with Nutch 0.8 it's
> > more like 2000+ threads...though you have to reduce the thread stack
> > size in this type of configuration.
> >
> Fetchlist seems to be sorted by url.This leads to many threads being
> blocked when crawler is configured to fetch by a low number of threads
> per host (default 1) and there are several urls from same host in the
> fetchlist.
>
> This could perhaps be improved by sorting by some other key?
>
> --
>  Sami Siren
>
>
>
>

Re: deleting URL duplicates - never actually deleted?

[Marko Bauhardt <mb...@media-style.com>]

>
> # De-duplicate indexes
> # "bogus" argument is ignored but needed due to
> # a bug in the number of args expected
> bin/nutch dedup crawl/segments bogus
>

The dedup command works only on many indexes and not on one or many  
segments. The directory structure of an index looks like:
index/part-00000/SOME_LUCENE_FILES

Here is an example how is the structure of an crawl:
crawl/segments/20060702232437
crawl/segments/20060702233040
crawl/linkdb
crawl/indexes //this is the index of the two segments

Now you can run dedup: bin/nutch dedup crawl/indexes

If you run dedup on a folder which contains segments, an exception  
should be thrown. Look at your logfiles and verify that the dedup  
process runs whithout exeptions.

Marko

Re: deleting URL duplicates - never actually deleted?

[Honda-Search Administrator <ad...@honda-search.com>]

Marko,

Currently the shell command is as follows:

---
# index new segment
bin/nutch index $s1

# update the database
bin/nutch updatedb crawl/db $s1

# De-duplicate indexes
# "bogus" argument is ignored but needed due to
# a bug in the number of args expected
bin/nutch dedup crawl/segments bogus

# Merge indexes
ls -d crawl/segments/* | xargs bin/nutch merge crawl/index
---

Should I actually switch the last two commands around?

Matt 

Original Message ----- 
From: "Marko Bauhardt" <mb...@media-style.com>
To: <nu...@lucene.apache.org>
Sent: Friday, June 30, 2006 2:57 AM
Subject: Re: deleting URL duplicates - never actually deleted?


> 
>  Do you delete the duplicates before you merge the index? Run first  
> the merge command and then the dedup command.
> 
> But a better way is you create one index of all segments with the  
> index command and then runs the dedup command of this one index.
> 
> Hope this Helps,
> Marko
> 
> 
> Am 29.06.2006 um 23:07 schrieb Honda-Search Administrator:
> 
>> Maybe someone can explain to me how this works.
>>
>> First, my setup.
>>
>> I create a fetchlist each night with FreeFetchlistTool and fetch  
>> those pages.  It often contains the same URLS that are already in  
>> the database, but this tool gets the newest copies of those URLs.
>>
>> I also run nutch dedup after everything is fetched, indexed, etc.   
>> I then merge the segments using the following command:
>>
>> ls -d $segments_dir/* | xargs bin/nutch merge $index_dir
>>
>> Every night the number of "duplicates" increases.  THis is so  
>> because the duplicates from the day before are not actually deleted  
>> (I assume).
>>
>> Is dedup removing them from some sort of master index and the  
>> segments retain their original information?
>>
>> If so, is there a way to merge the segments into one (or whatever)  
>> so that duplicate URLs do not exist?  Would mergesegs do this?
>>
>> Thanks for any help, and I hope my questionis clear.
>>
>> Matt
>>
>>
> 
> 
>

Re: deleting URL duplicates - never actually deleted?

[Marko Bauhardt <mb...@media-style.com>]

  Do you delete the duplicates before you merge the index? Run first  
the merge command and then the dedup command.

But a better way is you create one index of all segments with the  
index command and then runs the dedup command of this one index.

Hope this Helps,
Marko


Am 29.06.2006 um 23:07 schrieb Honda-Search Administrator:

> Maybe someone can explain to me how this works.
>
> First, my setup.
>
> I create a fetchlist each night with FreeFetchlistTool and fetch  
> those pages.  It often contains the same URLS that are already in  
> the database, but this tool gets the newest copies of those URLs.
>
> I also run nutch dedup after everything is fetched, indexed, etc.   
> I then merge the segments using the following command:
>
> ls -d $segments_dir/* | xargs bin/nutch merge $index_dir
>
> Every night the number of "duplicates" increases.  THis is so  
> because the duplicates from the day before are not actually deleted  
> (I assume).
>
> Is dedup removing them from some sort of master index and the  
> segments retain their original information?
>
> If so, is there a way to merge the segments into one (or whatever)  
> so that duplicate URLs do not exist?  Would mergesegs do this?
>
> Thanks for any help, and I hope my questionis clear.
>
> Matt
>
>

deleting URL duplicates - never actually deleted?

[Honda-Search Administrator <ad...@honda-search.com>]

Maybe someone can explain to me how this works.

First, my setup.

I create a fetchlist each night with FreeFetchlistTool and fetch those 
pages.  It often contains the same URLS that are already in the database, 
but this tool gets the newest copies of those URLs.

I also run nutch dedup after everything is fetched, indexed, etc.  I then 
merge the segments using the following command:

ls -d $segments_dir/* | xargs bin/nutch merge $index_dir

Every night the number of "duplicates" increases.  THis is so because the 
duplicates from the day before are not actually deleted (I assume).

Is dedup removing them from some sort of master index and the segments 
retain their original information?

If so, is there a way to merge the segments into one (or whatever) so that 
duplicate URLs do not exist?  Would mergesegs do this?

Thanks for any help, and I hope my questionis clear.

Matt

Re: .8 svn - fetcher performance..

[TDLN <di...@gmail.com>]

> Ok, this isn't true because it is sorted by using HashComparator. For
> some reason
> generated list contains some parts wich are more or less sorted by host
> and some
> parts looks more "random".

This is consistent with what I am seeing; the Fetcher slowing down for
a while, sometimes coming to a virtual halt (a lot of repeated fetch
speed statements, for instance "412052 pages, 95915 errors, 3.0
pages/s, 638 kb/s,"), then the Fetcher speeding up again and fetching
at acceptable fetch speeds.

Rgrds, Thomas

Re: .8 svn - fetcher performance..

[Sami Siren <ss...@gmail.com>]

>>
>> Fetchlist seems to be sorted by url.This leads to many threads being
>
Ok, this isn't true because it is sorted by using HashComparator. For 
some reason
generated list contains some parts wich are more or less sorted by host 
and some
parts looks more "random".

--
 Sami Siren