You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Matt Zytaruk <ma...@wavefire.com> on 2005/11/22 19:42:51 UTC
Fetcher url sorting
Hello all, just have a question on the fetcher in 0.8 dev.
A big problem I've been noticing with fetching is the number of max
delays exceeded errors. I recently did a fetch of about 600,000 pages
and 1 in 6 of those was never downloaded because of max delays exceeded.
This seems like an excessive amount, so I decided to look into the code
to see if I could find out how the url's are sorted for fetching, so
that maybe I could improve this to spread out the attempts to fetch from
each host. I saw that the urllist is partitioned by host at one point,
but I gather that is just so that all the urls from one host are sent to
the same server for fetching, to ensure politeness? I couldn't track
down any other sorting. Am I missing something here? Is there any other
way that the fetcher could be improved so that I don't get so many max
delays exceeded errors?
Thanks in advance.
-Matt Zytaruk
Re: Fetcher url sorting
Posted by Matt Zytaruk <ma...@wavefire.com>.
Thanks for the the help, Doug.
-Matt Zytaruk
Doug Cutting wrote:
> Matt Zytaruk wrote:
>
>> Well, if we want to fetch pages from N different sites, ideally we
>> should be able to have N threads running, without any of them having
>> to wait. I guess ideally what the fetcher should probably do is
>> instead of waiting, put the url it was trying to fetch back into the
>> queue to be tried later on, and grab a different one.
>
>
> The fetcher used to do this, and it ended up with huge queues. We
> capped the size of the queues, and dropped urls when their queue was
> full. But the fetcher still spent an age at the end, mostly idle,
> with a single thread emptying its queue. And there were some bugs in
> the queue synchronization that caused things to sometimes hang, but no
> one could ever figure out why.
>
> So the current fetcher's strategy is to, instead of queuing urls in
> order to drop them later, drop them now. And instead of queuing urls
> in order to wait later, wait now. It makes things a lot simpler. In
> the end the performance is similar, but you can see the cost of
> crawling big sites immediately, rather than only later. In either
> case you need to choose to drop things or run slowly.
>
>> I'm not so sure that accesses to each host are spread evenly
>> throughout the list, because the fetch list I was doing had tens of
>> thousands of different hosts and I was still getting a large amount
>> of threads trying to access the same host at the same time, even with
>> only 50 threads. Although maybe I'm wrong and that is how it would
>> act if the hosts were spread evenly throughout, I'm not sure, it just
>> seems like a lot.
>
>
> They're not spread exactly evenly, but randomly, which can be a bit
> lumpy. What percentage of urls in the fetch list are from a host that
> is exceeding max delays? If it is near 2%, or that host is slower
> than average, then you'll probably have issues with 50 threads.
>
> Doug
>
>
Re: Fetcher url sorting
Posted by Doug Cutting <cu...@nutch.org>.
Matt Zytaruk wrote:
> Well, if we want to fetch pages from N different sites, ideally we
> should be able to have N threads running, without any of them having to
> wait. I guess ideally what the fetcher should probably do is instead of
> waiting, put the url it was trying to fetch back into the queue to be
> tried later on, and grab a different one.
The fetcher used to do this, and it ended up with huge queues. We
capped the size of the queues, and dropped urls when their queue was
full. But the fetcher still spent an age at the end, mostly idle, with
a single thread emptying its queue. And there were some bugs in the
queue synchronization that caused things to sometimes hang, but no one
could ever figure out why.
So the current fetcher's strategy is to, instead of queuing urls in
order to drop them later, drop them now. And instead of queuing urls in
order to wait later, wait now. It makes things a lot simpler. In the
end the performance is similar, but you can see the cost of crawling big
sites immediately, rather than only later. In either case you need to
choose to drop things or run slowly.
> I'm not so sure that accesses to each host are spread evenly throughout
> the list, because the fetch list I was doing had tens of thousands of
> different hosts and I was still getting a large amount of threads trying
> to access the same host at the same time, even with only 50 threads.
> Although maybe I'm wrong and that is how it would act if the hosts were
> spread evenly throughout, I'm not sure, it just seems like a lot.
They're not spread exactly evenly, but randomly, which can be a bit
lumpy. What percentage of urls in the fetch list are from a host that
is exceeding max delays? If it is near 2%, or that host is slower than
average, then you'll probably have issues with 50 threads.
Doug
Re: Fetcher url sorting
Posted by Matt Zytaruk <ma...@wavefire.com>.
Well, if we want to fetch pages from N different sites, ideally we
should be able to have N threads running, without any of them having to
wait. I guess ideally what the fetcher should probably do is instead of
waiting, put the url it was trying to fetch back into the queue to be
tried later on, and grab a different one.
I'm not so sure that accesses to each host are spread evenly throughout
the list, because the fetch list I was doing had tens of thousands of
different hosts and I was still getting a large amount of threads trying
to access the same host at the same time, even with only 50 threads.
Although maybe I'm wrong and that is how it would act if the hosts were
spread evenly throughout, I'm not sure, it just seems like a lot.
Thanks for your help.
-Matt
Doug Cutting wrote:
> Matt Zytaruk wrote:
>
>> Indeed, that does work, although that ends up slowing down the fetch
>> a fair amount because a lot of threads end up idle, waiting, and I
>> was hoping to avoid that slowdown if possible.
>
>
> What should these threads be doing?
>
> If you have a site with N pages to fetch, and you want to fetch them
> all politely, then it will take at least fetcher.server.delay*N to
> fetch them all. The fetch list is sorted by the hash of the url, so
> accesses to each host should be spread fairly evenly through the list.
>
> Capping the number of pages per host (generate.max.per.host) will
> help, or, if you know the webmasters in question, you can consider
> increasing fetcher.threads.per.host.
>
> Doug
>
>
Re: Fetcher url sorting
Posted by Doug Cutting <cu...@nutch.org>.
Matt Zytaruk wrote:
> Indeed, that does work, although that ends up slowing down the fetch a
> fair amount because a lot of threads end up idle, waiting, and I was
> hoping to avoid that slowdown if possible.
What should these threads be doing?
If you have a site with N pages to fetch, and you want to fetch them all
politely, then it will take at least fetcher.server.delay*N to fetch
them all. The fetch list is sorted by the hash of the url, so accesses
to each host should be spread fairly evenly through the list.
Capping the number of pages per host (generate.max.per.host) will help,
or, if you know the webmasters in question, you can consider increasing
fetcher.threads.per.host.
Doug
Re: Fetcher url sorting
Posted by Matt Zytaruk <ma...@wavefire.com>.
Indeed, that does work, although that ends up slowing down the fetch a
fair amount because a lot of threads end up idle, waiting, and I was
hoping to avoid that slowdown if possible.
-Matt
Doug Cutting wrote:
> Please try simply increasing http.max.delays, perhaps to 100 or more.
>
> Probably we should increase the default, since with partitioned fetch
> lists this is much more common.
>
> Doug
>
> Matt Zytaruk wrote:
>
>> Hello all, just have a question on the fetcher in 0.8 dev.
>>
>> A big problem I've been noticing with fetching is the number of max
>> delays exceeded errors. I recently did a fetch of about 600,000 pages
>> and 1 in 6 of those was never downloaded because of max delays
>> exceeded. This seems like an excessive amount, so I decided to look
>> into the code to see if I could find out how the url's are sorted for
>> fetching, so that maybe I could improve this to spread out the
>> attempts to fetch from each host. I saw that the urllist is
>> partitioned by host at one point, but I gather that is just so that
>> all the urls from one host are sent to the same server for fetching,
>> to ensure politeness? I couldn't track down any other sorting. Am I
>> missing something here? Is there any other way that the fetcher could
>> be improved so that I don't get so many max delays exceeded errors?
>>
>> Thanks in advance.
>>
>> -Matt Zytaruk
>
>
>
Re: Fetcher url sorting
Posted by Doug Cutting <cu...@nutch.org>.
Please try simply increasing http.max.delays, perhaps to 100 or more.
Probably we should increase the default, since with partitioned fetch
lists this is much more common.
Doug
Matt Zytaruk wrote:
> Hello all, just have a question on the fetcher in 0.8 dev.
>
> A big problem I've been noticing with fetching is the number of max
> delays exceeded errors. I recently did a fetch of about 600,000 pages
> and 1 in 6 of those was never downloaded because of max delays exceeded.
> This seems like an excessive amount, so I decided to look into the code
> to see if I could find out how the url's are sorted for fetching, so
> that maybe I could improve this to spread out the attempts to fetch from
> each host. I saw that the urllist is partitioned by host at one point,
> but I gather that is just so that all the urls from one host are sent to
> the same server for fetching, to ensure politeness? I couldn't track
> down any other sorting. Am I missing something here? Is there any other
> way that the fetcher could be improved so that I don't get so many max
> delays exceeded errors?
>
> Thanks in advance.
>
> -Matt Zytaruk