You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Ali Safdar Kureishy <sa...@gmail.com> on 2012/06/12 11:15:59 UTC
How to ensure even distribution of the fetch phase across Hadoop nodes
Hi,
I have a hadoop cluster of 5 nodes. I want to ensure that the fetch phase
is distributed evenly across all the nodes (to maximize bandwidth etc).
However, if I generate a fetchlist of size 1000 urls, does this get
distributed equally across the nodes? Doesn't the fact that the size of the
fetchlist is < 64MB (block size) result in it being fetched from a single
node? If not, how is this distributed across the mappers evenly? Is there a
rough formulate I can use, to determine how many URLs I should fetch for an
equal distribution across my nodes, for a given block size setting?
Thanks,
Safdar
Re: How to ensure even distribution of the fetch phase across Hadoop nodes
Posted by Ali Safdar Kureishy <sa...@gmail.com>.
Thanks Ferdy.
With regards to excluding the reduce phase and your concern about the
sorted-ness of fetch output, if there were more than 1 reducer, there
wouldn't be a total ordering on the fetcher output anyway ... so perhaps
that renders the concern about the sort order moot.
Also, please correct me if I am wrong, but it seems that the jobs that
consume the fetcher output (updatecrawldb, invertlinks, solrindex etc) do
not require the input data to be in sorted order, so a map-only fetch()
might be fine.
Thanks,
Safdar
On Wed, Jun 13, 2012 at 5:35 PM, Ferdy Galema <fe...@kalooga.com>wrote:
> Hi,
>
> You should use set general number of reducers property in your cluster.
> Whenever a job (such a the Fetcher) does not explicitely set the number to
> a custom amount (using job.setNumReduceTasks(...)) it will just use the
> cluster-wide configured amount. Set mapred.reduce.tasks to the preferred
> amount.
>
> Indeed the IndentityReducer is used for the fetcher. Now that I think
> about it, the Fetcher could be redesigned to exclude the reduce phase and
> directly output to the fs. (Not applicable to Nutchgora). This would mean
> the output is not sorted though, and I'm not sure what the consequence of
> that would be. Correct me if I'm wrong.
>
> Ferdy.
>
> On Wed, Jun 13, 2012 at 2:52 PM, Ali Safdar Kureishy <
> safdar.kureishy@gmail.com> wrote:
>
>> Hi Julien/Lewis (and All),
>>
>> Alright, I found out how to achieve an even distribution of URLs for fetch
>> across the nodes of the cluster. It does not have to do with the
>> generate.max.count setting suggested by Julien (though that is important
>> too). The key is to set an appropriate value for the '*numLists*'
>> parameter
>>
>> to the generate() method. This ensures that the fetch-list is broken into
>> many files (blocks) at the end of the generate, which would yield the same
>> number of map tasks for those URLs during fetch, and those get evently
>> distributed across the cluster.
>>
>> So, with a 'numLists' setting of 16 during generate(), and a hadoop
>> cluster
>> of about 5 nodes, I was able to achieve a fetch() throughput of about 400K
>> URLs in 2 hours for the map phase. This was because the fetch was spread
>> across the 5 nodes, using 16 mappers total - and 2 map slots per node.
>> *
>> *
>> *However*, the reduce phase got stuck on *one* node and sat there for more
>>
>> than 1 hour! I feel this might be because there is only one reducer
>> aggregating the output from 16 map tasks. But the code doesn't specify a
>> reducer class for this job. So:
>> *a)* does that mean that the IdentityReducer is being used? The reducer
>>
>> seems to be CPU-bound, since the CPU has been at 100% for most of the time
>> on the single node that's performing the reduce. What is it doing? The
>> FetchOutputFormat class, to which data is finally output just funnels each
>> key+value into an appropriate folder (parse_text, parse_data, content), so
>> should not by itself account for the 100% utilization. Is it the
>> shuffle+sort on 1.4 million map output records causing this? Or something
>> else?
>> *b)* Is there any setting whereby I can increase the number of reducers
>>
>> used during the fetch() job?
>>
>> Thanks in advance!
>>
>> Cheers,
>> Safdar
>>
>>
>> On Tue, Jun 12, 2012 at 11:57 PM, Ali Safdar Kureishy <
>> safdar.kureishy@gmail.com> wrote:
>>
>> > Thanks Lewis and Julien, for your inputs.
>> >
>> > I will look into this a bit further and reply with some numbers, as seen
>> > with a fetchlist of 50K urls. It is late night here in my timezone, so
>> will
>> > look at this first thing in the morning.
>> >
>> > Thanks,
>> > Safdar
>> >
>> >
>> >
>> > On Tue, Jun 12, 2012 at 4:56 PM, Julien Nioche <
>> > lists.digitalpebble@gmail.com> wrote:
>> >
>> >> Guys,
>> >>
>> >> This has to do with the way URLs are grouped for politeness and not so
>> >> much
>> >> with the number of blocks in the input. Limiting the URLs by # host
>> >> names,
>> >> domains or IP is a way of ensuring an even distribution across the
>> >> cluster.
>> >> See nutch-default.xml for details
>> >>
>> >> J.
>> >>
>> >>
>> >> On 12 June 2012 13:06, Lewis John Mcgibbney <lewis.mcgibbney@gmail.com
>> >> >wrote:
>> >>
>> >> > Hi Ali,
>> >>
>> >>
>> >> > Please check out this post [0] I found. I need to agree with the
>> >> > response in the thread ans state that I don't know how Hadoop ensures
>> >> > even distribution of workload but we can assume that by explicitly
>> >> > specifying the mapper and reducers we can ensure that all 'will' be
>> >> > used across your cluster.
>> >> >
>> >> > hth
>> >> >
>> >> > [0] http://stackoverflow.com/questions/5748585/hadoop-workload
>> >> >
>> >> > On Tue, Jun 12, 2012 at 10:15 AM, Ali Safdar Kureishy
>> >> > <sa...@gmail.com> wrote:
>> >> > > Hi,
>> >> > >
>> >> > > I have a hadoop cluster of 5 nodes. I want to ensure that the fetch
>> >> phase
>> >> > > is distributed evenly across all the nodes (to maximize bandwidth
>> >> etc).
>> >> > > However, if I generate a fetchlist of size 1000 urls, does this get
>> >> > > distributed equally across the nodes? Doesn't the fact that the
>> size
>> >> of
>> >> > the
>> >> > > fetchlist is < 64MB (block size) result in it being fetched from a
>> >> single
>> >> > > node? If not, how is this distributed across the mappers evenly? Is
>> >> > there a
>> >> > > rough formulate I can use, to determine how many URLs I should
>> fetch
>> >> for
>> >> > an
>> >> > > equal distribution across my nodes, for a given block size setting?
>> >> > >
>> >> > > Thanks,
>> >> > > Safdar
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Lewis
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> *
>> >> *Open Source Solutions for Text Engineering
>> >>
>> >> http://digitalpebble.blogspot.com/
>> >> http://www.digitalpebble.com
>> >> http://twitter.com/digitalpebble
>> >>
>> >
>> >
>>
>
>
Re: How to ensure even distribution of the fetch phase across Hadoop nodes
Posted by Ferdy Galema <fe...@kalooga.com>.
Hi,
You should use set general number of reducers property in your cluster.
Whenever a job (such a the Fetcher) does not explicitely set the number to
a custom amount (using job.setNumReduceTasks(...)) it will just use the
cluster-wide configured amount. Set mapred.reduce.tasks to the preferred
amount.
Indeed the IndentityReducer is used for the fetcher. Now that I think about
it, the Fetcher could be redesigned to exclude the reduce phase and
directly output to the fs. (Not applicable to Nutchgora). This would mean
the output is not sorted though, and I'm not sure what the consequence of
that would be. Correct me if I'm wrong.
Ferdy.
On Wed, Jun 13, 2012 at 2:52 PM, Ali Safdar Kureishy <
safdar.kureishy@gmail.com> wrote:
> Hi Julien/Lewis (and All),
>
> Alright, I found out how to achieve an even distribution of URLs for fetch
> across the nodes of the cluster. It does not have to do with the
> generate.max.count setting suggested by Julien (though that is important
> too). The key is to set an appropriate value for the '*numLists*' parameter
> to the generate() method. This ensures that the fetch-list is broken into
> many files (blocks) at the end of the generate, which would yield the same
> number of map tasks for those URLs during fetch, and those get evently
> distributed across the cluster.
>
> So, with a 'numLists' setting of 16 during generate(), and a hadoop cluster
> of about 5 nodes, I was able to achieve a fetch() throughput of about 400K
> URLs in 2 hours for the map phase. This was because the fetch was spread
> across the 5 nodes, using 16 mappers total - and 2 map slots per node.
> *
> *
> *However*, the reduce phase got stuck on *one* node and sat there for more
> than 1 hour! I feel this might be because there is only one reducer
> aggregating the output from 16 map tasks. But the code doesn't specify a
> reducer class for this job. So:
> *a)* does that mean that the IdentityReducer is being used? The reducer
> seems to be CPU-bound, since the CPU has been at 100% for most of the time
> on the single node that's performing the reduce. What is it doing? The
> FetchOutputFormat class, to which data is finally output just funnels each
> key+value into an appropriate folder (parse_text, parse_data, content), so
> should not by itself account for the 100% utilization. Is it the
> shuffle+sort on 1.4 million map output records causing this? Or something
> else?
> *b)* Is there any setting whereby I can increase the number of reducers
> used during the fetch() job?
>
> Thanks in advance!
>
> Cheers,
> Safdar
>
>
> On Tue, Jun 12, 2012 at 11:57 PM, Ali Safdar Kureishy <
> safdar.kureishy@gmail.com> wrote:
>
> > Thanks Lewis and Julien, for your inputs.
> >
> > I will look into this a bit further and reply with some numbers, as seen
> > with a fetchlist of 50K urls. It is late night here in my timezone, so
> will
> > look at this first thing in the morning.
> >
> > Thanks,
> > Safdar
> >
> >
> >
> > On Tue, Jun 12, 2012 at 4:56 PM, Julien Nioche <
> > lists.digitalpebble@gmail.com> wrote:
> >
> >> Guys,
> >>
> >> This has to do with the way URLs are grouped for politeness and not so
> >> much
> >> with the number of blocks in the input. Limiting the URLs by # host
> >> names,
> >> domains or IP is a way of ensuring an even distribution across the
> >> cluster.
> >> See nutch-default.xml for details
> >>
> >> J.
> >>
> >>
> >> On 12 June 2012 13:06, Lewis John Mcgibbney <lewis.mcgibbney@gmail.com
> >> >wrote:
> >>
> >> > Hi Ali,
> >>
> >>
> >> > Please check out this post [0] I found. I need to agree with the
> >> > response in the thread ans state that I don't know how Hadoop ensures
> >> > even distribution of workload but we can assume that by explicitly
> >> > specifying the mapper and reducers we can ensure that all 'will' be
> >> > used across your cluster.
> >> >
> >> > hth
> >> >
> >> > [0] http://stackoverflow.com/questions/5748585/hadoop-workload
> >> >
> >> > On Tue, Jun 12, 2012 at 10:15 AM, Ali Safdar Kureishy
> >> > <sa...@gmail.com> wrote:
> >> > > Hi,
> >> > >
> >> > > I have a hadoop cluster of 5 nodes. I want to ensure that the fetch
> >> phase
> >> > > is distributed evenly across all the nodes (to maximize bandwidth
> >> etc).
> >> > > However, if I generate a fetchlist of size 1000 urls, does this get
> >> > > distributed equally across the nodes? Doesn't the fact that the size
> >> of
> >> > the
> >> > > fetchlist is < 64MB (block size) result in it being fetched from a
> >> single
> >> > > node? If not, how is this distributed across the mappers evenly? Is
> >> > there a
> >> > > rough formulate I can use, to determine how many URLs I should fetch
> >> for
> >> > an
> >> > > equal distribution across my nodes, for a given block size setting?
> >> > >
> >> > > Thanks,
> >> > > Safdar
> >> >
> >> >
> >> >
> >> > --
> >> > Lewis
> >> >
> >>
> >>
> >>
> >> --
> >> *
> >> *Open Source Solutions for Text Engineering
> >>
> >> http://digitalpebble.blogspot.com/
> >> http://www.digitalpebble.com
> >> http://twitter.com/digitalpebble
> >>
> >
> >
>
Re: How to ensure even distribution of the fetch phase across Hadoop nodes
Posted by Ali Safdar Kureishy <sa...@gmail.com>.
Hi Julien,
Thanks for the detailed response. My apologies for the confusion with the
generate.max.count parameter. I should have elaborated a bit more on that
earlier but didn't.
Yes, you are right -- that the even distribution of URLs across mappers
also depends on the distribution of URLs per host, which is why I have my
generate.max.count set to 50 -- to ensure that one host does not monopolize
the crawl. So, I was able to get a reasonable distribution of URLs across
the mappers after setting the numFetchers to 16. It didn't occur to me to
mention that additional detail, so thanks for calling it out.
Thanks for the tip about mapred.reduce.tasks. I will try that out shortly.
Regards,
Safdar
On Wed, Jun 13, 2012 at 5:33 PM, Julien Nioche <
lists.digitalpebble@gmail.com> wrote:
>
> Alright, I found out how to achieve an even distribution of URLs for fetch
>> across the nodes of the cluster. It does not have to do with the
>> generate.max.count setting suggested by Julien (though that is important
>> too).
>>
>
>
>> The key is to set an appropriate value for the '*numLists*' parameter
>>
>> to the generate() method. This ensures that the fetch-list is broken into
>> many files (blocks) at the end of the generate, which would yield the same
>> number of map tasks for those URLs during fetch, and those get evently
>> distributed across the cluster.
>>
>
> Ok, let's get a few things right. What you are referring to is called
> [-numFetchers] when using the command nutch generate. It splits the output
> into separate files which are then used as input by the Fetcher - it's not
> about blocks as such. Indeed this means that this will allow the slaves on
> the clusters to fetch from them - I forgot to mention it. However this does
> not guarantee an even distribution of the URLs in the fetchlists, hence my
> comment about generate.count.mode + generate.max.count
>
>
>>
>> So, with a 'numLists' setting of 16 during generate(), and a hadoop
>> cluster
>> of about 5 nodes, I was able to achieve a fetch() throughput of about 400K
>> URLs in 2 hours for the map phase.
>
>
> This was because the fetch was spread
>> across the 5 nodes, using 16 mappers total - and 2 map slots per node.
>>
>
> no. this depends mostly on the distribution of URLS per host/domain/IP in
> your fetchlist. If you have mostly URLs from a single hostname they will be
> dealt with by a single mapper and most other inputs will be empty
> regardless of the number of machines in your cluster. This is how we
> enforce politeness in Nutch.
>
> The results you are getting indicate that there probably is good
> distribution of URLs in your fetch list + increading the number of threads
> in the fetcher would have probably led to a comparable result
>
> I'm making this point as this is an important one and I want to make sure
> that anyone reading this thread later does not get confused by your comments
>
>
>
>> *
>> *
>> *However*, the reduce phase got stuck on *one* node and sat there for more
>>
>> than 1 hour! I feel this might be because there is only one reducer
>> aggregating the output from 16 map tasks. But the code doesn't specify a
>> reducer class for this job. So:
>>
>
> just set the number of reducers that you want to use with the standard
> Hadoop parameter "mapred.reduce.tasks"
>
>
>
>> *a)* does that mean that the IdentityReducer is being used? The reducer
>>
>> seems to be CPU-bound, since the CPU has been at 100% for most of the time
>> on the single node that's performing the reduce. What is it doing? The
>> FetchOutputFormat class, to which data is finally output just funnels each
>> key+value into an appropriate folder (parse_text, parse_data, content), so
>> should not by itself account for the 100% utilization. Is it the
>> shuffle+sort on 1.4 million map output records causing this? Or something
>> else?
>>
>
> use jstack and / or the hadoop web app to monitor your crawl and see where
> time is spent
>
>
>
>> *b)* Is there any setting whereby I can increase the number of reducers
>>
>> used during the fetch() job?
>>
>
> see above
>
>
> Julien
>
>
>>
>> Thanks in advance!
>>
>> Cheers,
>> Safdar
>>
>>
>> On Tue, Jun 12, 2012 at 11:57 PM, Ali Safdar Kureishy <
>> safdar.kureishy@gmail.com> wrote:
>>
>> > Thanks Lewis and Julien, for your inputs.
>> >
>> > I will look into this a bit further and reply with some numbers, as seen
>> > with a fetchlist of 50K urls. It is late night here in my timezone, so
>> will
>> > look at this first thing in the morning.
>> >
>> > Thanks,
>> > Safdar
>> >
>> >
>> >
>> > On Tue, Jun 12, 2012 at 4:56 PM, Julien Nioche <
>> > lists.digitalpebble@gmail.com> wrote:
>> >
>> >> Guys,
>> >>
>> >> This has to do with the way URLs are grouped for politeness and not so
>> >> much
>> >> with the number of blocks in the input. Limiting the URLs by # host
>> >> names,
>> >> domains or IP is a way of ensuring an even distribution across the
>> >> cluster.
>> >> See nutch-default.xml for details
>> >>
>> >> J.
>> >>
>> >>
>> >> On 12 June 2012 13:06, Lewis John Mcgibbney <lewis.mcgibbney@gmail.com
>> >> >wrote:
>> >>
>> >> > Hi Ali,
>> >>
>> >>
>> >> > Please check out this post [0] I found. I need to agree with the
>> >> > response in the thread ans state that I don't know how Hadoop ensures
>> >> > even distribution of workload but we can assume that by explicitly
>> >> > specifying the mapper and reducers we can ensure that all 'will' be
>> >> > used across your cluster.
>> >> >
>> >> > hth
>> >> >
>> >> > [0] http://stackoverflow.com/questions/5748585/hadoop-workload
>> >> >
>> >> > On Tue, Jun 12, 2012 at 10:15 AM, Ali Safdar Kureishy
>> >> > <sa...@gmail.com> wrote:
>> >> > > Hi,
>> >> > >
>> >> > > I have a hadoop cluster of 5 nodes. I want to ensure that the fetch
>> >> phase
>> >> > > is distributed evenly across all the nodes (to maximize bandwidth
>> >> etc).
>> >> > > However, if I generate a fetchlist of size 1000 urls, does this get
>> >> > > distributed equally across the nodes? Doesn't the fact that the
>> size
>> >> of
>> >> > the
>> >> > > fetchlist is < 64MB (block size) result in it being fetched from a
>> >> single
>> >> > > node? If not, how is this distributed across the mappers evenly? Is
>> >> > there a
>> >> > > rough formulate I can use, to determine how many URLs I should
>> fetch
>> >> for
>> >> > an
>> >> > > equal distribution across my nodes, for a given block size setting?
>> >> > >
>> >> > > Thanks,
>> >> > > Safdar
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Lewis
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> *
>> >> *Open Source Solutions for Text Engineering
>> >>
>> >> http://digitalpebble.blogspot.com/
>> >> http://www.digitalpebble.com
>> >> http://twitter.com/digitalpebble
>> >>
>> >
>> >
>>
>
>
>
> --
> *
> *
> Open Source Solutions for Text Engineering
>
> http://digitalpebble.blogspot.com/
> http://www.digitalpebble.com
> http://twitter.com/digitalpebble
>
>
Re: How to ensure even distribution of the fetch phase across Hadoop nodes
Posted by Julien Nioche <li...@gmail.com>.
> Alright, I found out how to achieve an even distribution of URLs for fetch
> across the nodes of the cluster. It does not have to do with the
> generate.max.count setting suggested by Julien (though that is important
> too).
>
> The key is to set an appropriate value for the '*numLists*' parameter
> to the generate() method. This ensures that the fetch-list is broken into
> many files (blocks) at the end of the generate, which would yield the same
> number of map tasks for those URLs during fetch, and those get evently
> distributed across the cluster.
>
Ok, let's get a few things right. What you are referring to is called
[-numFetchers] when using the command nutch generate. It splits the output
into separate files which are then used as input by the Fetcher - it's not
about blocks as such. Indeed this means that this will allow the slaves on
the clusters to fetch from them - I forgot to mention it. However this does
not guarantee an even distribution of the URLs in the fetchlists, hence my
comment about generate.count.mode + generate.max.count
>
> So, with a 'numLists' setting of 16 during generate(), and a hadoop cluster
> of about 5 nodes, I was able to achieve a fetch() throughput of about 400K
> URLs in 2 hours for the map phase.
This was because the fetch was spread
> across the 5 nodes, using 16 mappers total - and 2 map slots per node.
>
no. this depends mostly on the distribution of URLS per host/domain/IP in
your fetchlist. If you have mostly URLs from a single hostname they will be
dealt with by a single mapper and most other inputs will be empty
regardless of the number of machines in your cluster. This is how we
enforce politeness in Nutch.
The results you are getting indicate that there probably is good
distribution of URLs in your fetch list + increading the number of threads
in the fetcher would have probably led to a comparable result
I'm making this point as this is an important one and I want to make sure
that anyone reading this thread later does not get confused by your comments
> *
> *
> *However*, the reduce phase got stuck on *one* node and sat there for more
> than 1 hour! I feel this might be because there is only one reducer
> aggregating the output from 16 map tasks. But the code doesn't specify a
> reducer class for this job. So:
>
just set the number of reducers that you want to use with the standard
Hadoop parameter "mapred.reduce.tasks"
> *a)* does that mean that the IdentityReducer is being used? The reducer
> seems to be CPU-bound, since the CPU has been at 100% for most of the time
> on the single node that's performing the reduce. What is it doing? The
> FetchOutputFormat class, to which data is finally output just funnels each
> key+value into an appropriate folder (parse_text, parse_data, content), so
> should not by itself account for the 100% utilization. Is it the
> shuffle+sort on 1.4 million map output records causing this? Or something
> else?
>
use jstack and / or the hadoop web app to monitor your crawl and see where
time is spent
> *b)* Is there any setting whereby I can increase the number of reducers
> used during the fetch() job?
>
see above
Julien
>
> Thanks in advance!
>
> Cheers,
> Safdar
>
>
> On Tue, Jun 12, 2012 at 11:57 PM, Ali Safdar Kureishy <
> safdar.kureishy@gmail.com> wrote:
>
> > Thanks Lewis and Julien, for your inputs.
> >
> > I will look into this a bit further and reply with some numbers, as seen
> > with a fetchlist of 50K urls. It is late night here in my timezone, so
> will
> > look at this first thing in the morning.
> >
> > Thanks,
> > Safdar
> >
> >
> >
> > On Tue, Jun 12, 2012 at 4:56 PM, Julien Nioche <
> > lists.digitalpebble@gmail.com> wrote:
> >
> >> Guys,
> >>
> >> This has to do with the way URLs are grouped for politeness and not so
> >> much
> >> with the number of blocks in the input. Limiting the URLs by # host
> >> names,
> >> domains or IP is a way of ensuring an even distribution across the
> >> cluster.
> >> See nutch-default.xml for details
> >>
> >> J.
> >>
> >>
> >> On 12 June 2012 13:06, Lewis John Mcgibbney <lewis.mcgibbney@gmail.com
> >> >wrote:
> >>
> >> > Hi Ali,
> >>
> >>
> >> > Please check out this post [0] I found. I need to agree with the
> >> > response in the thread ans state that I don't know how Hadoop ensures
> >> > even distribution of workload but we can assume that by explicitly
> >> > specifying the mapper and reducers we can ensure that all 'will' be
> >> > used across your cluster.
> >> >
> >> > hth
> >> >
> >> > [0] http://stackoverflow.com/questions/5748585/hadoop-workload
> >> >
> >> > On Tue, Jun 12, 2012 at 10:15 AM, Ali Safdar Kureishy
> >> > <sa...@gmail.com> wrote:
> >> > > Hi,
> >> > >
> >> > > I have a hadoop cluster of 5 nodes. I want to ensure that the fetch
> >> phase
> >> > > is distributed evenly across all the nodes (to maximize bandwidth
> >> etc).
> >> > > However, if I generate a fetchlist of size 1000 urls, does this get
> >> > > distributed equally across the nodes? Doesn't the fact that the size
> >> of
> >> > the
> >> > > fetchlist is < 64MB (block size) result in it being fetched from a
> >> single
> >> > > node? If not, how is this distributed across the mappers evenly? Is
> >> > there a
> >> > > rough formulate I can use, to determine how many URLs I should fetch
> >> for
> >> > an
> >> > > equal distribution across my nodes, for a given block size setting?
> >> > >
> >> > > Thanks,
> >> > > Safdar
> >> >
> >> >
> >> >
> >> > --
> >> > Lewis
> >> >
> >>
> >>
> >>
> >> --
> >> *
> >> *Open Source Solutions for Text Engineering
> >>
> >> http://digitalpebble.blogspot.com/
> >> http://www.digitalpebble.com
> >> http://twitter.com/digitalpebble
> >>
> >
> >
>
--
*
*Open Source Solutions for Text Engineering
http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble
Re: How to ensure even distribution of the fetch phase across Hadoop nodes
Posted by Ali Safdar Kureishy <sa...@gmail.com>.
Hi Julien/Lewis (and All),
Alright, I found out how to achieve an even distribution of URLs for fetch
across the nodes of the cluster. It does not have to do with the
generate.max.count setting suggested by Julien (though that is important
too). The key is to set an appropriate value for the '*numLists*' parameter
to the generate() method. This ensures that the fetch-list is broken into
many files (blocks) at the end of the generate, which would yield the same
number of map tasks for those URLs during fetch, and those get evently
distributed across the cluster.
So, with a 'numLists' setting of 16 during generate(), and a hadoop cluster
of about 5 nodes, I was able to achieve a fetch() throughput of about 400K
URLs in 2 hours for the map phase. This was because the fetch was spread
across the 5 nodes, using 16 mappers total - and 2 map slots per node.
*
*
*However*, the reduce phase got stuck on *one* node and sat there for more
than 1 hour! I feel this might be because there is only one reducer
aggregating the output from 16 map tasks. But the code doesn't specify a
reducer class for this job. So:
*a)* does that mean that the IdentityReducer is being used? The reducer
seems to be CPU-bound, since the CPU has been at 100% for most of the time
on the single node that's performing the reduce. What is it doing? The
FetchOutputFormat class, to which data is finally output just funnels each
key+value into an appropriate folder (parse_text, parse_data, content), so
should not by itself account for the 100% utilization. Is it the
shuffle+sort on 1.4 million map output records causing this? Or something
else?
*b)* Is there any setting whereby I can increase the number of reducers
used during the fetch() job?
Thanks in advance!
Cheers,
Safdar
On Tue, Jun 12, 2012 at 11:57 PM, Ali Safdar Kureishy <
safdar.kureishy@gmail.com> wrote:
> Thanks Lewis and Julien, for your inputs.
>
> I will look into this a bit further and reply with some numbers, as seen
> with a fetchlist of 50K urls. It is late night here in my timezone, so will
> look at this first thing in the morning.
>
> Thanks,
> Safdar
>
>
>
> On Tue, Jun 12, 2012 at 4:56 PM, Julien Nioche <
> lists.digitalpebble@gmail.com> wrote:
>
>> Guys,
>>
>> This has to do with the way URLs are grouped for politeness and not so
>> much
>> with the number of blocks in the input. Limiting the URLs by # host
>> names,
>> domains or IP is a way of ensuring an even distribution across the
>> cluster.
>> See nutch-default.xml for details
>>
>> J.
>>
>>
>> On 12 June 2012 13:06, Lewis John Mcgibbney <lewis.mcgibbney@gmail.com
>> >wrote:
>>
>> > Hi Ali,
>>
>>
>> > Please check out this post [0] I found. I need to agree with the
>> > response in the thread ans state that I don't know how Hadoop ensures
>> > even distribution of workload but we can assume that by explicitly
>> > specifying the mapper and reducers we can ensure that all 'will' be
>> > used across your cluster.
>> >
>> > hth
>> >
>> > [0] http://stackoverflow.com/questions/5748585/hadoop-workload
>> >
>> > On Tue, Jun 12, 2012 at 10:15 AM, Ali Safdar Kureishy
>> > <sa...@gmail.com> wrote:
>> > > Hi,
>> > >
>> > > I have a hadoop cluster of 5 nodes. I want to ensure that the fetch
>> phase
>> > > is distributed evenly across all the nodes (to maximize bandwidth
>> etc).
>> > > However, if I generate a fetchlist of size 1000 urls, does this get
>> > > distributed equally across the nodes? Doesn't the fact that the size
>> of
>> > the
>> > > fetchlist is < 64MB (block size) result in it being fetched from a
>> single
>> > > node? If not, how is this distributed across the mappers evenly? Is
>> > there a
>> > > rough formulate I can use, to determine how many URLs I should fetch
>> for
>> > an
>> > > equal distribution across my nodes, for a given block size setting?
>> > >
>> > > Thanks,
>> > > Safdar
>> >
>> >
>> >
>> > --
>> > Lewis
>> >
>>
>>
>>
>> --
>> *
>> *Open Source Solutions for Text Engineering
>>
>> http://digitalpebble.blogspot.com/
>> http://www.digitalpebble.com
>> http://twitter.com/digitalpebble
>>
>
>
Re: How to ensure even distribution of the fetch phase across Hadoop nodes
Posted by Ali Safdar Kureishy <sa...@gmail.com>.
Thanks Lewis and Julien, for your inputs.
I will look into this a bit further and reply with some numbers, as seen
with a fetchlist of 50K urls. It is late night here in my timezone, so will
look at this first thing in the morning.
Thanks,
Safdar
On Tue, Jun 12, 2012 at 4:56 PM, Julien Nioche <
lists.digitalpebble@gmail.com> wrote:
> Guys,
>
> This has to do with the way URLs are grouped for politeness and not so much
> with the number of blocks in the input. Limiting the URLs by # host names,
> domains or IP is a way of ensuring an even distribution across the cluster.
> See nutch-default.xml for details
>
> J.
>
>
> On 12 June 2012 13:06, Lewis John Mcgibbney <lewis.mcgibbney@gmail.com
> >wrote:
>
> > Hi Ali,
>
>
> > Please check out this post [0] I found. I need to agree with the
> > response in the thread ans state that I don't know how Hadoop ensures
> > even distribution of workload but we can assume that by explicitly
> > specifying the mapper and reducers we can ensure that all 'will' be
> > used across your cluster.
> >
> > hth
> >
> > [0] http://stackoverflow.com/questions/5748585/hadoop-workload
> >
> > On Tue, Jun 12, 2012 at 10:15 AM, Ali Safdar Kureishy
> > <sa...@gmail.com> wrote:
> > > Hi,
> > >
> > > I have a hadoop cluster of 5 nodes. I want to ensure that the fetch
> phase
> > > is distributed evenly across all the nodes (to maximize bandwidth etc).
> > > However, if I generate a fetchlist of size 1000 urls, does this get
> > > distributed equally across the nodes? Doesn't the fact that the size of
> > the
> > > fetchlist is < 64MB (block size) result in it being fetched from a
> single
> > > node? If not, how is this distributed across the mappers evenly? Is
> > there a
> > > rough formulate I can use, to determine how many URLs I should fetch
> for
> > an
> > > equal distribution across my nodes, for a given block size setting?
> > >
> > > Thanks,
> > > Safdar
> >
> >
> >
> > --
> > Lewis
> >
>
>
>
> --
> *
> *Open Source Solutions for Text Engineering
>
> http://digitalpebble.blogspot.com/
> http://www.digitalpebble.com
> http://twitter.com/digitalpebble
>
Re: How to ensure even distribution of the fetch phase across Hadoop nodes
Posted by Julien Nioche <li...@gmail.com>.
Guys,
This has to do with the way URLs are grouped for politeness and not so much
with the number of blocks in the input. Limiting the URLs by # host names,
domains or IP is a way of ensuring an even distribution across the cluster.
See nutch-default.xml for details
J.
On 12 June 2012 13:06, Lewis John Mcgibbney <le...@gmail.com>wrote:
> Hi Ali,
> Please check out this post [0] I found. I need to agree with the
> response in the thread ans state that I don't know how Hadoop ensures
> even distribution of workload but we can assume that by explicitly
> specifying the mapper and reducers we can ensure that all 'will' be
> used across your cluster.
>
> hth
>
> [0] http://stackoverflow.com/questions/5748585/hadoop-workload
>
> On Tue, Jun 12, 2012 at 10:15 AM, Ali Safdar Kureishy
> <sa...@gmail.com> wrote:
> > Hi,
> >
> > I have a hadoop cluster of 5 nodes. I want to ensure that the fetch phase
> > is distributed evenly across all the nodes (to maximize bandwidth etc).
> > However, if I generate a fetchlist of size 1000 urls, does this get
> > distributed equally across the nodes? Doesn't the fact that the size of
> the
> > fetchlist is < 64MB (block size) result in it being fetched from a single
> > node? If not, how is this distributed across the mappers evenly? Is
> there a
> > rough formulate I can use, to determine how many URLs I should fetch for
> an
> > equal distribution across my nodes, for a given block size setting?
> >
> > Thanks,
> > Safdar
>
>
>
> --
> Lewis
>
--
*
*Open Source Solutions for Text Engineering
http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble
Re: How to ensure even distribution of the fetch phase across Hadoop nodes
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Ali,
Please check out this post [0] I found. I need to agree with the
response in the thread ans state that I don't know how Hadoop ensures
even distribution of workload but we can assume that by explicitly
specifying the mapper and reducers we can ensure that all 'will' be
used across your cluster.
hth
[0] http://stackoverflow.com/questions/5748585/hadoop-workload
On Tue, Jun 12, 2012 at 10:15 AM, Ali Safdar Kureishy
<sa...@gmail.com> wrote:
> Hi,
>
> I have a hadoop cluster of 5 nodes. I want to ensure that the fetch phase
> is distributed evenly across all the nodes (to maximize bandwidth etc).
> However, if I generate a fetchlist of size 1000 urls, does this get
> distributed equally across the nodes? Doesn't the fact that the size of the
> fetchlist is < 64MB (block size) result in it being fetched from a single
> node? If not, how is this distributed across the mappers evenly? Is there a
> rough formulate I can use, to determine how many URLs I should fetch for an
> equal distribution across my nodes, for a given block size setting?
>
> Thanks,
> Safdar
--
Lewis