You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Sheng Guo <en...@gmail.com> on 2012/06/23 04:27:01 UTC
How can I set the mapper number for pig script?
Hi everyone,
Sorry to bother.
How can I configure the number of mappers for my pig script?
Thanks a lot!
Sheng
Re: How can I set the mapper number for pig script?
Posted by Scott Foster <sc...@gmail.com>.
You are right that if you have a CPU intensive mapper then having more
mappers will help in that case. As suggested you can reduce the block
size of the files you are processing and disable split combination and
you'll end up with more mappers in your job.
One correction to my previous email, the setting to turn off split
combination is:
SET pig.noSplitCombination true;
scott.
Re: How can I set the mapper number for pig script?
Posted by John Meagher <jo...@gmail.com>.
Another option is to either reduce the block sizes of the input data
or disabling the combine input format and splitting the data into more
files.
On Sat, Jun 23, 2012 at 5:58 PM, Yang <te...@gmail.com> wrote:
> hi Sheng:
>
> I had exactly the same problem as you did.
>
> right now with hadoop 0.20 and above you can't do it anymore, because the
> new mapreduce.lib.input.FileInputFormat disabled the original
> mapred.map.tasks control to compute the goalSize in
> getSplits() method. ---- the old mapred.FileInputFormat class had this
> control
>
> I submitted https://issues.apache.org/jira/browse/HADOOP-8503 to add back
> this control
>
>
> because pig actually compiles some hadoop classes into its own jar ,
> including this FileInputFormat class, you could actually work around this
> by patching your own hadoop jar, then build pig with this jar, and then use
> your re-built pig in production. you need to make sure to use the full pig
> jar instead of the pig-withouthadoop.jar.
>
> you can also kind of achieve part of the same goal by setting
> mapreduces.max.split.size, but this is rather inflexible, and if your pig
> script generates several MR jobs, the same split size will hold for all the
> jobs, which may not be ideal, if one stage consumes a lot more input data
> than another.
>
>
> Yang
>
> On Sat, Jun 23, 2012 at 1:48 PM, Sheng Guo <en...@gmail.com> wrote:
>
>> Thanks for all your help.
>>
>> My pig script may have some cpu-intensive job like nlp processing, so it
>> would be helpful if I have multiple mappers running. Correct me if I am
>> wrong.
>> Thanks,
>>
>> Sheng
>>
>> On Sat, Jun 23, 2012 at 9:40 AM, Scott Foster <scottf.concur@gmail.com
>> >wrote:
>>
>> > You can also turn off split combination completely and then the number
>> > of mappers will equal the number of blocks
>> > SET pig.noSplitCombination false;
>> >
>> > Adding mappers may not make your process run faster since the time to
>> > read the data may be less than the overhead of creating a new JVM for
>> > each map task.
>> >
>> > scott.
>> >
>>
Re: How can I set the mapper number for pig script?
Posted by Yang <te...@gmail.com>.
hi Sheng:
I had exactly the same problem as you did.
right now with hadoop 0.20 and above you can't do it anymore, because the
new mapreduce.lib.input.FileInputFormat disabled the original
mapred.map.tasks control to compute the goalSize in
getSplits() method. ---- the old mapred.FileInputFormat class had this
control
I submitted https://issues.apache.org/jira/browse/HADOOP-8503 to add back
this control
because pig actually compiles some hadoop classes into its own jar ,
including this FileInputFormat class, you could actually work around this
by patching your own hadoop jar, then build pig with this jar, and then use
your re-built pig in production. you need to make sure to use the full pig
jar instead of the pig-withouthadoop.jar.
you can also kind of achieve part of the same goal by setting
mapreduces.max.split.size, but this is rather inflexible, and if your pig
script generates several MR jobs, the same split size will hold for all the
jobs, which may not be ideal, if one stage consumes a lot more input data
than another.
Yang
On Sat, Jun 23, 2012 at 1:48 PM, Sheng Guo <en...@gmail.com> wrote:
> Thanks for all your help.
>
> My pig script may have some cpu-intensive job like nlp processing, so it
> would be helpful if I have multiple mappers running. Correct me if I am
> wrong.
> Thanks,
>
> Sheng
>
> On Sat, Jun 23, 2012 at 9:40 AM, Scott Foster <scottf.concur@gmail.com
> >wrote:
>
> > You can also turn off split combination completely and then the number
> > of mappers will equal the number of blocks
> > SET pig.noSplitCombination false;
> >
> > Adding mappers may not make your process run faster since the time to
> > read the data may be less than the overhead of creating a new JVM for
> > each map task.
> >
> > scott.
> >
>
Re: How can I set the mapper number for pig script?
Posted by Sheng Guo <en...@gmail.com>.
Thanks for all your help.
My pig script may have some cpu-intensive job like nlp processing, so it
would be helpful if I have multiple mappers running. Correct me if I am
wrong.
Thanks,
Sheng
On Sat, Jun 23, 2012 at 9:40 AM, Scott Foster <sc...@gmail.com>wrote:
> You can also turn off split combination completely and then the number
> of mappers will equal the number of blocks
> SET pig.noSplitCombination false;
>
> Adding mappers may not make your process run faster since the time to
> read the data may be less than the overhead of creating a new JVM for
> each map task.
>
> scott.
>
Re: How can I set the mapper number for pig script?
Posted by Scott Foster <sc...@gmail.com>.
You can also turn off split combination completely and then the number
of mappers will equal the number of blocks
SET pig.noSplitCombination false;
Adding mappers may not make your process run faster since the time to
read the data may be less than the overhead of creating a new JVM for
each map task.
scott.
Re: How can I set the mapper number for pig script?
Posted by Stan Rosenberg <st...@gmail.com>.
On Sat, Jun 23, 2012 at 3:30 AM, Sheng Guo <en...@gmail.com> wrote:
> I know it is automatically set. But I have a large data set, I want it
> allocate more mappers during midnight so that more computing resource could
> be used to speed up.
> Any suggestions?
Pig uses CombineInputFormat by default which attempts to combine a set
of physical input splits into one logical input split.
I use the following setting to control the number of mappers in some
of my benchmarking scripts:
-- combine upto this many bytes into a composite input split, i.e., per mapper
SET pig.maxCombinedSplitSize 250000000;
Note that your are absolute min. is constrained by the smallest block
size in your input set.
Re: How can I set the mapper number for pig script?
Posted by Sheng Guo <en...@gmail.com>.
I know it is automatically set. But I have a large data set, I want it
allocate more mappers during midnight so that more computing resource could
be used to speed up.
Any suggestions?
thanks,
On Fri, Jun 22, 2012 at 9:31 PM, Jagat Singh <ja...@gmail.com> wrote:
> Numbers of mappers are selected based on input size by hadoop automatically
>
> -----------
> Sent from Mobile , short and crisp.
> On 23-Jun-2012 7:57 AM, "Sheng Guo" <en...@gmail.com> wrote:
>
> > Hi everyone,
> >
> > Sorry to bother.
> >
> > How can I configure the number of mappers for my pig script?
> >
> > Thanks a lot!
> >
> > Sheng
> >
>
Re: How can I set the mapper number for pig script?
Posted by Jagat Singh <ja...@gmail.com>.
Numbers of mappers are selected based on input size by hadoop automatically
-----------
Sent from Mobile , short and crisp.
On 23-Jun-2012 7:57 AM, "Sheng Guo" <en...@gmail.com> wrote:
> Hi everyone,
>
> Sorry to bother.
>
> How can I configure the number of mappers for my pig script?
>
> Thanks a lot!
>
> Sheng
>