You are viewing a plain text version of this content. The canonical link for it is here.

Posted to hdfs-user@hadoop.apache.org by Patcharee Thongtra <Pa...@uni.no> on 2014/04/11 14:22:41 UTC

Number of map task

Hi,

I wrote a custom InputFormat. When I ran the pig script Load function 
using this InputFormat, the number of InputSplit > 1, but there was only 
1 map task handling these splits.

Does the number of Map task not correspond to the number of splits?

I think the job will be done quicker if there are more Map tasks?

Patcharee

Re: Number of map task

Posted by Harsh J <ha...@cloudera.com>.

Hi,

Pig applies a CombineFileInputFormat wrapper on top of the storage by
default: http://pig.apache.org/docs/r0.12.0/perf.html#combine-files

Please also ask Pig specific questions on its own list
(user@pig.apache.org) for reaching the right audience.

On Fri, Apr 11, 2014 at 5:52 PM, Patcharee Thongtra
<Pa...@uni.no> wrote:
> Hi,
>
> I wrote a custom InputFormat. When I ran the pig script Load function using
> this InputFormat, the number of InputSplit > 1, but there was only 1 map
> task handling these splits.
>
> Does the number of Map task not correspond to the number of splits?
>
> I think the job will be done quicker if there are more Map tasks?
>
> Patcharee



-- 
Harsh J

Re: Number of map task

Posted by Harsh J <ha...@cloudera.com>.

Hi,

Pig applies a CombineFileInputFormat wrapper on top of the storage by
default: http://pig.apache.org/docs/r0.12.0/perf.html#combine-files

Please also ask Pig specific questions on its own list
(user@pig.apache.org) for reaching the right audience.

On Fri, Apr 11, 2014 at 5:52 PM, Patcharee Thongtra
<Pa...@uni.no> wrote:
> Hi,
>
> I wrote a custom InputFormat. When I ran the pig script Load function using
> this InputFormat, the number of InputSplit > 1, but there was only 1 map
> task handling these splits.
>
> Does the number of Map task not correspond to the number of splits?
>
> I think the job will be done quicker if there are more Map tasks?
>
> Patcharee



-- 
Harsh J

Re: Number of map task

Posted by Harsh J <ha...@cloudera.com>.

Hi,

Pig applies a CombineFileInputFormat wrapper on top of the storage by
default: http://pig.apache.org/docs/r0.12.0/perf.html#combine-files

Please also ask Pig specific questions on its own list
(user@pig.apache.org) for reaching the right audience.

On Fri, Apr 11, 2014 at 5:52 PM, Patcharee Thongtra
<Pa...@uni.no> wrote:
> Hi,
>
> I wrote a custom InputFormat. When I ran the pig script Load function using
> this InputFormat, the number of InputSplit > 1, but there was only 1 map
> task handling these splits.
>
> Does the number of Map task not correspond to the number of splits?
>
> I think the job will be done quicker if there are more Map tasks?
>
> Patcharee



-- 
Harsh J

Re: Number of map task

Posted by Harsh J <ha...@cloudera.com>.

Hi,

Pig applies a CombineFileInputFormat wrapper on top of the storage by
default: http://pig.apache.org/docs/r0.12.0/perf.html#combine-files

Please also ask Pig specific questions on its own list
(user@pig.apache.org) for reaching the right audience.

On Fri, Apr 11, 2014 at 5:52 PM, Patcharee Thongtra
<Pa...@uni.no> wrote:
> Hi,
>
> I wrote a custom InputFormat. When I ran the pig script Load function using
> this InputFormat, the number of InputSplit > 1, but there was only 1 map
> task handling these splits.
>
> Does the number of Map task not correspond to the number of splits?
>
> I think the job will be done quicker if there are more Map tasks?
>
> Patcharee



-- 
Harsh J