You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by samir das mohapatra <sa...@gmail.com> on 2013/03/12 11:27:34 UTC
Why hadoop is spawing two map over file size 1.5 KB ?
Hi All,
I have very fundamental doubt, I have file having size 1.5KB and block
size is default block size, But i could see two mapper it got creted during
the Job. Could you please help to get whole picture why it is .
Regards,
samir.
Re: Why hadoop is spawing two map over file size 1.5 KB ?
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
What's your input-format?
Thanks,
+Vinod Kumar Vavilapalli
On Mar 12, 2013, at 3:27 AM, samir das mohapatra wrote:
> Hi All,
> I have very fundamental doubt, I have file having size 1.5KB and block size is default block size, But i could see two mapper it got creted during the Job. Could you please help to get whole picture why it is .
>
> Regards,
> samir.
Re: Why hadoop is spawing two map over file size 1.5 KB ?
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
What's your input-format?
Thanks,
+Vinod Kumar Vavilapalli
On Mar 12, 2013, at 3:27 AM, samir das mohapatra wrote:
> Hi All,
> I have very fundamental doubt, I have file having size 1.5KB and block size is default block size, But i could see two mapper it got creted during the Job. Could you please help to get whole picture why it is .
>
> Regards,
> samir.
Re: 答复: Why hadoop is spawing two map over file size 1.5 KB ?
Posted by yypvsxf19870706 <yy...@gmail.com>.
Hi
We just have discussed about the similar problem(the title is Increase the number of mappers in PM mode),as I thought.
First of all you can not setup the the number of mappers.however you change the mapred.max.split.size, or mapred.min.split.size ,the number of the mapper will change automatically.
I think the two mappers which you got in your situation,that is because the value of the mapred.max.split.size, or mapred.min.split.size.
Do please check that .and appreciate that you give the results.
Regards.
�����ҵ� iPhone
�� 2013-3-15��0:04����־ǿ <wa...@360.cn> д����
> please check parameter "mapred.map.tasks" is equals to 2?
> ������: Yanbo Liang [yanbohappy@gmail.com]
> ����ʱ��: 2013��3��14�� 23:11
> �ռ���: user@hadoop.apache.org
> ����: Re: Why hadoop is spawing two map over file size 1.5 KB ?
>
> I guess may be one of them is the speculative execution.
> You can check the parameter "mapred.map.tasks.speculative.execution" to ensure whether it is allowed speculative execution.
> You can get the precise information that whether it is speculative map task from the tasktracker log.
>
> 2013/3/12 samir das mohapatra <sa...@gmail.com>
>> Hi All,
>> I have very fundamental doubt, I have file having size 1.5KB and block size is default block size, But i could see two mapper it got creted during the Job. Could you please help to get whole picture why it is .
>>
>> Regards,
>> samir.
>
Re: 答复: Why hadoop is spawing two map over file size 1.5 KB ?
Posted by yypvsxf19870706 <yy...@gmail.com>.
Hi
We just have discussed about the similar problem(the title is Increase the number of mappers in PM mode),as I thought.
First of all you can not setup the the number of mappers.however you change the mapred.max.split.size, or mapred.min.split.size ,the number of the mapper will change automatically.
I think the two mappers which you got in your situation,that is because the value of the mapred.max.split.size, or mapred.min.split.size.
Do please check that .and appreciate that you give the results.
Regards.
发自我的 iPhone
在 2013-3-15，0:04，王志强 <wa...@360.cn> 写道：
> please check parameter "mapred.map.tasks" is equals to 2?
> 发件人: Yanbo Liang [yanbohappy@gmail.com]
> 发送时间: 2013年3月14日 23:11
> 收件人: user@hadoop.apache.org
> 主题: Re: Why hadoop is spawing two map over file size 1.5 KB ?
>
> I guess may be one of them is the speculative execution.
> You can check the parameter "mapred.map.tasks.speculative.execution" to ensure whether it is allowed speculative execution.
> You can get the precise information that whether it is speculative map task from the tasktracker log.
>
> 2013/3/12 samir das mohapatra <sa...@gmail.com>
>> Hi All,
>> I have very fundamental doubt, I have file having size 1.5KB and block size is default block size, But i could see two mapper it got creted during the Job. Could you please help to get whole picture why it is .
>>
>> Regards,
>> samir.
>
Re: 答复: Why hadoop is spawing two map over file size 1.5 KB ?
Posted by yypvsxf19870706 <yy...@gmail.com>.
Hi
We just have discussed about the similar problem(the title is Increase the number of mappers in PM mode),as I thought.
First of all you can not setup the the number of mappers.however you change the mapred.max.split.size, or mapred.min.split.size ,the number of the mapper will change automatically.
I think the two mappers which you got in your situation,that is because the value of the mapred.max.split.size, or mapred.min.split.size.
Do please check that .and appreciate that you give the results.
Regards.
发自我的 iPhone
在 2013-3-15，0:04，王志强 <wa...@360.cn> 写道：
> please check parameter "mapred.map.tasks" is equals to 2?
> 发件人: Yanbo Liang [yanbohappy@gmail.com]
> 发送时间: 2013年3月14日 23:11
> 收件人: user@hadoop.apache.org
> 主题: Re: Why hadoop is spawing two map over file size 1.5 KB ?
>
> I guess may be one of them is the speculative execution.
> You can check the parameter "mapred.map.tasks.speculative.execution" to ensure whether it is allowed speculative execution.
> You can get the precise information that whether it is speculative map task from the tasktracker log.
>
> 2013/3/12 samir das mohapatra <sa...@gmail.com>
>> Hi All,
>> I have very fundamental doubt, I have file having size 1.5KB and block size is default block size, But i could see two mapper it got creted during the Job. Could you please help to get whole picture why it is .
>>
>> Regards,
>> samir.
>
Re: 答复: Why hadoop is spawing two map over file size 1.5 KB ?
Posted by yypvsxf19870706 <yy...@gmail.com>.
Hi
We just have discussed about the similar problem(the title is Increase the number of mappers in PM mode),as I thought.
First of all you can not setup the the number of mappers.however you change the mapred.max.split.size, or mapred.min.split.size ,the number of the mapper will change automatically.
I think the two mappers which you got in your situation,that is because the value of the mapred.max.split.size, or mapred.min.split.size.
Do please check that .and appreciate that you give the results.
Regards.
�����ҵ� iPhone
�� 2013-3-15��0:04����־ǿ <wa...@360.cn> д����
> please check parameter "mapred.map.tasks" is equals to 2?
> ������: Yanbo Liang [yanbohappy@gmail.com]
> ����ʱ��: 2013��3��14�� 23:11
> �ռ���: user@hadoop.apache.org
> ����: Re: Why hadoop is spawing two map over file size 1.5 KB ?
>
> I guess may be one of them is the speculative execution.
> You can check the parameter "mapred.map.tasks.speculative.execution" to ensure whether it is allowed speculative execution.
> You can get the precise information that whether it is speculative map task from the tasktracker log.
>
> 2013/3/12 samir das mohapatra <sa...@gmail.com>
>> Hi All,
>> I have very fundamental doubt, I have file having size 1.5KB and block size is default block size, But i could see two mapper it got creted during the Job. Could you please help to get whole picture why it is .
>>
>> Regards,
>> samir.
>
答复: Why hadoop is spawing two map over file size 1.5 KB ?
Posted by 王志强 <wa...@360.cn>.
please check parameter "mapred.map.tasks" is equals to 2?
________________________________
发件人: Yanbo Liang [yanbohappy@gmail.com]
发送时间: 2013年3月14日 23:11
收件人: user@hadoop.apache.org
主题: Re: Why hadoop is spawing two map over file size 1.5 KB ?
I guess may be one of them is the speculative execution.
You can check the parameter "mapred.map.tasks.speculative.execution" to ensure whether it is allowed speculative execution.
You can get the precise information that whether it is speculative map task from the tasktracker log.
2013/3/12 samir das mohapatra <sa...@gmail.com>>
Hi All,
I have very fundamental doubt, I have file having size 1.5KB and block size is default block size, But i could see two mapper it got creted during the Job. Could you please help to get whole picture why it is .
Regards,
samir.
答复: Why hadoop is spawing two map over file size 1.5 KB ?
Posted by 王志强 <wa...@360.cn>.
please check parameter "mapred.map.tasks" is equals to 2?
________________________________
发件人: Yanbo Liang [yanbohappy@gmail.com]
发送时间: 2013年3月14日 23:11
收件人: user@hadoop.apache.org
主题: Re: Why hadoop is spawing two map over file size 1.5 KB ?
I guess may be one of them is the speculative execution.
You can check the parameter "mapred.map.tasks.speculative.execution" to ensure whether it is allowed speculative execution.
You can get the precise information that whether it is speculative map task from the tasktracker log.
2013/3/12 samir das mohapatra <sa...@gmail.com>>
Hi All,
I have very fundamental doubt, I have file having size 1.5KB and block size is default block size, But i could see two mapper it got creted during the Job. Could you please help to get whole picture why it is .
Regards,
samir.
答复: Why hadoop is spawing two map over file size 1.5 KB ?
Posted by 王志强 <wa...@360.cn>.
please check parameter "mapred.map.tasks" is equals to 2?
________________________________
发件人: Yanbo Liang [yanbohappy@gmail.com]
发送时间: 2013年3月14日 23:11
收件人: user@hadoop.apache.org
主题: Re: Why hadoop is spawing two map over file size 1.5 KB ?
I guess may be one of them is the speculative execution.
You can check the parameter "mapred.map.tasks.speculative.execution" to ensure whether it is allowed speculative execution.
You can get the precise information that whether it is speculative map task from the tasktracker log.
2013/3/12 samir das mohapatra <sa...@gmail.com>>
Hi All,
I have very fundamental doubt, I have file having size 1.5KB and block size is default block size, But i could see two mapper it got creted during the Job. Could you please help to get whole picture why it is .
Regards,
samir.
答复: Why hadoop is spawing two map over file size 1.5 KB ?
Posted by 王志强 <wa...@360.cn>.
please check parameter "mapred.map.tasks" is equals to 2?
________________________________
发件人: Yanbo Liang [yanbohappy@gmail.com]
发送时间: 2013年3月14日 23:11
收件人: user@hadoop.apache.org
主题: Re: Why hadoop is spawing two map over file size 1.5 KB ?
I guess may be one of them is the speculative execution.
You can check the parameter "mapred.map.tasks.speculative.execution" to ensure whether it is allowed speculative execution.
You can get the precise information that whether it is speculative map task from the tasktracker log.
2013/3/12 samir das mohapatra <sa...@gmail.com>>
Hi All,
I have very fundamental doubt, I have file having size 1.5KB and block size is default block size, But i could see two mapper it got creted during the Job. Could you please help to get whole picture why it is .
Regards,
samir.
Re: Why hadoop is spawing two map over file size 1.5 KB ?
Posted by Yanbo Liang <ya...@gmail.com>.
I guess may be one of them is the speculative execution.
You can check the parameter "mapred.map.tasks.speculative.execution" to
ensure whether it is allowed speculative execution.
You can get the precise information that whether it is speculative map task
from the tasktracker log.
2013/3/12 samir das mohapatra <sa...@gmail.com>
> Hi All,
> I have very fundamental doubt, I have file having size 1.5KB and block
> size is default block size, But i could see two mapper it got creted during
> the Job. Could you please help to get whole picture why it is .
>
> Regards,
> samir.
>
Re: Why hadoop is spawing two map over file size 1.5 KB ?
Posted by Yanbo Liang <ya...@gmail.com>.
I guess may be one of them is the speculative execution.
You can check the parameter "mapred.map.tasks.speculative.execution" to
ensure whether it is allowed speculative execution.
You can get the precise information that whether it is speculative map task
from the tasktracker log.
2013/3/12 samir das mohapatra <sa...@gmail.com>
> Hi All,
> I have very fundamental doubt, I have file having size 1.5KB and block
> size is default block size, But i could see two mapper it got creted during
> the Job. Could you please help to get whole picture why it is .
>
> Regards,
> samir.
>
Re: Why hadoop is spawing two map over file size 1.5 KB ?
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
What's your input-format?
Thanks,
+Vinod Kumar Vavilapalli
On Mar 12, 2013, at 3:27 AM, samir das mohapatra wrote:
> Hi All,
> I have very fundamental doubt, I have file having size 1.5KB and block size is default block size, But i could see two mapper it got creted during the Job. Could you please help to get whole picture why it is .
>
> Regards,
> samir.
Re: Why hadoop is spawing two map over file size 1.5 KB ?
Posted by Yanbo Liang <ya...@gmail.com>.
I guess may be one of them is the speculative execution.
You can check the parameter "mapred.map.tasks.speculative.execution" to
ensure whether it is allowed speculative execution.
You can get the precise information that whether it is speculative map task
from the tasktracker log.
2013/3/12 samir das mohapatra <sa...@gmail.com>
> Hi All,
> I have very fundamental doubt, I have file having size 1.5KB and block
> size is default block size, But i could see two mapper it got creted during
> the Job. Could you please help to get whole picture why it is .
>
> Regards,
> samir.
>
Re: Why hadoop is spawing two map over file size 1.5 KB ?
Posted by Yanbo Liang <ya...@gmail.com>.
I guess may be one of them is the speculative execution.
You can check the parameter "mapred.map.tasks.speculative.execution" to
ensure whether it is allowed speculative execution.
You can get the precise information that whether it is speculative map task
from the tasktracker log.
2013/3/12 samir das mohapatra <sa...@gmail.com>
> Hi All,
> I have very fundamental doubt, I have file having size 1.5KB and block
> size is default block size, But i could see two mapper it got creted during
> the Job. Could you please help to get whole picture why it is .
>
> Regards,
> samir.
>
Re: Why hadoop is spawing two map over file size 1.5 KB ?
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
What's your input-format?
Thanks,
+Vinod Kumar Vavilapalli
On Mar 12, 2013, at 3:27 AM, samir das mohapatra wrote:
> Hi All,
> I have very fundamental doubt, I have file having size 1.5KB and block size is default block size, But i could see two mapper it got creted during the Job. Could you please help to get whole picture why it is .
>
> Regards,
> samir.