You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Amit Kumar <am...@msn.com> on 2015/04/09 13:35:26 UTC
Not able to run more than one map task
Hi All,
We recently started working on Hadoop. We have setup the hadoop in pseduo distribution mode along with oozie.
Every developer has set it up on his laptop. The problem is that we are not able to run more than one map task concurrently on our laptops. Resource manager is not allowing more than one task on our machine.
My task gets completed if I submit it without Oozie. Oozie requires one map task for its own functioning. Actual task that oozie submit does not start.
Here is my configuration
-- Hadoop setup in Pseudo distribution mode -- Hadoop Version - 2.6-- Oozie Version - 4.0.1
Regards,Amit
Re: Not able to run more than one map task
Posted by Drake민영근 <dr...@nexr.com>.
Hi, Amit.
Test these:
Increase "yarn.nodemanager.resource.memory-mb" beyond 8192. That's ok for
testing. And decrease "mapreduce.map.memory.mb" to 256 and add
"yarn.nodemanager.vmem-check-enabled" to "false" in yarn-site.xml.
Thanks.
Drake 민영근 Ph.D
kt NexR
On Sat, Apr 11, 2015 at 8:01 AM, Niels Basjes <Ni...@basjes.nl> wrote:
> Just curious: what is the input for your job ? If it is a single gzipped
> file then that is the cause of getting exactly 1 mapper.
>
> Niels
>
> On Fri, Apr 10, 2015, 09:21 Amit Kumar <am...@msn.com> wrote:
>
>> Thanks a lot Harsha for replying
>>
>> This problem has waster at least last one week.
>>
>> We tried what you suggested. Could you please take a look at the
>> configuration and suggest if we missed c?
>>
>> System RAM : 8GB
>> CPU : 4 threads each with 2 cores.
>> # Disks : 1
>>
>> MR2:
>>
>> mapreduce.map.memory.mb : 512
>> mapreduce.tasktracker.map.tasks.maximum : 4
>>
>> Yarn:
>>
>> yarn.app.mapreduce.am.resource.mb : 512
>> yarn.nodemanager.resource.cpu-vcores : 4
>> yarn.scheduler.minimum-allocation-mb : 512
>> yarn.nodemanager.resource.memory-mb : 5080
>>
>>
>> Regards,
>> Amit
>>
>>
>>
>> > From: harsh@cloudera.com
>> > Date: Fri, 10 Apr 2015 10:20:24 +0530
>> > Subject: Re: Not able to run more than one map task
>> > To: user@hadoop.apache.org
>>
>> >
>> > You are likely memory/vcore starved in the NM's configs. Increase your
>> > yarn.nodemanager.resource.memory-mb and
>> > yarn.nodemanager.resource.cpu-vcores configs, or consider lowering the
>> > MR job memory request values to gain more parallelism.
>> >
>> > On Thu, Apr 9, 2015 at 5:05 PM, Amit Kumar <am...@msn.com> wrote:
>> > > Hi All,
>> > >
>> > > We recently started working on Hadoop. We have setup the hadoop in
>> pseduo
>> > > distribution mode along with oozie.
>> > >
>> > > Every developer has set it up on his laptop. The problem is that we
>> are not
>> > > able to run more than one map task concurrently on our laptops.
>> Resource
>> > > manager is not allowing more than one task on our machine.
>> > >
>> > > My task gets completed if I submit it without Oozie. Oozie requires
>> one map
>> > > task for its own functioning. Actual task that oozie submit does not
>> start.
>> > >
>> > > Here is my configuration
>> > >
>> > > -- Hadoop setup in Pseudo distribution mode
>> > > -- Hadoop Version - 2.6
>> > > -- Oozie Version - 4.0.1
>> > >
>> > > Regards,
>> > > Amit
>> >
>> >
>> >
>> > --
>> > Harsh J
>>
>
Re: Not able to run more than one map task
Posted by Drake민영근 <dr...@nexr.com>.
Hi, Amit.
Test these:
Increase "yarn.nodemanager.resource.memory-mb" beyond 8192. That's ok for
testing. And decrease "mapreduce.map.memory.mb" to 256 and add
"yarn.nodemanager.vmem-check-enabled" to "false" in yarn-site.xml.
Thanks.
Drake 민영근 Ph.D
kt NexR
On Sat, Apr 11, 2015 at 8:01 AM, Niels Basjes <Ni...@basjes.nl> wrote:
> Just curious: what is the input for your job ? If it is a single gzipped
> file then that is the cause of getting exactly 1 mapper.
>
> Niels
>
> On Fri, Apr 10, 2015, 09:21 Amit Kumar <am...@msn.com> wrote:
>
>> Thanks a lot Harsha for replying
>>
>> This problem has waster at least last one week.
>>
>> We tried what you suggested. Could you please take a look at the
>> configuration and suggest if we missed c?
>>
>> System RAM : 8GB
>> CPU : 4 threads each with 2 cores.
>> # Disks : 1
>>
>> MR2:
>>
>> mapreduce.map.memory.mb : 512
>> mapreduce.tasktracker.map.tasks.maximum : 4
>>
>> Yarn:
>>
>> yarn.app.mapreduce.am.resource.mb : 512
>> yarn.nodemanager.resource.cpu-vcores : 4
>> yarn.scheduler.minimum-allocation-mb : 512
>> yarn.nodemanager.resource.memory-mb : 5080
>>
>>
>> Regards,
>> Amit
>>
>>
>>
>> > From: harsh@cloudera.com
>> > Date: Fri, 10 Apr 2015 10:20:24 +0530
>> > Subject: Re: Not able to run more than one map task
>> > To: user@hadoop.apache.org
>>
>> >
>> > You are likely memory/vcore starved in the NM's configs. Increase your
>> > yarn.nodemanager.resource.memory-mb and
>> > yarn.nodemanager.resource.cpu-vcores configs, or consider lowering the
>> > MR job memory request values to gain more parallelism.
>> >
>> > On Thu, Apr 9, 2015 at 5:05 PM, Amit Kumar <am...@msn.com> wrote:
>> > > Hi All,
>> > >
>> > > We recently started working on Hadoop. We have setup the hadoop in
>> pseduo
>> > > distribution mode along with oozie.
>> > >
>> > > Every developer has set it up on his laptop. The problem is that we
>> are not
>> > > able to run more than one map task concurrently on our laptops.
>> Resource
>> > > manager is not allowing more than one task on our machine.
>> > >
>> > > My task gets completed if I submit it without Oozie. Oozie requires
>> one map
>> > > task for its own functioning. Actual task that oozie submit does not
>> start.
>> > >
>> > > Here is my configuration
>> > >
>> > > -- Hadoop setup in Pseudo distribution mode
>> > > -- Hadoop Version - 2.6
>> > > -- Oozie Version - 4.0.1
>> > >
>> > > Regards,
>> > > Amit
>> >
>> >
>> >
>> > --
>> > Harsh J
>>
>
Re: Not able to run more than one map task
Posted by Drake민영근 <dr...@nexr.com>.
Hi, Amit.
Test these:
Increase "yarn.nodemanager.resource.memory-mb" beyond 8192. That's ok for
testing. And decrease "mapreduce.map.memory.mb" to 256 and add
"yarn.nodemanager.vmem-check-enabled" to "false" in yarn-site.xml.
Thanks.
Drake 민영근 Ph.D
kt NexR
On Sat, Apr 11, 2015 at 8:01 AM, Niels Basjes <Ni...@basjes.nl> wrote:
> Just curious: what is the input for your job ? If it is a single gzipped
> file then that is the cause of getting exactly 1 mapper.
>
> Niels
>
> On Fri, Apr 10, 2015, 09:21 Amit Kumar <am...@msn.com> wrote:
>
>> Thanks a lot Harsha for replying
>>
>> This problem has waster at least last one week.
>>
>> We tried what you suggested. Could you please take a look at the
>> configuration and suggest if we missed c?
>>
>> System RAM : 8GB
>> CPU : 4 threads each with 2 cores.
>> # Disks : 1
>>
>> MR2:
>>
>> mapreduce.map.memory.mb : 512
>> mapreduce.tasktracker.map.tasks.maximum : 4
>>
>> Yarn:
>>
>> yarn.app.mapreduce.am.resource.mb : 512
>> yarn.nodemanager.resource.cpu-vcores : 4
>> yarn.scheduler.minimum-allocation-mb : 512
>> yarn.nodemanager.resource.memory-mb : 5080
>>
>>
>> Regards,
>> Amit
>>
>>
>>
>> > From: harsh@cloudera.com
>> > Date: Fri, 10 Apr 2015 10:20:24 +0530
>> > Subject: Re: Not able to run more than one map task
>> > To: user@hadoop.apache.org
>>
>> >
>> > You are likely memory/vcore starved in the NM's configs. Increase your
>> > yarn.nodemanager.resource.memory-mb and
>> > yarn.nodemanager.resource.cpu-vcores configs, or consider lowering the
>> > MR job memory request values to gain more parallelism.
>> >
>> > On Thu, Apr 9, 2015 at 5:05 PM, Amit Kumar <am...@msn.com> wrote:
>> > > Hi All,
>> > >
>> > > We recently started working on Hadoop. We have setup the hadoop in
>> pseduo
>> > > distribution mode along with oozie.
>> > >
>> > > Every developer has set it up on his laptop. The problem is that we
>> are not
>> > > able to run more than one map task concurrently on our laptops.
>> Resource
>> > > manager is not allowing more than one task on our machine.
>> > >
>> > > My task gets completed if I submit it without Oozie. Oozie requires
>> one map
>> > > task for its own functioning. Actual task that oozie submit does not
>> start.
>> > >
>> > > Here is my configuration
>> > >
>> > > -- Hadoop setup in Pseudo distribution mode
>> > > -- Hadoop Version - 2.6
>> > > -- Oozie Version - 4.0.1
>> > >
>> > > Regards,
>> > > Amit
>> >
>> >
>> >
>> > --
>> > Harsh J
>>
>
Re: Not able to run more than one map task
Posted by Drake민영근 <dr...@nexr.com>.
Hi, Amit.
Test these:
Increase "yarn.nodemanager.resource.memory-mb" beyond 8192. That's ok for
testing. And decrease "mapreduce.map.memory.mb" to 256 and add
"yarn.nodemanager.vmem-check-enabled" to "false" in yarn-site.xml.
Thanks.
Drake 민영근 Ph.D
kt NexR
On Sat, Apr 11, 2015 at 8:01 AM, Niels Basjes <Ni...@basjes.nl> wrote:
> Just curious: what is the input for your job ? If it is a single gzipped
> file then that is the cause of getting exactly 1 mapper.
>
> Niels
>
> On Fri, Apr 10, 2015, 09:21 Amit Kumar <am...@msn.com> wrote:
>
>> Thanks a lot Harsha for replying
>>
>> This problem has waster at least last one week.
>>
>> We tried what you suggested. Could you please take a look at the
>> configuration and suggest if we missed c?
>>
>> System RAM : 8GB
>> CPU : 4 threads each with 2 cores.
>> # Disks : 1
>>
>> MR2:
>>
>> mapreduce.map.memory.mb : 512
>> mapreduce.tasktracker.map.tasks.maximum : 4
>>
>> Yarn:
>>
>> yarn.app.mapreduce.am.resource.mb : 512
>> yarn.nodemanager.resource.cpu-vcores : 4
>> yarn.scheduler.minimum-allocation-mb : 512
>> yarn.nodemanager.resource.memory-mb : 5080
>>
>>
>> Regards,
>> Amit
>>
>>
>>
>> > From: harsh@cloudera.com
>> > Date: Fri, 10 Apr 2015 10:20:24 +0530
>> > Subject: Re: Not able to run more than one map task
>> > To: user@hadoop.apache.org
>>
>> >
>> > You are likely memory/vcore starved in the NM's configs. Increase your
>> > yarn.nodemanager.resource.memory-mb and
>> > yarn.nodemanager.resource.cpu-vcores configs, or consider lowering the
>> > MR job memory request values to gain more parallelism.
>> >
>> > On Thu, Apr 9, 2015 at 5:05 PM, Amit Kumar <am...@msn.com> wrote:
>> > > Hi All,
>> > >
>> > > We recently started working on Hadoop. We have setup the hadoop in
>> pseduo
>> > > distribution mode along with oozie.
>> > >
>> > > Every developer has set it up on his laptop. The problem is that we
>> are not
>> > > able to run more than one map task concurrently on our laptops.
>> Resource
>> > > manager is not allowing more than one task on our machine.
>> > >
>> > > My task gets completed if I submit it without Oozie. Oozie requires
>> one map
>> > > task for its own functioning. Actual task that oozie submit does not
>> start.
>> > >
>> > > Here is my configuration
>> > >
>> > > -- Hadoop setup in Pseudo distribution mode
>> > > -- Hadoop Version - 2.6
>> > > -- Oozie Version - 4.0.1
>> > >
>> > > Regards,
>> > > Amit
>> >
>> >
>> >
>> > --
>> > Harsh J
>>
>
Re: Not able to run more than one map task
Posted by Niels Basjes <Ni...@basjes.nl>.
Just curious: what is the input for your job ? If it is a single gzipped
file then that is the cause of getting exactly 1 mapper.
Niels
On Fri, Apr 10, 2015, 09:21 Amit Kumar <am...@msn.com> wrote:
> Thanks a lot Harsha for replying
>
> This problem has waster at least last one week.
>
> We tried what you suggested. Could you please take a look at the
> configuration and suggest if we missed c?
>
> System RAM : 8GB
> CPU : 4 threads each with 2 cores.
> # Disks : 1
>
> MR2:
>
> mapreduce.map.memory.mb : 512
> mapreduce.tasktracker.map.tasks.maximum : 4
>
> Yarn:
>
> yarn.app.mapreduce.am.resource.mb : 512
> yarn.nodemanager.resource.cpu-vcores : 4
> yarn.scheduler.minimum-allocation-mb : 512
> yarn.nodemanager.resource.memory-mb : 5080
>
>
> Regards,
> Amit
>
>
>
> > From: harsh@cloudera.com
> > Date: Fri, 10 Apr 2015 10:20:24 +0530
> > Subject: Re: Not able to run more than one map task
> > To: user@hadoop.apache.org
>
> >
> > You are likely memory/vcore starved in the NM's configs. Increase your
> > yarn.nodemanager.resource.memory-mb and
> > yarn.nodemanager.resource.cpu-vcores configs, or consider lowering the
> > MR job memory request values to gain more parallelism.
> >
> > On Thu, Apr 9, 2015 at 5:05 PM, Amit Kumar <am...@msn.com> wrote:
> > > Hi All,
> > >
> > > We recently started working on Hadoop. We have setup the hadoop in
> pseduo
> > > distribution mode along with oozie.
> > >
> > > Every developer has set it up on his laptop. The problem is that we
> are not
> > > able to run more than one map task concurrently on our laptops.
> Resource
> > > manager is not allowing more than one task on our machine.
> > >
> > > My task gets completed if I submit it without Oozie. Oozie requires
> one map
> > > task for its own functioning. Actual task that oozie submit does not
> start.
> > >
> > > Here is my configuration
> > >
> > > -- Hadoop setup in Pseudo distribution mode
> > > -- Hadoop Version - 2.6
> > > -- Oozie Version - 4.0.1
> > >
> > > Regards,
> > > Amit
> >
> >
> >
> > --
> > Harsh J
>
Re: Not able to run more than one map task
Posted by Niels Basjes <Ni...@basjes.nl>.
Just curious: what is the input for your job ? If it is a single gzipped
file then that is the cause of getting exactly 1 mapper.
Niels
On Fri, Apr 10, 2015, 09:21 Amit Kumar <am...@msn.com> wrote:
> Thanks a lot Harsha for replying
>
> This problem has waster at least last one week.
>
> We tried what you suggested. Could you please take a look at the
> configuration and suggest if we missed c?
>
> System RAM : 8GB
> CPU : 4 threads each with 2 cores.
> # Disks : 1
>
> MR2:
>
> mapreduce.map.memory.mb : 512
> mapreduce.tasktracker.map.tasks.maximum : 4
>
> Yarn:
>
> yarn.app.mapreduce.am.resource.mb : 512
> yarn.nodemanager.resource.cpu-vcores : 4
> yarn.scheduler.minimum-allocation-mb : 512
> yarn.nodemanager.resource.memory-mb : 5080
>
>
> Regards,
> Amit
>
>
>
> > From: harsh@cloudera.com
> > Date: Fri, 10 Apr 2015 10:20:24 +0530
> > Subject: Re: Not able to run more than one map task
> > To: user@hadoop.apache.org
>
> >
> > You are likely memory/vcore starved in the NM's configs. Increase your
> > yarn.nodemanager.resource.memory-mb and
> > yarn.nodemanager.resource.cpu-vcores configs, or consider lowering the
> > MR job memory request values to gain more parallelism.
> >
> > On Thu, Apr 9, 2015 at 5:05 PM, Amit Kumar <am...@msn.com> wrote:
> > > Hi All,
> > >
> > > We recently started working on Hadoop. We have setup the hadoop in
> pseduo
> > > distribution mode along with oozie.
> > >
> > > Every developer has set it up on his laptop. The problem is that we
> are not
> > > able to run more than one map task concurrently on our laptops.
> Resource
> > > manager is not allowing more than one task on our machine.
> > >
> > > My task gets completed if I submit it without Oozie. Oozie requires
> one map
> > > task for its own functioning. Actual task that oozie submit does not
> start.
> > >
> > > Here is my configuration
> > >
> > > -- Hadoop setup in Pseudo distribution mode
> > > -- Hadoop Version - 2.6
> > > -- Oozie Version - 4.0.1
> > >
> > > Regards,
> > > Amit
> >
> >
> >
> > --
> > Harsh J
>
Re: Not able to run more than one map task
Posted by Niels Basjes <Ni...@basjes.nl>.
Just curious: what is the input for your job ? If it is a single gzipped
file then that is the cause of getting exactly 1 mapper.
Niels
On Fri, Apr 10, 2015, 09:21 Amit Kumar <am...@msn.com> wrote:
> Thanks a lot Harsha for replying
>
> This problem has waster at least last one week.
>
> We tried what you suggested. Could you please take a look at the
> configuration and suggest if we missed c?
>
> System RAM : 8GB
> CPU : 4 threads each with 2 cores.
> # Disks : 1
>
> MR2:
>
> mapreduce.map.memory.mb : 512
> mapreduce.tasktracker.map.tasks.maximum : 4
>
> Yarn:
>
> yarn.app.mapreduce.am.resource.mb : 512
> yarn.nodemanager.resource.cpu-vcores : 4
> yarn.scheduler.minimum-allocation-mb : 512
> yarn.nodemanager.resource.memory-mb : 5080
>
>
> Regards,
> Amit
>
>
>
> > From: harsh@cloudera.com
> > Date: Fri, 10 Apr 2015 10:20:24 +0530
> > Subject: Re: Not able to run more than one map task
> > To: user@hadoop.apache.org
>
> >
> > You are likely memory/vcore starved in the NM's configs. Increase your
> > yarn.nodemanager.resource.memory-mb and
> > yarn.nodemanager.resource.cpu-vcores configs, or consider lowering the
> > MR job memory request values to gain more parallelism.
> >
> > On Thu, Apr 9, 2015 at 5:05 PM, Amit Kumar <am...@msn.com> wrote:
> > > Hi All,
> > >
> > > We recently started working on Hadoop. We have setup the hadoop in
> pseduo
> > > distribution mode along with oozie.
> > >
> > > Every developer has set it up on his laptop. The problem is that we
> are not
> > > able to run more than one map task concurrently on our laptops.
> Resource
> > > manager is not allowing more than one task on our machine.
> > >
> > > My task gets completed if I submit it without Oozie. Oozie requires
> one map
> > > task for its own functioning. Actual task that oozie submit does not
> start.
> > >
> > > Here is my configuration
> > >
> > > -- Hadoop setup in Pseudo distribution mode
> > > -- Hadoop Version - 2.6
> > > -- Oozie Version - 4.0.1
> > >
> > > Regards,
> > > Amit
> >
> >
> >
> > --
> > Harsh J
>
Re: Not able to run more than one map task
Posted by Niels Basjes <Ni...@basjes.nl>.
Just curious: what is the input for your job ? If it is a single gzipped
file then that is the cause of getting exactly 1 mapper.
Niels
On Fri, Apr 10, 2015, 09:21 Amit Kumar <am...@msn.com> wrote:
> Thanks a lot Harsha for replying
>
> This problem has waster at least last one week.
>
> We tried what you suggested. Could you please take a look at the
> configuration and suggest if we missed c?
>
> System RAM : 8GB
> CPU : 4 threads each with 2 cores.
> # Disks : 1
>
> MR2:
>
> mapreduce.map.memory.mb : 512
> mapreduce.tasktracker.map.tasks.maximum : 4
>
> Yarn:
>
> yarn.app.mapreduce.am.resource.mb : 512
> yarn.nodemanager.resource.cpu-vcores : 4
> yarn.scheduler.minimum-allocation-mb : 512
> yarn.nodemanager.resource.memory-mb : 5080
>
>
> Regards,
> Amit
>
>
>
> > From: harsh@cloudera.com
> > Date: Fri, 10 Apr 2015 10:20:24 +0530
> > Subject: Re: Not able to run more than one map task
> > To: user@hadoop.apache.org
>
> >
> > You are likely memory/vcore starved in the NM's configs. Increase your
> > yarn.nodemanager.resource.memory-mb and
> > yarn.nodemanager.resource.cpu-vcores configs, or consider lowering the
> > MR job memory request values to gain more parallelism.
> >
> > On Thu, Apr 9, 2015 at 5:05 PM, Amit Kumar <am...@msn.com> wrote:
> > > Hi All,
> > >
> > > We recently started working on Hadoop. We have setup the hadoop in
> pseduo
> > > distribution mode along with oozie.
> > >
> > > Every developer has set it up on his laptop. The problem is that we
> are not
> > > able to run more than one map task concurrently on our laptops.
> Resource
> > > manager is not allowing more than one task on our machine.
> > >
> > > My task gets completed if I submit it without Oozie. Oozie requires
> one map
> > > task for its own functioning. Actual task that oozie submit does not
> start.
> > >
> > > Here is my configuration
> > >
> > > -- Hadoop setup in Pseudo distribution mode
> > > -- Hadoop Version - 2.6
> > > -- Oozie Version - 4.0.1
> > >
> > > Regards,
> > > Amit
> >
> >
> >
> > --
> > Harsh J
>
RE: Not able to run more than one map task
Posted by Amit Kumar <am...@msn.com>.
Thanks a lot Harsha for replying
This problem has waster at least last one week.
We tried what you suggested. Could you please take a look at the configuration and suggest if we missed c?
System RAM : 8GB
CPU : 4 threads each with 2 cores.
# Disks : 1MR2:mapreduce.map.memory.mb : 512mapreduce.tasktracker.map.tasks.maximum : 4Yarn:yarn.app.mapreduce.am.resource.mb : 512yarn.nodemanager.resource.cpu-vcores : 4yarn.scheduler.minimum-allocation-mb : 512yarn.nodemanager.resource.memory-mb : 5080
Regards,Amit
> From: harsh@cloudera.com
> Date: Fri, 10 Apr 2015 10:20:24 +0530
> Subject: Re: Not able to run more than one map task
> To: user@hadoop.apache.org
>
> You are likely memory/vcore starved in the NM's configs. Increase your
> yarn.nodemanager.resource.memory-mb and
> yarn.nodemanager.resource.cpu-vcores configs, or consider lowering the
> MR job memory request values to gain more parallelism.
>
> On Thu, Apr 9, 2015 at 5:05 PM, Amit Kumar <am...@msn.com> wrote:
> > Hi All,
> >
> > We recently started working on Hadoop. We have setup the hadoop in pseduo
> > distribution mode along with oozie.
> >
> > Every developer has set it up on his laptop. The problem is that we are not
> > able to run more than one map task concurrently on our laptops. Resource
> > manager is not allowing more than one task on our machine.
> >
> > My task gets completed if I submit it without Oozie. Oozie requires one map
> > task for its own functioning. Actual task that oozie submit does not start.
> >
> > Here is my configuration
> >
> > -- Hadoop setup in Pseudo distribution mode
> > -- Hadoop Version - 2.6
> > -- Oozie Version - 4.0.1
> >
> > Regards,
> > Amit
>
>
>
> --
> Harsh J
RE: Not able to run more than one map task
Posted by Amit Kumar <am...@msn.com>.
Thanks a lot Harsha for replying
This problem has waster at least last one week.
We tried what you suggested. Could you please take a look at the configuration and suggest if we missed c?
System RAM : 8GB
CPU : 4 threads each with 2 cores.
# Disks : 1MR2:mapreduce.map.memory.mb : 512mapreduce.tasktracker.map.tasks.maximum : 4Yarn:yarn.app.mapreduce.am.resource.mb : 512yarn.nodemanager.resource.cpu-vcores : 4yarn.scheduler.minimum-allocation-mb : 512yarn.nodemanager.resource.memory-mb : 5080
Regards,Amit
> From: harsh@cloudera.com
> Date: Fri, 10 Apr 2015 10:20:24 +0530
> Subject: Re: Not able to run more than one map task
> To: user@hadoop.apache.org
>
> You are likely memory/vcore starved in the NM's configs. Increase your
> yarn.nodemanager.resource.memory-mb and
> yarn.nodemanager.resource.cpu-vcores configs, or consider lowering the
> MR job memory request values to gain more parallelism.
>
> On Thu, Apr 9, 2015 at 5:05 PM, Amit Kumar <am...@msn.com> wrote:
> > Hi All,
> >
> > We recently started working on Hadoop. We have setup the hadoop in pseduo
> > distribution mode along with oozie.
> >
> > Every developer has set it up on his laptop. The problem is that we are not
> > able to run more than one map task concurrently on our laptops. Resource
> > manager is not allowing more than one task on our machine.
> >
> > My task gets completed if I submit it without Oozie. Oozie requires one map
> > task for its own functioning. Actual task that oozie submit does not start.
> >
> > Here is my configuration
> >
> > -- Hadoop setup in Pseudo distribution mode
> > -- Hadoop Version - 2.6
> > -- Oozie Version - 4.0.1
> >
> > Regards,
> > Amit
>
>
>
> --
> Harsh J
RE: Not able to run more than one map task
Posted by Amit Kumar <am...@msn.com>.
Thanks a lot Harsha for replying
This problem has waster at least last one week.
We tried what you suggested. Could you please take a look at the configuration and suggest if we missed c?
System RAM : 8GB
CPU : 4 threads each with 2 cores.
# Disks : 1MR2:mapreduce.map.memory.mb : 512mapreduce.tasktracker.map.tasks.maximum : 4Yarn:yarn.app.mapreduce.am.resource.mb : 512yarn.nodemanager.resource.cpu-vcores : 4yarn.scheduler.minimum-allocation-mb : 512yarn.nodemanager.resource.memory-mb : 5080
Regards,Amit
> From: harsh@cloudera.com
> Date: Fri, 10 Apr 2015 10:20:24 +0530
> Subject: Re: Not able to run more than one map task
> To: user@hadoop.apache.org
>
> You are likely memory/vcore starved in the NM's configs. Increase your
> yarn.nodemanager.resource.memory-mb and
> yarn.nodemanager.resource.cpu-vcores configs, or consider lowering the
> MR job memory request values to gain more parallelism.
>
> On Thu, Apr 9, 2015 at 5:05 PM, Amit Kumar <am...@msn.com> wrote:
> > Hi All,
> >
> > We recently started working on Hadoop. We have setup the hadoop in pseduo
> > distribution mode along with oozie.
> >
> > Every developer has set it up on his laptop. The problem is that we are not
> > able to run more than one map task concurrently on our laptops. Resource
> > manager is not allowing more than one task on our machine.
> >
> > My task gets completed if I submit it without Oozie. Oozie requires one map
> > task for its own functioning. Actual task that oozie submit does not start.
> >
> > Here is my configuration
> >
> > -- Hadoop setup in Pseudo distribution mode
> > -- Hadoop Version - 2.6
> > -- Oozie Version - 4.0.1
> >
> > Regards,
> > Amit
>
>
>
> --
> Harsh J
RE: Not able to run more than one map task
Posted by Amit Kumar <am...@msn.com>.
Thanks a lot Harsha for replying
This problem has waster at least last one week.
We tried what you suggested. Could you please take a look at the configuration and suggest if we missed c?
System RAM : 8GB
CPU : 4 threads each with 2 cores.
# Disks : 1MR2:mapreduce.map.memory.mb : 512mapreduce.tasktracker.map.tasks.maximum : 4Yarn:yarn.app.mapreduce.am.resource.mb : 512yarn.nodemanager.resource.cpu-vcores : 4yarn.scheduler.minimum-allocation-mb : 512yarn.nodemanager.resource.memory-mb : 5080
Regards,Amit
> From: harsh@cloudera.com
> Date: Fri, 10 Apr 2015 10:20:24 +0530
> Subject: Re: Not able to run more than one map task
> To: user@hadoop.apache.org
>
> You are likely memory/vcore starved in the NM's configs. Increase your
> yarn.nodemanager.resource.memory-mb and
> yarn.nodemanager.resource.cpu-vcores configs, or consider lowering the
> MR job memory request values to gain more parallelism.
>
> On Thu, Apr 9, 2015 at 5:05 PM, Amit Kumar <am...@msn.com> wrote:
> > Hi All,
> >
> > We recently started working on Hadoop. We have setup the hadoop in pseduo
> > distribution mode along with oozie.
> >
> > Every developer has set it up on his laptop. The problem is that we are not
> > able to run more than one map task concurrently on our laptops. Resource
> > manager is not allowing more than one task on our machine.
> >
> > My task gets completed if I submit it without Oozie. Oozie requires one map
> > task for its own functioning. Actual task that oozie submit does not start.
> >
> > Here is my configuration
> >
> > -- Hadoop setup in Pseudo distribution mode
> > -- Hadoop Version - 2.6
> > -- Oozie Version - 4.0.1
> >
> > Regards,
> > Amit
>
>
>
> --
> Harsh J
Re: Not able to run more than one map task
Posted by Harsh J <ha...@cloudera.com>.
You are likely memory/vcore starved in the NM's configs. Increase your
yarn.nodemanager.resource.memory-mb and
yarn.nodemanager.resource.cpu-vcores configs, or consider lowering the
MR job memory request values to gain more parallelism.
On Thu, Apr 9, 2015 at 5:05 PM, Amit Kumar <am...@msn.com> wrote:
> Hi All,
>
> We recently started working on Hadoop. We have setup the hadoop in pseduo
> distribution mode along with oozie.
>
> Every developer has set it up on his laptop. The problem is that we are not
> able to run more than one map task concurrently on our laptops. Resource
> manager is not allowing more than one task on our machine.
>
> My task gets completed if I submit it without Oozie. Oozie requires one map
> task for its own functioning. Actual task that oozie submit does not start.
>
> Here is my configuration
>
> -- Hadoop setup in Pseudo distribution mode
> -- Hadoop Version - 2.6
> -- Oozie Version - 4.0.1
>
> Regards,
> Amit
--
Harsh J
Re: Not able to run more than one map task
Posted by Harsh J <ha...@cloudera.com>.
You are likely memory/vcore starved in the NM's configs. Increase your
yarn.nodemanager.resource.memory-mb and
yarn.nodemanager.resource.cpu-vcores configs, or consider lowering the
MR job memory request values to gain more parallelism.
On Thu, Apr 9, 2015 at 5:05 PM, Amit Kumar <am...@msn.com> wrote:
> Hi All,
>
> We recently started working on Hadoop. We have setup the hadoop in pseduo
> distribution mode along with oozie.
>
> Every developer has set it up on his laptop. The problem is that we are not
> able to run more than one map task concurrently on our laptops. Resource
> manager is not allowing more than one task on our machine.
>
> My task gets completed if I submit it without Oozie. Oozie requires one map
> task for its own functioning. Actual task that oozie submit does not start.
>
> Here is my configuration
>
> -- Hadoop setup in Pseudo distribution mode
> -- Hadoop Version - 2.6
> -- Oozie Version - 4.0.1
>
> Regards,
> Amit
--
Harsh J
Re: Not able to run more than one map task
Posted by Harsh J <ha...@cloudera.com>.
You are likely memory/vcore starved in the NM's configs. Increase your
yarn.nodemanager.resource.memory-mb and
yarn.nodemanager.resource.cpu-vcores configs, or consider lowering the
MR job memory request values to gain more parallelism.
On Thu, Apr 9, 2015 at 5:05 PM, Amit Kumar <am...@msn.com> wrote:
> Hi All,
>
> We recently started working on Hadoop. We have setup the hadoop in pseduo
> distribution mode along with oozie.
>
> Every developer has set it up on his laptop. The problem is that we are not
> able to run more than one map task concurrently on our laptops. Resource
> manager is not allowing more than one task on our machine.
>
> My task gets completed if I submit it without Oozie. Oozie requires one map
> task for its own functioning. Actual task that oozie submit does not start.
>
> Here is my configuration
>
> -- Hadoop setup in Pseudo distribution mode
> -- Hadoop Version - 2.6
> -- Oozie Version - 4.0.1
>
> Regards,
> Amit
--
Harsh J
Re: Not able to run more than one map task
Posted by Harsh J <ha...@cloudera.com>.
You are likely memory/vcore starved in the NM's configs. Increase your
yarn.nodemanager.resource.memory-mb and
yarn.nodemanager.resource.cpu-vcores configs, or consider lowering the
MR job memory request values to gain more parallelism.
On Thu, Apr 9, 2015 at 5:05 PM, Amit Kumar <am...@msn.com> wrote:
> Hi All,
>
> We recently started working on Hadoop. We have setup the hadoop in pseduo
> distribution mode along with oozie.
>
> Every developer has set it up on his laptop. The problem is that we are not
> able to run more than one map task concurrently on our laptops. Resource
> manager is not allowing more than one task on our machine.
>
> My task gets completed if I submit it without Oozie. Oozie requires one map
> task for its own functioning. Actual task that oozie submit does not start.
>
> Here is my configuration
>
> -- Hadoop setup in Pseudo distribution mode
> -- Hadoop Version - 2.6
> -- Oozie Version - 4.0.1
>
> Regards,
> Amit
--
Harsh J