You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by hari <ha...@gmail.com> on 2014/05/27 09:31:38 UTC
YARN creates only 1 container
Hi,
When using YARN 2.2.0 version, only 1 container is created
for an application in the entire cluster.
The single container is created at an arbitrary node
for every run. This happens when running any application from
the examples jar (e.g., wordcount). Currently only one application is
run at a time. The input datasize is > 200GB.
I am setting custom values that affect concurrent container count.
These config parameters were mostly taken from:
http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
These wasn't much description elsewhere on how the container count would be
decided.
The settings are:
mapred-site.xml
mapreduce.map.cpu.vcores = 1
mapreduce.reduce.cpu.vcores = 1
mapreduce.map.memory.mb = 1024
mapreduce.reduce.memory.mb = 1024
mapreduce.map.java.opts = -Xmx1024m
mapreduce.reduce.java.opts = -Xmx1024m
yarn.xml
yarn.nodemanager.resource.memory-mb = 65536
yarn.nodemanager.resource.cpu-vcores = 16
>From these settings, each node should be running 16 containers.
Let me know if there might be something else affecting the container
count.
thanks,
hari
Re: YARN creates only 1 container
Posted by hari <ha...@gmail.com>.
Just an update on this: turns out that in 2.2.0 version, the container
count is implemented by using the memory
resource, even though there are configs for vcores. Things might have
changed since
then.
Thanks for the suggestions. yarn.xml was a typo, should have been
yarn-site.xml.
About the heap size, it shouldn't be a problem unless the memory allocation
footprint of the
container exceeds or reaches close to the configured heap size. This
happened when
those settings were used. For other cases, the memory values were
different.
On Wed, Jul 2, 2014 at 6:27 PM, Adam Kawa <ka...@gmail.com> wrote:
> You also might want to increase values for mapreduce.{map,reduce}.memory.mb
> to 1280 or 1536 or so (assuming that mapreduce.{map,reduce}.java.opts =
> -Xmx1024m).
>
> mapreduce.{map,reduce}.memory.mb is logical size of the container and it
> should be larger than mapreduce.{map,reduce}.java.opts that specifies the
> heap for JVM running the actual task.
>
>
> 2014-06-27 5:43 GMT+02:00 Wangda Tan <wh...@gmail.com>:
>
> It should be yarn-site.xml not yarn.xml.
>> yarn.xml will not be added to $CLASSPATH
>>
>> Thanks,
>> Wangda
>>
>>
>> On Wed, May 28, 2014 at 8:56 AM, hari <ha...@gmail.com> wrote:
>>
>>> The issue was not related the configuration related to containers. Due
>>> to
>>> misconfiguration, the Application master was not able to contact
>>> resourcemanager
>>> causing in the 1 container problem.
>>>
>>> However, the total containers allocated still is not as expected. The
>>> configuration settings
>>> should have resulted in 16 containers per node, but it is allocating 64
>>> containers per node.
>>>
>>> Reiterating the config parameters here again:
>>>
>>> mapred-site.xml
>>> mapreduce.map.cpu.vcores = 1
>>> mapreduce.reduce.cpu.vcores = 1
>>> mapreduce.map.memory.mb = 1024
>>> mapreduce.reduce.memory.mb = 1024
>>> mapreduce.map.java.opts = -Xmx1024m
>>> mapreduce.reduce.java.opts = -Xmx1024m
>>>
>>> yarn.xml
>>> yarn.nodemanager.resource.memory-mb = 65536
>>> yarn.nodemanager.resource.cpu-vcores = 16
>>> yarn.scheduler.minimum-allocation-mb = 1024
>>> yarn.scheduler.maximum-allocation-mb = 2048
>>> yarn.scheduler.minimum-allocation-vcores = 1
>>> yarn.scheduler.maximum-allocation-vcores = 1
>>>
>>> Is there anything else that might be causing this problem ?
>>>
>>> thanks,
>>> hari
>>>
>>>
>>>
>>>
>>>
>>> On Tue, May 27, 2014 at 3:31 AM, hari <ha...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> When using YARN 2.2.0 version, only 1 container is created
>>>> for an application in the entire cluster.
>>>> The single container is created at an arbitrary node
>>>> for every run. This happens when running any application from
>>>> the examples jar (e.g., wordcount). Currently only one application is
>>>> run at a time. The input datasize is > 200GB.
>>>>
>>>> I am setting custom values that affect concurrent container count.
>>>> These config parameters were mostly taken from:
>>>>
>>>> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>>>> These wasn't much description elsewhere on how the container count
>>>> would be
>>>> decided.
>>>>
>>>> The settings are:
>>>>
>>>> mapred-site.xml
>>>> mapreduce.map.cpu.vcores = 1
>>>> mapreduce.reduce.cpu.vcores = 1
>>>> mapreduce.map.memory.mb = 1024
>>>> mapreduce.reduce.memory.mb = 1024
>>>> mapreduce.map.java.opts = -Xmx1024m
>>>> mapreduce.reduce.java.opts = -Xmx1024m
>>>>
>>>> yarn.xml
>>>> yarn.nodemanager.resource.memory-mb = 65536
>>>> yarn.nodemanager.resource.cpu-vcores = 16
>>>>
>>>> From these settings, each node should be running 16 containers.
>>>>
>>>> Let me know if there might be something else affecting the container
>>>> count.
>>>>
>>>> thanks,
>>>> hari
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
Re: YARN creates only 1 container
Posted by hari <ha...@gmail.com>.
Just an update on this: turns out that in 2.2.0 version, the container
count is implemented by using the memory
resource, even though there are configs for vcores. Things might have
changed since
then.
Thanks for the suggestions. yarn.xml was a typo, should have been
yarn-site.xml.
About the heap size, it shouldn't be a problem unless the memory allocation
footprint of the
container exceeds or reaches close to the configured heap size. This
happened when
those settings were used. For other cases, the memory values were
different.
On Wed, Jul 2, 2014 at 6:27 PM, Adam Kawa <ka...@gmail.com> wrote:
> You also might want to increase values for mapreduce.{map,reduce}.memory.mb
> to 1280 or 1536 or so (assuming that mapreduce.{map,reduce}.java.opts =
> -Xmx1024m).
>
> mapreduce.{map,reduce}.memory.mb is logical size of the container and it
> should be larger than mapreduce.{map,reduce}.java.opts that specifies the
> heap for JVM running the actual task.
>
>
> 2014-06-27 5:43 GMT+02:00 Wangda Tan <wh...@gmail.com>:
>
> It should be yarn-site.xml not yarn.xml.
>> yarn.xml will not be added to $CLASSPATH
>>
>> Thanks,
>> Wangda
>>
>>
>> On Wed, May 28, 2014 at 8:56 AM, hari <ha...@gmail.com> wrote:
>>
>>> The issue was not related the configuration related to containers. Due
>>> to
>>> misconfiguration, the Application master was not able to contact
>>> resourcemanager
>>> causing in the 1 container problem.
>>>
>>> However, the total containers allocated still is not as expected. The
>>> configuration settings
>>> should have resulted in 16 containers per node, but it is allocating 64
>>> containers per node.
>>>
>>> Reiterating the config parameters here again:
>>>
>>> mapred-site.xml
>>> mapreduce.map.cpu.vcores = 1
>>> mapreduce.reduce.cpu.vcores = 1
>>> mapreduce.map.memory.mb = 1024
>>> mapreduce.reduce.memory.mb = 1024
>>> mapreduce.map.java.opts = -Xmx1024m
>>> mapreduce.reduce.java.opts = -Xmx1024m
>>>
>>> yarn.xml
>>> yarn.nodemanager.resource.memory-mb = 65536
>>> yarn.nodemanager.resource.cpu-vcores = 16
>>> yarn.scheduler.minimum-allocation-mb = 1024
>>> yarn.scheduler.maximum-allocation-mb = 2048
>>> yarn.scheduler.minimum-allocation-vcores = 1
>>> yarn.scheduler.maximum-allocation-vcores = 1
>>>
>>> Is there anything else that might be causing this problem ?
>>>
>>> thanks,
>>> hari
>>>
>>>
>>>
>>>
>>>
>>> On Tue, May 27, 2014 at 3:31 AM, hari <ha...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> When using YARN 2.2.0 version, only 1 container is created
>>>> for an application in the entire cluster.
>>>> The single container is created at an arbitrary node
>>>> for every run. This happens when running any application from
>>>> the examples jar (e.g., wordcount). Currently only one application is
>>>> run at a time. The input datasize is > 200GB.
>>>>
>>>> I am setting custom values that affect concurrent container count.
>>>> These config parameters were mostly taken from:
>>>>
>>>> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>>>> These wasn't much description elsewhere on how the container count
>>>> would be
>>>> decided.
>>>>
>>>> The settings are:
>>>>
>>>> mapred-site.xml
>>>> mapreduce.map.cpu.vcores = 1
>>>> mapreduce.reduce.cpu.vcores = 1
>>>> mapreduce.map.memory.mb = 1024
>>>> mapreduce.reduce.memory.mb = 1024
>>>> mapreduce.map.java.opts = -Xmx1024m
>>>> mapreduce.reduce.java.opts = -Xmx1024m
>>>>
>>>> yarn.xml
>>>> yarn.nodemanager.resource.memory-mb = 65536
>>>> yarn.nodemanager.resource.cpu-vcores = 16
>>>>
>>>> From these settings, each node should be running 16 containers.
>>>>
>>>> Let me know if there might be something else affecting the container
>>>> count.
>>>>
>>>> thanks,
>>>> hari
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
Re: YARN creates only 1 container
Posted by hari <ha...@gmail.com>.
Just an update on this: turns out that in 2.2.0 version, the container
count is implemented by using the memory
resource, even though there are configs for vcores. Things might have
changed since
then.
Thanks for the suggestions. yarn.xml was a typo, should have been
yarn-site.xml.
About the heap size, it shouldn't be a problem unless the memory allocation
footprint of the
container exceeds or reaches close to the configured heap size. This
happened when
those settings were used. For other cases, the memory values were
different.
On Wed, Jul 2, 2014 at 6:27 PM, Adam Kawa <ka...@gmail.com> wrote:
> You also might want to increase values for mapreduce.{map,reduce}.memory.mb
> to 1280 or 1536 or so (assuming that mapreduce.{map,reduce}.java.opts =
> -Xmx1024m).
>
> mapreduce.{map,reduce}.memory.mb is logical size of the container and it
> should be larger than mapreduce.{map,reduce}.java.opts that specifies the
> heap for JVM running the actual task.
>
>
> 2014-06-27 5:43 GMT+02:00 Wangda Tan <wh...@gmail.com>:
>
> It should be yarn-site.xml not yarn.xml.
>> yarn.xml will not be added to $CLASSPATH
>>
>> Thanks,
>> Wangda
>>
>>
>> On Wed, May 28, 2014 at 8:56 AM, hari <ha...@gmail.com> wrote:
>>
>>> The issue was not related the configuration related to containers. Due
>>> to
>>> misconfiguration, the Application master was not able to contact
>>> resourcemanager
>>> causing in the 1 container problem.
>>>
>>> However, the total containers allocated still is not as expected. The
>>> configuration settings
>>> should have resulted in 16 containers per node, but it is allocating 64
>>> containers per node.
>>>
>>> Reiterating the config parameters here again:
>>>
>>> mapred-site.xml
>>> mapreduce.map.cpu.vcores = 1
>>> mapreduce.reduce.cpu.vcores = 1
>>> mapreduce.map.memory.mb = 1024
>>> mapreduce.reduce.memory.mb = 1024
>>> mapreduce.map.java.opts = -Xmx1024m
>>> mapreduce.reduce.java.opts = -Xmx1024m
>>>
>>> yarn.xml
>>> yarn.nodemanager.resource.memory-mb = 65536
>>> yarn.nodemanager.resource.cpu-vcores = 16
>>> yarn.scheduler.minimum-allocation-mb = 1024
>>> yarn.scheduler.maximum-allocation-mb = 2048
>>> yarn.scheduler.minimum-allocation-vcores = 1
>>> yarn.scheduler.maximum-allocation-vcores = 1
>>>
>>> Is there anything else that might be causing this problem ?
>>>
>>> thanks,
>>> hari
>>>
>>>
>>>
>>>
>>>
>>> On Tue, May 27, 2014 at 3:31 AM, hari <ha...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> When using YARN 2.2.0 version, only 1 container is created
>>>> for an application in the entire cluster.
>>>> The single container is created at an arbitrary node
>>>> for every run. This happens when running any application from
>>>> the examples jar (e.g., wordcount). Currently only one application is
>>>> run at a time. The input datasize is > 200GB.
>>>>
>>>> I am setting custom values that affect concurrent container count.
>>>> These config parameters were mostly taken from:
>>>>
>>>> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>>>> These wasn't much description elsewhere on how the container count
>>>> would be
>>>> decided.
>>>>
>>>> The settings are:
>>>>
>>>> mapred-site.xml
>>>> mapreduce.map.cpu.vcores = 1
>>>> mapreduce.reduce.cpu.vcores = 1
>>>> mapreduce.map.memory.mb = 1024
>>>> mapreduce.reduce.memory.mb = 1024
>>>> mapreduce.map.java.opts = -Xmx1024m
>>>> mapreduce.reduce.java.opts = -Xmx1024m
>>>>
>>>> yarn.xml
>>>> yarn.nodemanager.resource.memory-mb = 65536
>>>> yarn.nodemanager.resource.cpu-vcores = 16
>>>>
>>>> From these settings, each node should be running 16 containers.
>>>>
>>>> Let me know if there might be something else affecting the container
>>>> count.
>>>>
>>>> thanks,
>>>> hari
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
Re: YARN creates only 1 container
Posted by hari <ha...@gmail.com>.
Just an update on this: turns out that in 2.2.0 version, the container
count is implemented by using the memory
resource, even though there are configs for vcores. Things might have
changed since
then.
Thanks for the suggestions. yarn.xml was a typo, should have been
yarn-site.xml.
About the heap size, it shouldn't be a problem unless the memory allocation
footprint of the
container exceeds or reaches close to the configured heap size. This
happened when
those settings were used. For other cases, the memory values were
different.
On Wed, Jul 2, 2014 at 6:27 PM, Adam Kawa <ka...@gmail.com> wrote:
> You also might want to increase values for mapreduce.{map,reduce}.memory.mb
> to 1280 or 1536 or so (assuming that mapreduce.{map,reduce}.java.opts =
> -Xmx1024m).
>
> mapreduce.{map,reduce}.memory.mb is logical size of the container and it
> should be larger than mapreduce.{map,reduce}.java.opts that specifies the
> heap for JVM running the actual task.
>
>
> 2014-06-27 5:43 GMT+02:00 Wangda Tan <wh...@gmail.com>:
>
> It should be yarn-site.xml not yarn.xml.
>> yarn.xml will not be added to $CLASSPATH
>>
>> Thanks,
>> Wangda
>>
>>
>> On Wed, May 28, 2014 at 8:56 AM, hari <ha...@gmail.com> wrote:
>>
>>> The issue was not related the configuration related to containers. Due
>>> to
>>> misconfiguration, the Application master was not able to contact
>>> resourcemanager
>>> causing in the 1 container problem.
>>>
>>> However, the total containers allocated still is not as expected. The
>>> configuration settings
>>> should have resulted in 16 containers per node, but it is allocating 64
>>> containers per node.
>>>
>>> Reiterating the config parameters here again:
>>>
>>> mapred-site.xml
>>> mapreduce.map.cpu.vcores = 1
>>> mapreduce.reduce.cpu.vcores = 1
>>> mapreduce.map.memory.mb = 1024
>>> mapreduce.reduce.memory.mb = 1024
>>> mapreduce.map.java.opts = -Xmx1024m
>>> mapreduce.reduce.java.opts = -Xmx1024m
>>>
>>> yarn.xml
>>> yarn.nodemanager.resource.memory-mb = 65536
>>> yarn.nodemanager.resource.cpu-vcores = 16
>>> yarn.scheduler.minimum-allocation-mb = 1024
>>> yarn.scheduler.maximum-allocation-mb = 2048
>>> yarn.scheduler.minimum-allocation-vcores = 1
>>> yarn.scheduler.maximum-allocation-vcores = 1
>>>
>>> Is there anything else that might be causing this problem ?
>>>
>>> thanks,
>>> hari
>>>
>>>
>>>
>>>
>>>
>>> On Tue, May 27, 2014 at 3:31 AM, hari <ha...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> When using YARN 2.2.0 version, only 1 container is created
>>>> for an application in the entire cluster.
>>>> The single container is created at an arbitrary node
>>>> for every run. This happens when running any application from
>>>> the examples jar (e.g., wordcount). Currently only one application is
>>>> run at a time. The input datasize is > 200GB.
>>>>
>>>> I am setting custom values that affect concurrent container count.
>>>> These config parameters were mostly taken from:
>>>>
>>>> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>>>> These wasn't much description elsewhere on how the container count
>>>> would be
>>>> decided.
>>>>
>>>> The settings are:
>>>>
>>>> mapred-site.xml
>>>> mapreduce.map.cpu.vcores = 1
>>>> mapreduce.reduce.cpu.vcores = 1
>>>> mapreduce.map.memory.mb = 1024
>>>> mapreduce.reduce.memory.mb = 1024
>>>> mapreduce.map.java.opts = -Xmx1024m
>>>> mapreduce.reduce.java.opts = -Xmx1024m
>>>>
>>>> yarn.xml
>>>> yarn.nodemanager.resource.memory-mb = 65536
>>>> yarn.nodemanager.resource.cpu-vcores = 16
>>>>
>>>> From these settings, each node should be running 16 containers.
>>>>
>>>> Let me know if there might be something else affecting the container
>>>> count.
>>>>
>>>> thanks,
>>>> hari
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
Re: YARN creates only 1 container
Posted by Adam Kawa <ka...@gmail.com>.
You also might want to increase values for mapreduce.{map,reduce}.memory.mb
to 1280 or 1536 or so (assuming that mapreduce.{map,reduce}.java.opts =
-Xmx1024m).
mapreduce.{map,reduce}.memory.mb is logical size of the container and it
should be larger than mapreduce.{map,reduce}.java.opts that specifies the
heap for JVM running the actual task.
2014-06-27 5:43 GMT+02:00 Wangda Tan <wh...@gmail.com>:
> It should be yarn-site.xml not yarn.xml.
> yarn.xml will not be added to $CLASSPATH
>
> Thanks,
> Wangda
>
>
> On Wed, May 28, 2014 at 8:56 AM, hari <ha...@gmail.com> wrote:
>
>> The issue was not related the configuration related to containers. Due to
>> misconfiguration, the Application master was not able to contact
>> resourcemanager
>> causing in the 1 container problem.
>>
>> However, the total containers allocated still is not as expected. The
>> configuration settings
>> should have resulted in 16 containers per node, but it is allocating 64
>> containers per node.
>>
>> Reiterating the config parameters here again:
>>
>> mapred-site.xml
>> mapreduce.map.cpu.vcores = 1
>> mapreduce.reduce.cpu.vcores = 1
>> mapreduce.map.memory.mb = 1024
>> mapreduce.reduce.memory.mb = 1024
>> mapreduce.map.java.opts = -Xmx1024m
>> mapreduce.reduce.java.opts = -Xmx1024m
>>
>> yarn.xml
>> yarn.nodemanager.resource.memory-mb = 65536
>> yarn.nodemanager.resource.cpu-vcores = 16
>> yarn.scheduler.minimum-allocation-mb = 1024
>> yarn.scheduler.maximum-allocation-mb = 2048
>> yarn.scheduler.minimum-allocation-vcores = 1
>> yarn.scheduler.maximum-allocation-vcores = 1
>>
>> Is there anything else that might be causing this problem ?
>>
>> thanks,
>> hari
>>
>>
>>
>>
>>
>> On Tue, May 27, 2014 at 3:31 AM, hari <ha...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> When using YARN 2.2.0 version, only 1 container is created
>>> for an application in the entire cluster.
>>> The single container is created at an arbitrary node
>>> for every run. This happens when running any application from
>>> the examples jar (e.g., wordcount). Currently only one application is
>>> run at a time. The input datasize is > 200GB.
>>>
>>> I am setting custom values that affect concurrent container count.
>>> These config parameters were mostly taken from:
>>>
>>> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>>> These wasn't much description elsewhere on how the container count would
>>> be
>>> decided.
>>>
>>> The settings are:
>>>
>>> mapred-site.xml
>>> mapreduce.map.cpu.vcores = 1
>>> mapreduce.reduce.cpu.vcores = 1
>>> mapreduce.map.memory.mb = 1024
>>> mapreduce.reduce.memory.mb = 1024
>>> mapreduce.map.java.opts = -Xmx1024m
>>> mapreduce.reduce.java.opts = -Xmx1024m
>>>
>>> yarn.xml
>>> yarn.nodemanager.resource.memory-mb = 65536
>>> yarn.nodemanager.resource.cpu-vcores = 16
>>>
>>> From these settings, each node should be running 16 containers.
>>>
>>> Let me know if there might be something else affecting the container
>>> count.
>>>
>>> thanks,
>>> hari
>>>
>>>
>>>
>>>
>>>
>>>
>>
>
Re: YARN creates only 1 container
Posted by Adam Kawa <ka...@gmail.com>.
You also might want to increase values for mapreduce.{map,reduce}.memory.mb
to 1280 or 1536 or so (assuming that mapreduce.{map,reduce}.java.opts =
-Xmx1024m).
mapreduce.{map,reduce}.memory.mb is logical size of the container and it
should be larger than mapreduce.{map,reduce}.java.opts that specifies the
heap for JVM running the actual task.
2014-06-27 5:43 GMT+02:00 Wangda Tan <wh...@gmail.com>:
> It should be yarn-site.xml not yarn.xml.
> yarn.xml will not be added to $CLASSPATH
>
> Thanks,
> Wangda
>
>
> On Wed, May 28, 2014 at 8:56 AM, hari <ha...@gmail.com> wrote:
>
>> The issue was not related the configuration related to containers. Due to
>> misconfiguration, the Application master was not able to contact
>> resourcemanager
>> causing in the 1 container problem.
>>
>> However, the total containers allocated still is not as expected. The
>> configuration settings
>> should have resulted in 16 containers per node, but it is allocating 64
>> containers per node.
>>
>> Reiterating the config parameters here again:
>>
>> mapred-site.xml
>> mapreduce.map.cpu.vcores = 1
>> mapreduce.reduce.cpu.vcores = 1
>> mapreduce.map.memory.mb = 1024
>> mapreduce.reduce.memory.mb = 1024
>> mapreduce.map.java.opts = -Xmx1024m
>> mapreduce.reduce.java.opts = -Xmx1024m
>>
>> yarn.xml
>> yarn.nodemanager.resource.memory-mb = 65536
>> yarn.nodemanager.resource.cpu-vcores = 16
>> yarn.scheduler.minimum-allocation-mb = 1024
>> yarn.scheduler.maximum-allocation-mb = 2048
>> yarn.scheduler.minimum-allocation-vcores = 1
>> yarn.scheduler.maximum-allocation-vcores = 1
>>
>> Is there anything else that might be causing this problem ?
>>
>> thanks,
>> hari
>>
>>
>>
>>
>>
>> On Tue, May 27, 2014 at 3:31 AM, hari <ha...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> When using YARN 2.2.0 version, only 1 container is created
>>> for an application in the entire cluster.
>>> The single container is created at an arbitrary node
>>> for every run. This happens when running any application from
>>> the examples jar (e.g., wordcount). Currently only one application is
>>> run at a time. The input datasize is > 200GB.
>>>
>>> I am setting custom values that affect concurrent container count.
>>> These config parameters were mostly taken from:
>>>
>>> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>>> These wasn't much description elsewhere on how the container count would
>>> be
>>> decided.
>>>
>>> The settings are:
>>>
>>> mapred-site.xml
>>> mapreduce.map.cpu.vcores = 1
>>> mapreduce.reduce.cpu.vcores = 1
>>> mapreduce.map.memory.mb = 1024
>>> mapreduce.reduce.memory.mb = 1024
>>> mapreduce.map.java.opts = -Xmx1024m
>>> mapreduce.reduce.java.opts = -Xmx1024m
>>>
>>> yarn.xml
>>> yarn.nodemanager.resource.memory-mb = 65536
>>> yarn.nodemanager.resource.cpu-vcores = 16
>>>
>>> From these settings, each node should be running 16 containers.
>>>
>>> Let me know if there might be something else affecting the container
>>> count.
>>>
>>> thanks,
>>> hari
>>>
>>>
>>>
>>>
>>>
>>>
>>
>
Re: YARN creates only 1 container
Posted by Adam Kawa <ka...@gmail.com>.
You also might want to increase values for mapreduce.{map,reduce}.memory.mb
to 1280 or 1536 or so (assuming that mapreduce.{map,reduce}.java.opts =
-Xmx1024m).
mapreduce.{map,reduce}.memory.mb is logical size of the container and it
should be larger than mapreduce.{map,reduce}.java.opts that specifies the
heap for JVM running the actual task.
2014-06-27 5:43 GMT+02:00 Wangda Tan <wh...@gmail.com>:
> It should be yarn-site.xml not yarn.xml.
> yarn.xml will not be added to $CLASSPATH
>
> Thanks,
> Wangda
>
>
> On Wed, May 28, 2014 at 8:56 AM, hari <ha...@gmail.com> wrote:
>
>> The issue was not related the configuration related to containers. Due to
>> misconfiguration, the Application master was not able to contact
>> resourcemanager
>> causing in the 1 container problem.
>>
>> However, the total containers allocated still is not as expected. The
>> configuration settings
>> should have resulted in 16 containers per node, but it is allocating 64
>> containers per node.
>>
>> Reiterating the config parameters here again:
>>
>> mapred-site.xml
>> mapreduce.map.cpu.vcores = 1
>> mapreduce.reduce.cpu.vcores = 1
>> mapreduce.map.memory.mb = 1024
>> mapreduce.reduce.memory.mb = 1024
>> mapreduce.map.java.opts = -Xmx1024m
>> mapreduce.reduce.java.opts = -Xmx1024m
>>
>> yarn.xml
>> yarn.nodemanager.resource.memory-mb = 65536
>> yarn.nodemanager.resource.cpu-vcores = 16
>> yarn.scheduler.minimum-allocation-mb = 1024
>> yarn.scheduler.maximum-allocation-mb = 2048
>> yarn.scheduler.minimum-allocation-vcores = 1
>> yarn.scheduler.maximum-allocation-vcores = 1
>>
>> Is there anything else that might be causing this problem ?
>>
>> thanks,
>> hari
>>
>>
>>
>>
>>
>> On Tue, May 27, 2014 at 3:31 AM, hari <ha...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> When using YARN 2.2.0 version, only 1 container is created
>>> for an application in the entire cluster.
>>> The single container is created at an arbitrary node
>>> for every run. This happens when running any application from
>>> the examples jar (e.g., wordcount). Currently only one application is
>>> run at a time. The input datasize is > 200GB.
>>>
>>> I am setting custom values that affect concurrent container count.
>>> These config parameters were mostly taken from:
>>>
>>> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>>> These wasn't much description elsewhere on how the container count would
>>> be
>>> decided.
>>>
>>> The settings are:
>>>
>>> mapred-site.xml
>>> mapreduce.map.cpu.vcores = 1
>>> mapreduce.reduce.cpu.vcores = 1
>>> mapreduce.map.memory.mb = 1024
>>> mapreduce.reduce.memory.mb = 1024
>>> mapreduce.map.java.opts = -Xmx1024m
>>> mapreduce.reduce.java.opts = -Xmx1024m
>>>
>>> yarn.xml
>>> yarn.nodemanager.resource.memory-mb = 65536
>>> yarn.nodemanager.resource.cpu-vcores = 16
>>>
>>> From these settings, each node should be running 16 containers.
>>>
>>> Let me know if there might be something else affecting the container
>>> count.
>>>
>>> thanks,
>>> hari
>>>
>>>
>>>
>>>
>>>
>>>
>>
>
Re: YARN creates only 1 container
Posted by Adam Kawa <ka...@gmail.com>.
You also might want to increase values for mapreduce.{map,reduce}.memory.mb
to 1280 or 1536 or so (assuming that mapreduce.{map,reduce}.java.opts =
-Xmx1024m).
mapreduce.{map,reduce}.memory.mb is logical size of the container and it
should be larger than mapreduce.{map,reduce}.java.opts that specifies the
heap for JVM running the actual task.
2014-06-27 5:43 GMT+02:00 Wangda Tan <wh...@gmail.com>:
> It should be yarn-site.xml not yarn.xml.
> yarn.xml will not be added to $CLASSPATH
>
> Thanks,
> Wangda
>
>
> On Wed, May 28, 2014 at 8:56 AM, hari <ha...@gmail.com> wrote:
>
>> The issue was not related the configuration related to containers. Due to
>> misconfiguration, the Application master was not able to contact
>> resourcemanager
>> causing in the 1 container problem.
>>
>> However, the total containers allocated still is not as expected. The
>> configuration settings
>> should have resulted in 16 containers per node, but it is allocating 64
>> containers per node.
>>
>> Reiterating the config parameters here again:
>>
>> mapred-site.xml
>> mapreduce.map.cpu.vcores = 1
>> mapreduce.reduce.cpu.vcores = 1
>> mapreduce.map.memory.mb = 1024
>> mapreduce.reduce.memory.mb = 1024
>> mapreduce.map.java.opts = -Xmx1024m
>> mapreduce.reduce.java.opts = -Xmx1024m
>>
>> yarn.xml
>> yarn.nodemanager.resource.memory-mb = 65536
>> yarn.nodemanager.resource.cpu-vcores = 16
>> yarn.scheduler.minimum-allocation-mb = 1024
>> yarn.scheduler.maximum-allocation-mb = 2048
>> yarn.scheduler.minimum-allocation-vcores = 1
>> yarn.scheduler.maximum-allocation-vcores = 1
>>
>> Is there anything else that might be causing this problem ?
>>
>> thanks,
>> hari
>>
>>
>>
>>
>>
>> On Tue, May 27, 2014 at 3:31 AM, hari <ha...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> When using YARN 2.2.0 version, only 1 container is created
>>> for an application in the entire cluster.
>>> The single container is created at an arbitrary node
>>> for every run. This happens when running any application from
>>> the examples jar (e.g., wordcount). Currently only one application is
>>> run at a time. The input datasize is > 200GB.
>>>
>>> I am setting custom values that affect concurrent container count.
>>> These config parameters were mostly taken from:
>>>
>>> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>>> These wasn't much description elsewhere on how the container count would
>>> be
>>> decided.
>>>
>>> The settings are:
>>>
>>> mapred-site.xml
>>> mapreduce.map.cpu.vcores = 1
>>> mapreduce.reduce.cpu.vcores = 1
>>> mapreduce.map.memory.mb = 1024
>>> mapreduce.reduce.memory.mb = 1024
>>> mapreduce.map.java.opts = -Xmx1024m
>>> mapreduce.reduce.java.opts = -Xmx1024m
>>>
>>> yarn.xml
>>> yarn.nodemanager.resource.memory-mb = 65536
>>> yarn.nodemanager.resource.cpu-vcores = 16
>>>
>>> From these settings, each node should be running 16 containers.
>>>
>>> Let me know if there might be something else affecting the container
>>> count.
>>>
>>> thanks,
>>> hari
>>>
>>>
>>>
>>>
>>>
>>>
>>
>
Re: YARN creates only 1 container
Posted by Wangda Tan <wh...@gmail.com>.
It should be yarn-site.xml not yarn.xml.
yarn.xml will not be added to $CLASSPATH
Thanks,
Wangda
On Wed, May 28, 2014 at 8:56 AM, hari <ha...@gmail.com> wrote:
> The issue was not related the configuration related to containers. Due to
> misconfiguration, the Application master was not able to contact
> resourcemanager
> causing in the 1 container problem.
>
> However, the total containers allocated still is not as expected. The
> configuration settings
> should have resulted in 16 containers per node, but it is allocating 64
> containers per node.
>
> Reiterating the config parameters here again:
>
> mapred-site.xml
> mapreduce.map.cpu.vcores = 1
> mapreduce.reduce.cpu.vcores = 1
> mapreduce.map.memory.mb = 1024
> mapreduce.reduce.memory.mb = 1024
> mapreduce.map.java.opts = -Xmx1024m
> mapreduce.reduce.java.opts = -Xmx1024m
>
> yarn.xml
> yarn.nodemanager.resource.memory-mb = 65536
> yarn.nodemanager.resource.cpu-vcores = 16
> yarn.scheduler.minimum-allocation-mb = 1024
> yarn.scheduler.maximum-allocation-mb = 2048
> yarn.scheduler.minimum-allocation-vcores = 1
> yarn.scheduler.maximum-allocation-vcores = 1
>
> Is there anything else that might be causing this problem ?
>
> thanks,
> hari
>
>
>
>
>
> On Tue, May 27, 2014 at 3:31 AM, hari <ha...@gmail.com> wrote:
>
>> Hi,
>>
>> When using YARN 2.2.0 version, only 1 container is created
>> for an application in the entire cluster.
>> The single container is created at an arbitrary node
>> for every run. This happens when running any application from
>> the examples jar (e.g., wordcount). Currently only one application is
>> run at a time. The input datasize is > 200GB.
>>
>> I am setting custom values that affect concurrent container count.
>> These config parameters were mostly taken from:
>>
>> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>> These wasn't much description elsewhere on how the container count would
>> be
>> decided.
>>
>> The settings are:
>>
>> mapred-site.xml
>> mapreduce.map.cpu.vcores = 1
>> mapreduce.reduce.cpu.vcores = 1
>> mapreduce.map.memory.mb = 1024
>> mapreduce.reduce.memory.mb = 1024
>> mapreduce.map.java.opts = -Xmx1024m
>> mapreduce.reduce.java.opts = -Xmx1024m
>>
>> yarn.xml
>> yarn.nodemanager.resource.memory-mb = 65536
>> yarn.nodemanager.resource.cpu-vcores = 16
>>
>> From these settings, each node should be running 16 containers.
>>
>> Let me know if there might be something else affecting the container
>> count.
>>
>> thanks,
>> hari
>>
>>
>>
>>
>>
>>
>
Re: YARN creates only 1 container
Posted by Wangda Tan <wh...@gmail.com>.
It should be yarn-site.xml not yarn.xml.
yarn.xml will not be added to $CLASSPATH
Thanks,
Wangda
On Wed, May 28, 2014 at 8:56 AM, hari <ha...@gmail.com> wrote:
> The issue was not related the configuration related to containers. Due to
> misconfiguration, the Application master was not able to contact
> resourcemanager
> causing in the 1 container problem.
>
> However, the total containers allocated still is not as expected. The
> configuration settings
> should have resulted in 16 containers per node, but it is allocating 64
> containers per node.
>
> Reiterating the config parameters here again:
>
> mapred-site.xml
> mapreduce.map.cpu.vcores = 1
> mapreduce.reduce.cpu.vcores = 1
> mapreduce.map.memory.mb = 1024
> mapreduce.reduce.memory.mb = 1024
> mapreduce.map.java.opts = -Xmx1024m
> mapreduce.reduce.java.opts = -Xmx1024m
>
> yarn.xml
> yarn.nodemanager.resource.memory-mb = 65536
> yarn.nodemanager.resource.cpu-vcores = 16
> yarn.scheduler.minimum-allocation-mb = 1024
> yarn.scheduler.maximum-allocation-mb = 2048
> yarn.scheduler.minimum-allocation-vcores = 1
> yarn.scheduler.maximum-allocation-vcores = 1
>
> Is there anything else that might be causing this problem ?
>
> thanks,
> hari
>
>
>
>
>
> On Tue, May 27, 2014 at 3:31 AM, hari <ha...@gmail.com> wrote:
>
>> Hi,
>>
>> When using YARN 2.2.0 version, only 1 container is created
>> for an application in the entire cluster.
>> The single container is created at an arbitrary node
>> for every run. This happens when running any application from
>> the examples jar (e.g., wordcount). Currently only one application is
>> run at a time. The input datasize is > 200GB.
>>
>> I am setting custom values that affect concurrent container count.
>> These config parameters were mostly taken from:
>>
>> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>> These wasn't much description elsewhere on how the container count would
>> be
>> decided.
>>
>> The settings are:
>>
>> mapred-site.xml
>> mapreduce.map.cpu.vcores = 1
>> mapreduce.reduce.cpu.vcores = 1
>> mapreduce.map.memory.mb = 1024
>> mapreduce.reduce.memory.mb = 1024
>> mapreduce.map.java.opts = -Xmx1024m
>> mapreduce.reduce.java.opts = -Xmx1024m
>>
>> yarn.xml
>> yarn.nodemanager.resource.memory-mb = 65536
>> yarn.nodemanager.resource.cpu-vcores = 16
>>
>> From these settings, each node should be running 16 containers.
>>
>> Let me know if there might be something else affecting the container
>> count.
>>
>> thanks,
>> hari
>>
>>
>>
>>
>>
>>
>
Re: YARN creates only 1 container
Posted by hari <ha...@gmail.com>.
Thanks. That might be reason why there are 64 containers instead of 16.
But, cgroups is not mentioned in a more recent article when discussing how
containers count is determined:
http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
.
It is also not mentioned when describing the vcore parameters:
http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
Should we consider with certainty that vcore parameters are not enforced if
cgroups are not configured ?
thanks,
hari
On Tue, May 27, 2014 at 10:17 PM, Pradeep Gollakota <pr...@gmail.com>wrote:
> I believe it's behaving as expected. It will spawn 64 containers because
> that's how much memory you have available. The vcores isn't harshly
> enforced since CPUs can be elastic. This blog from cloudera explain how to
> enforce CPU limits using CGroups.
> http://blog.cloudera.com/blog/2013/12/managing-multiple-resources-in-hadoop-2-with-yarn/
>
>
> On Tue, May 27, 2014 at 8:56 PM, hari <ha...@gmail.com> wrote:
>
>> The issue was not related the configuration related to containers. Due to
>> misconfiguration, the Application master was not able to contact
>> resourcemanager
>> causing in the 1 container problem.
>>
>> However, the total containers allocated still is not as expected. The
>> configuration settings
>> should have resulted in 16 containers per node, but it is allocating 64
>> containers per node.
>>
>> Reiterating the config parameters here again:
>>
>> mapred-site.xml
>> mapreduce.map.cpu.vcores = 1
>> mapreduce.reduce.cpu.vcores = 1
>> mapreduce.map.memory.mb = 1024
>> mapreduce.reduce.memory.mb = 1024
>> mapreduce.map.java.opts = -Xmx1024m
>> mapreduce.reduce.java.opts = -Xmx1024m
>>
>> yarn.xml
>> yarn.nodemanager.resource.memory-mb = 65536
>> yarn.nodemanager.resource.cpu-vcores = 16
>> yarn.scheduler.minimum-allocation-mb = 1024
>> yarn.scheduler.maximum-allocation-mb = 2048
>> yarn.scheduler.minimum-allocation-vcores = 1
>> yarn.scheduler.maximum-allocation-vcores = 1
>>
>> Is there anything else that might be causing this problem ?
>>
>> thanks,
>> hari
>>
>>
>>
>>
>>
>> On Tue, May 27, 2014 at 3:31 AM, hari <ha...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> When using YARN 2.2.0 version, only 1 container is created
>>> for an application in the entire cluster.
>>> The single container is created at an arbitrary node
>>> for every run. This happens when running any application from
>>> the examples jar (e.g., wordcount). Currently only one application is
>>> run at a time. The input datasize is > 200GB.
>>>
>>> I am setting custom values that affect concurrent container count.
>>> These config parameters were mostly taken from:
>>>
>>> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>>> These wasn't much description elsewhere on how the container count would
>>> be
>>> decided.
>>>
>>> The settings are:
>>>
>>> mapred-site.xml
>>> mapreduce.map.cpu.vcores = 1
>>> mapreduce.reduce.cpu.vcores = 1
>>> mapreduce.map.memory.mb = 1024
>>> mapreduce.reduce.memory.mb = 1024
>>> mapreduce.map.java.opts = -Xmx1024m
>>> mapreduce.reduce.java.opts = -Xmx1024m
>>>
>>> yarn.xml
>>> yarn.nodemanager.resource.memory-mb = 65536
>>> yarn.nodemanager.resource.cpu-vcores = 16
>>>
>>> From these settings, each node should be running 16 containers.
>>>
>>> Let me know if there might be something else affecting the container
>>> count.
>>>
>>> thanks,
>>> hari
>>>
>>>
>>>
>>>
>>>
>>>
>>
>
Re: YARN creates only 1 container
Posted by hari <ha...@gmail.com>.
Thanks. That might be reason why there are 64 containers instead of 16.
But, cgroups is not mentioned in a more recent article when discussing how
containers count is determined:
http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
.
It is also not mentioned when describing the vcore parameters:
http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
Should we consider with certainty that vcore parameters are not enforced if
cgroups are not configured ?
thanks,
hari
On Tue, May 27, 2014 at 10:17 PM, Pradeep Gollakota <pr...@gmail.com>wrote:
> I believe it's behaving as expected. It will spawn 64 containers because
> that's how much memory you have available. The vcores isn't harshly
> enforced since CPUs can be elastic. This blog from cloudera explain how to
> enforce CPU limits using CGroups.
> http://blog.cloudera.com/blog/2013/12/managing-multiple-resources-in-hadoop-2-with-yarn/
>
>
> On Tue, May 27, 2014 at 8:56 PM, hari <ha...@gmail.com> wrote:
>
>> The issue was not related the configuration related to containers. Due to
>> misconfiguration, the Application master was not able to contact
>> resourcemanager
>> causing in the 1 container problem.
>>
>> However, the total containers allocated still is not as expected. The
>> configuration settings
>> should have resulted in 16 containers per node, but it is allocating 64
>> containers per node.
>>
>> Reiterating the config parameters here again:
>>
>> mapred-site.xml
>> mapreduce.map.cpu.vcores = 1
>> mapreduce.reduce.cpu.vcores = 1
>> mapreduce.map.memory.mb = 1024
>> mapreduce.reduce.memory.mb = 1024
>> mapreduce.map.java.opts = -Xmx1024m
>> mapreduce.reduce.java.opts = -Xmx1024m
>>
>> yarn.xml
>> yarn.nodemanager.resource.memory-mb = 65536
>> yarn.nodemanager.resource.cpu-vcores = 16
>> yarn.scheduler.minimum-allocation-mb = 1024
>> yarn.scheduler.maximum-allocation-mb = 2048
>> yarn.scheduler.minimum-allocation-vcores = 1
>> yarn.scheduler.maximum-allocation-vcores = 1
>>
>> Is there anything else that might be causing this problem ?
>>
>> thanks,
>> hari
>>
>>
>>
>>
>>
>> On Tue, May 27, 2014 at 3:31 AM, hari <ha...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> When using YARN 2.2.0 version, only 1 container is created
>>> for an application in the entire cluster.
>>> The single container is created at an arbitrary node
>>> for every run. This happens when running any application from
>>> the examples jar (e.g., wordcount). Currently only one application is
>>> run at a time. The input datasize is > 200GB.
>>>
>>> I am setting custom values that affect concurrent container count.
>>> These config parameters were mostly taken from:
>>>
>>> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>>> These wasn't much description elsewhere on how the container count would
>>> be
>>> decided.
>>>
>>> The settings are:
>>>
>>> mapred-site.xml
>>> mapreduce.map.cpu.vcores = 1
>>> mapreduce.reduce.cpu.vcores = 1
>>> mapreduce.map.memory.mb = 1024
>>> mapreduce.reduce.memory.mb = 1024
>>> mapreduce.map.java.opts = -Xmx1024m
>>> mapreduce.reduce.java.opts = -Xmx1024m
>>>
>>> yarn.xml
>>> yarn.nodemanager.resource.memory-mb = 65536
>>> yarn.nodemanager.resource.cpu-vcores = 16
>>>
>>> From these settings, each node should be running 16 containers.
>>>
>>> Let me know if there might be something else affecting the container
>>> count.
>>>
>>> thanks,
>>> hari
>>>
>>>
>>>
>>>
>>>
>>>
>>
>
Re: YARN creates only 1 container
Posted by hari <ha...@gmail.com>.
Thanks. That might be reason why there are 64 containers instead of 16.
But, cgroups is not mentioned in a more recent article when discussing how
containers count is determined:
http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
.
It is also not mentioned when describing the vcore parameters:
http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
Should we consider with certainty that vcore parameters are not enforced if
cgroups are not configured ?
thanks,
hari
On Tue, May 27, 2014 at 10:17 PM, Pradeep Gollakota <pr...@gmail.com>wrote:
> I believe it's behaving as expected. It will spawn 64 containers because
> that's how much memory you have available. The vcores isn't harshly
> enforced since CPUs can be elastic. This blog from cloudera explain how to
> enforce CPU limits using CGroups.
> http://blog.cloudera.com/blog/2013/12/managing-multiple-resources-in-hadoop-2-with-yarn/
>
>
> On Tue, May 27, 2014 at 8:56 PM, hari <ha...@gmail.com> wrote:
>
>> The issue was not related the configuration related to containers. Due to
>> misconfiguration, the Application master was not able to contact
>> resourcemanager
>> causing in the 1 container problem.
>>
>> However, the total containers allocated still is not as expected. The
>> configuration settings
>> should have resulted in 16 containers per node, but it is allocating 64
>> containers per node.
>>
>> Reiterating the config parameters here again:
>>
>> mapred-site.xml
>> mapreduce.map.cpu.vcores = 1
>> mapreduce.reduce.cpu.vcores = 1
>> mapreduce.map.memory.mb = 1024
>> mapreduce.reduce.memory.mb = 1024
>> mapreduce.map.java.opts = -Xmx1024m
>> mapreduce.reduce.java.opts = -Xmx1024m
>>
>> yarn.xml
>> yarn.nodemanager.resource.memory-mb = 65536
>> yarn.nodemanager.resource.cpu-vcores = 16
>> yarn.scheduler.minimum-allocation-mb = 1024
>> yarn.scheduler.maximum-allocation-mb = 2048
>> yarn.scheduler.minimum-allocation-vcores = 1
>> yarn.scheduler.maximum-allocation-vcores = 1
>>
>> Is there anything else that might be causing this problem ?
>>
>> thanks,
>> hari
>>
>>
>>
>>
>>
>> On Tue, May 27, 2014 at 3:31 AM, hari <ha...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> When using YARN 2.2.0 version, only 1 container is created
>>> for an application in the entire cluster.
>>> The single container is created at an arbitrary node
>>> for every run. This happens when running any application from
>>> the examples jar (e.g., wordcount). Currently only one application is
>>> run at a time. The input datasize is > 200GB.
>>>
>>> I am setting custom values that affect concurrent container count.
>>> These config parameters were mostly taken from:
>>>
>>> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>>> These wasn't much description elsewhere on how the container count would
>>> be
>>> decided.
>>>
>>> The settings are:
>>>
>>> mapred-site.xml
>>> mapreduce.map.cpu.vcores = 1
>>> mapreduce.reduce.cpu.vcores = 1
>>> mapreduce.map.memory.mb = 1024
>>> mapreduce.reduce.memory.mb = 1024
>>> mapreduce.map.java.opts = -Xmx1024m
>>> mapreduce.reduce.java.opts = -Xmx1024m
>>>
>>> yarn.xml
>>> yarn.nodemanager.resource.memory-mb = 65536
>>> yarn.nodemanager.resource.cpu-vcores = 16
>>>
>>> From these settings, each node should be running 16 containers.
>>>
>>> Let me know if there might be something else affecting the container
>>> count.
>>>
>>> thanks,
>>> hari
>>>
>>>
>>>
>>>
>>>
>>>
>>
>
Re: YARN creates only 1 container
Posted by hari <ha...@gmail.com>.
Thanks. That might be reason why there are 64 containers instead of 16.
But, cgroups is not mentioned in a more recent article when discussing how
containers count is determined:
http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
.
It is also not mentioned when describing the vcore parameters:
http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
Should we consider with certainty that vcore parameters are not enforced if
cgroups are not configured ?
thanks,
hari
On Tue, May 27, 2014 at 10:17 PM, Pradeep Gollakota <pr...@gmail.com>wrote:
> I believe it's behaving as expected. It will spawn 64 containers because
> that's how much memory you have available. The vcores isn't harshly
> enforced since CPUs can be elastic. This blog from cloudera explain how to
> enforce CPU limits using CGroups.
> http://blog.cloudera.com/blog/2013/12/managing-multiple-resources-in-hadoop-2-with-yarn/
>
>
> On Tue, May 27, 2014 at 8:56 PM, hari <ha...@gmail.com> wrote:
>
>> The issue was not related the configuration related to containers. Due to
>> misconfiguration, the Application master was not able to contact
>> resourcemanager
>> causing in the 1 container problem.
>>
>> However, the total containers allocated still is not as expected. The
>> configuration settings
>> should have resulted in 16 containers per node, but it is allocating 64
>> containers per node.
>>
>> Reiterating the config parameters here again:
>>
>> mapred-site.xml
>> mapreduce.map.cpu.vcores = 1
>> mapreduce.reduce.cpu.vcores = 1
>> mapreduce.map.memory.mb = 1024
>> mapreduce.reduce.memory.mb = 1024
>> mapreduce.map.java.opts = -Xmx1024m
>> mapreduce.reduce.java.opts = -Xmx1024m
>>
>> yarn.xml
>> yarn.nodemanager.resource.memory-mb = 65536
>> yarn.nodemanager.resource.cpu-vcores = 16
>> yarn.scheduler.minimum-allocation-mb = 1024
>> yarn.scheduler.maximum-allocation-mb = 2048
>> yarn.scheduler.minimum-allocation-vcores = 1
>> yarn.scheduler.maximum-allocation-vcores = 1
>>
>> Is there anything else that might be causing this problem ?
>>
>> thanks,
>> hari
>>
>>
>>
>>
>>
>> On Tue, May 27, 2014 at 3:31 AM, hari <ha...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> When using YARN 2.2.0 version, only 1 container is created
>>> for an application in the entire cluster.
>>> The single container is created at an arbitrary node
>>> for every run. This happens when running any application from
>>> the examples jar (e.g., wordcount). Currently only one application is
>>> run at a time. The input datasize is > 200GB.
>>>
>>> I am setting custom values that affect concurrent container count.
>>> These config parameters were mostly taken from:
>>>
>>> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>>> These wasn't much description elsewhere on how the container count would
>>> be
>>> decided.
>>>
>>> The settings are:
>>>
>>> mapred-site.xml
>>> mapreduce.map.cpu.vcores = 1
>>> mapreduce.reduce.cpu.vcores = 1
>>> mapreduce.map.memory.mb = 1024
>>> mapreduce.reduce.memory.mb = 1024
>>> mapreduce.map.java.opts = -Xmx1024m
>>> mapreduce.reduce.java.opts = -Xmx1024m
>>>
>>> yarn.xml
>>> yarn.nodemanager.resource.memory-mb = 65536
>>> yarn.nodemanager.resource.cpu-vcores = 16
>>>
>>> From these settings, each node should be running 16 containers.
>>>
>>> Let me know if there might be something else affecting the container
>>> count.
>>>
>>> thanks,
>>> hari
>>>
>>>
>>>
>>>
>>>
>>>
>>
>
Re: YARN creates only 1 container
Posted by Pradeep Gollakota <pr...@gmail.com>.
I believe it's behaving as expected. It will spawn 64 containers because
that's how much memory you have available. The vcores isn't harshly
enforced since CPUs can be elastic. This blog from cloudera explain how to
enforce CPU limits using CGroups.
http://blog.cloudera.com/blog/2013/12/managing-multiple-resources-in-hadoop-2-with-yarn/
On Tue, May 27, 2014 at 8:56 PM, hari <ha...@gmail.com> wrote:
> The issue was not related the configuration related to containers. Due to
> misconfiguration, the Application master was not able to contact
> resourcemanager
> causing in the 1 container problem.
>
> However, the total containers allocated still is not as expected. The
> configuration settings
> should have resulted in 16 containers per node, but it is allocating 64
> containers per node.
>
> Reiterating the config parameters here again:
>
> mapred-site.xml
> mapreduce.map.cpu.vcores = 1
> mapreduce.reduce.cpu.vcores = 1
> mapreduce.map.memory.mb = 1024
> mapreduce.reduce.memory.mb = 1024
> mapreduce.map.java.opts = -Xmx1024m
> mapreduce.reduce.java.opts = -Xmx1024m
>
> yarn.xml
> yarn.nodemanager.resource.memory-mb = 65536
> yarn.nodemanager.resource.cpu-vcores = 16
> yarn.scheduler.minimum-allocation-mb = 1024
> yarn.scheduler.maximum-allocation-mb = 2048
> yarn.scheduler.minimum-allocation-vcores = 1
> yarn.scheduler.maximum-allocation-vcores = 1
>
> Is there anything else that might be causing this problem ?
>
> thanks,
> hari
>
>
>
>
>
> On Tue, May 27, 2014 at 3:31 AM, hari <ha...@gmail.com> wrote:
>
>> Hi,
>>
>> When using YARN 2.2.0 version, only 1 container is created
>> for an application in the entire cluster.
>> The single container is created at an arbitrary node
>> for every run. This happens when running any application from
>> the examples jar (e.g., wordcount). Currently only one application is
>> run at a time. The input datasize is > 200GB.
>>
>> I am setting custom values that affect concurrent container count.
>> These config parameters were mostly taken from:
>>
>> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>> These wasn't much description elsewhere on how the container count would
>> be
>> decided.
>>
>> The settings are:
>>
>> mapred-site.xml
>> mapreduce.map.cpu.vcores = 1
>> mapreduce.reduce.cpu.vcores = 1
>> mapreduce.map.memory.mb = 1024
>> mapreduce.reduce.memory.mb = 1024
>> mapreduce.map.java.opts = -Xmx1024m
>> mapreduce.reduce.java.opts = -Xmx1024m
>>
>> yarn.xml
>> yarn.nodemanager.resource.memory-mb = 65536
>> yarn.nodemanager.resource.cpu-vcores = 16
>>
>> From these settings, each node should be running 16 containers.
>>
>> Let me know if there might be something else affecting the container
>> count.
>>
>> thanks,
>> hari
>>
>>
>>
>>
>>
>>
>
Re: YARN creates only 1 container
Posted by Wangda Tan <wh...@gmail.com>.
It should be yarn-site.xml not yarn.xml.
yarn.xml will not be added to $CLASSPATH
Thanks,
Wangda
On Wed, May 28, 2014 at 8:56 AM, hari <ha...@gmail.com> wrote:
> The issue was not related the configuration related to containers. Due to
> misconfiguration, the Application master was not able to contact
> resourcemanager
> causing in the 1 container problem.
>
> However, the total containers allocated still is not as expected. The
> configuration settings
> should have resulted in 16 containers per node, but it is allocating 64
> containers per node.
>
> Reiterating the config parameters here again:
>
> mapred-site.xml
> mapreduce.map.cpu.vcores = 1
> mapreduce.reduce.cpu.vcores = 1
> mapreduce.map.memory.mb = 1024
> mapreduce.reduce.memory.mb = 1024
> mapreduce.map.java.opts = -Xmx1024m
> mapreduce.reduce.java.opts = -Xmx1024m
>
> yarn.xml
> yarn.nodemanager.resource.memory-mb = 65536
> yarn.nodemanager.resource.cpu-vcores = 16
> yarn.scheduler.minimum-allocation-mb = 1024
> yarn.scheduler.maximum-allocation-mb = 2048
> yarn.scheduler.minimum-allocation-vcores = 1
> yarn.scheduler.maximum-allocation-vcores = 1
>
> Is there anything else that might be causing this problem ?
>
> thanks,
> hari
>
>
>
>
>
> On Tue, May 27, 2014 at 3:31 AM, hari <ha...@gmail.com> wrote:
>
>> Hi,
>>
>> When using YARN 2.2.0 version, only 1 container is created
>> for an application in the entire cluster.
>> The single container is created at an arbitrary node
>> for every run. This happens when running any application from
>> the examples jar (e.g., wordcount). Currently only one application is
>> run at a time. The input datasize is > 200GB.
>>
>> I am setting custom values that affect concurrent container count.
>> These config parameters were mostly taken from:
>>
>> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>> These wasn't much description elsewhere on how the container count would
>> be
>> decided.
>>
>> The settings are:
>>
>> mapred-site.xml
>> mapreduce.map.cpu.vcores = 1
>> mapreduce.reduce.cpu.vcores = 1
>> mapreduce.map.memory.mb = 1024
>> mapreduce.reduce.memory.mb = 1024
>> mapreduce.map.java.opts = -Xmx1024m
>> mapreduce.reduce.java.opts = -Xmx1024m
>>
>> yarn.xml
>> yarn.nodemanager.resource.memory-mb = 65536
>> yarn.nodemanager.resource.cpu-vcores = 16
>>
>> From these settings, each node should be running 16 containers.
>>
>> Let me know if there might be something else affecting the container
>> count.
>>
>> thanks,
>> hari
>>
>>
>>
>>
>>
>>
>
Re: YARN creates only 1 container
Posted by Pradeep Gollakota <pr...@gmail.com>.
I believe it's behaving as expected. It will spawn 64 containers because
that's how much memory you have available. The vcores isn't harshly
enforced since CPUs can be elastic. This blog from cloudera explain how to
enforce CPU limits using CGroups.
http://blog.cloudera.com/blog/2013/12/managing-multiple-resources-in-hadoop-2-with-yarn/
On Tue, May 27, 2014 at 8:56 PM, hari <ha...@gmail.com> wrote:
> The issue was not related the configuration related to containers. Due to
> misconfiguration, the Application master was not able to contact
> resourcemanager
> causing in the 1 container problem.
>
> However, the total containers allocated still is not as expected. The
> configuration settings
> should have resulted in 16 containers per node, but it is allocating 64
> containers per node.
>
> Reiterating the config parameters here again:
>
> mapred-site.xml
> mapreduce.map.cpu.vcores = 1
> mapreduce.reduce.cpu.vcores = 1
> mapreduce.map.memory.mb = 1024
> mapreduce.reduce.memory.mb = 1024
> mapreduce.map.java.opts = -Xmx1024m
> mapreduce.reduce.java.opts = -Xmx1024m
>
> yarn.xml
> yarn.nodemanager.resource.memory-mb = 65536
> yarn.nodemanager.resource.cpu-vcores = 16
> yarn.scheduler.minimum-allocation-mb = 1024
> yarn.scheduler.maximum-allocation-mb = 2048
> yarn.scheduler.minimum-allocation-vcores = 1
> yarn.scheduler.maximum-allocation-vcores = 1
>
> Is there anything else that might be causing this problem ?
>
> thanks,
> hari
>
>
>
>
>
> On Tue, May 27, 2014 at 3:31 AM, hari <ha...@gmail.com> wrote:
>
>> Hi,
>>
>> When using YARN 2.2.0 version, only 1 container is created
>> for an application in the entire cluster.
>> The single container is created at an arbitrary node
>> for every run. This happens when running any application from
>> the examples jar (e.g., wordcount). Currently only one application is
>> run at a time. The input datasize is > 200GB.
>>
>> I am setting custom values that affect concurrent container count.
>> These config parameters were mostly taken from:
>>
>> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>> These wasn't much description elsewhere on how the container count would
>> be
>> decided.
>>
>> The settings are:
>>
>> mapred-site.xml
>> mapreduce.map.cpu.vcores = 1
>> mapreduce.reduce.cpu.vcores = 1
>> mapreduce.map.memory.mb = 1024
>> mapreduce.reduce.memory.mb = 1024
>> mapreduce.map.java.opts = -Xmx1024m
>> mapreduce.reduce.java.opts = -Xmx1024m
>>
>> yarn.xml
>> yarn.nodemanager.resource.memory-mb = 65536
>> yarn.nodemanager.resource.cpu-vcores = 16
>>
>> From these settings, each node should be running 16 containers.
>>
>> Let me know if there might be something else affecting the container
>> count.
>>
>> thanks,
>> hari
>>
>>
>>
>>
>>
>>
>
Re: YARN creates only 1 container
Posted by Pradeep Gollakota <pr...@gmail.com>.
I believe it's behaving as expected. It will spawn 64 containers because
that's how much memory you have available. The vcores isn't harshly
enforced since CPUs can be elastic. This blog from cloudera explain how to
enforce CPU limits using CGroups.
http://blog.cloudera.com/blog/2013/12/managing-multiple-resources-in-hadoop-2-with-yarn/
On Tue, May 27, 2014 at 8:56 PM, hari <ha...@gmail.com> wrote:
> The issue was not related the configuration related to containers. Due to
> misconfiguration, the Application master was not able to contact
> resourcemanager
> causing in the 1 container problem.
>
> However, the total containers allocated still is not as expected. The
> configuration settings
> should have resulted in 16 containers per node, but it is allocating 64
> containers per node.
>
> Reiterating the config parameters here again:
>
> mapred-site.xml
> mapreduce.map.cpu.vcores = 1
> mapreduce.reduce.cpu.vcores = 1
> mapreduce.map.memory.mb = 1024
> mapreduce.reduce.memory.mb = 1024
> mapreduce.map.java.opts = -Xmx1024m
> mapreduce.reduce.java.opts = -Xmx1024m
>
> yarn.xml
> yarn.nodemanager.resource.memory-mb = 65536
> yarn.nodemanager.resource.cpu-vcores = 16
> yarn.scheduler.minimum-allocation-mb = 1024
> yarn.scheduler.maximum-allocation-mb = 2048
> yarn.scheduler.minimum-allocation-vcores = 1
> yarn.scheduler.maximum-allocation-vcores = 1
>
> Is there anything else that might be causing this problem ?
>
> thanks,
> hari
>
>
>
>
>
> On Tue, May 27, 2014 at 3:31 AM, hari <ha...@gmail.com> wrote:
>
>> Hi,
>>
>> When using YARN 2.2.0 version, only 1 container is created
>> for an application in the entire cluster.
>> The single container is created at an arbitrary node
>> for every run. This happens when running any application from
>> the examples jar (e.g., wordcount). Currently only one application is
>> run at a time. The input datasize is > 200GB.
>>
>> I am setting custom values that affect concurrent container count.
>> These config parameters were mostly taken from:
>>
>> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>> These wasn't much description elsewhere on how the container count would
>> be
>> decided.
>>
>> The settings are:
>>
>> mapred-site.xml
>> mapreduce.map.cpu.vcores = 1
>> mapreduce.reduce.cpu.vcores = 1
>> mapreduce.map.memory.mb = 1024
>> mapreduce.reduce.memory.mb = 1024
>> mapreduce.map.java.opts = -Xmx1024m
>> mapreduce.reduce.java.opts = -Xmx1024m
>>
>> yarn.xml
>> yarn.nodemanager.resource.memory-mb = 65536
>> yarn.nodemanager.resource.cpu-vcores = 16
>>
>> From these settings, each node should be running 16 containers.
>>
>> Let me know if there might be something else affecting the container
>> count.
>>
>> thanks,
>> hari
>>
>>
>>
>>
>>
>>
>
Re: YARN creates only 1 container
Posted by Pradeep Gollakota <pr...@gmail.com>.
I believe it's behaving as expected. It will spawn 64 containers because
that's how much memory you have available. The vcores isn't harshly
enforced since CPUs can be elastic. This blog from cloudera explain how to
enforce CPU limits using CGroups.
http://blog.cloudera.com/blog/2013/12/managing-multiple-resources-in-hadoop-2-with-yarn/
On Tue, May 27, 2014 at 8:56 PM, hari <ha...@gmail.com> wrote:
> The issue was not related the configuration related to containers. Due to
> misconfiguration, the Application master was not able to contact
> resourcemanager
> causing in the 1 container problem.
>
> However, the total containers allocated still is not as expected. The
> configuration settings
> should have resulted in 16 containers per node, but it is allocating 64
> containers per node.
>
> Reiterating the config parameters here again:
>
> mapred-site.xml
> mapreduce.map.cpu.vcores = 1
> mapreduce.reduce.cpu.vcores = 1
> mapreduce.map.memory.mb = 1024
> mapreduce.reduce.memory.mb = 1024
> mapreduce.map.java.opts = -Xmx1024m
> mapreduce.reduce.java.opts = -Xmx1024m
>
> yarn.xml
> yarn.nodemanager.resource.memory-mb = 65536
> yarn.nodemanager.resource.cpu-vcores = 16
> yarn.scheduler.minimum-allocation-mb = 1024
> yarn.scheduler.maximum-allocation-mb = 2048
> yarn.scheduler.minimum-allocation-vcores = 1
> yarn.scheduler.maximum-allocation-vcores = 1
>
> Is there anything else that might be causing this problem ?
>
> thanks,
> hari
>
>
>
>
>
> On Tue, May 27, 2014 at 3:31 AM, hari <ha...@gmail.com> wrote:
>
>> Hi,
>>
>> When using YARN 2.2.0 version, only 1 container is created
>> for an application in the entire cluster.
>> The single container is created at an arbitrary node
>> for every run. This happens when running any application from
>> the examples jar (e.g., wordcount). Currently only one application is
>> run at a time. The input datasize is > 200GB.
>>
>> I am setting custom values that affect concurrent container count.
>> These config parameters were mostly taken from:
>>
>> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>> These wasn't much description elsewhere on how the container count would
>> be
>> decided.
>>
>> The settings are:
>>
>> mapred-site.xml
>> mapreduce.map.cpu.vcores = 1
>> mapreduce.reduce.cpu.vcores = 1
>> mapreduce.map.memory.mb = 1024
>> mapreduce.reduce.memory.mb = 1024
>> mapreduce.map.java.opts = -Xmx1024m
>> mapreduce.reduce.java.opts = -Xmx1024m
>>
>> yarn.xml
>> yarn.nodemanager.resource.memory-mb = 65536
>> yarn.nodemanager.resource.cpu-vcores = 16
>>
>> From these settings, each node should be running 16 containers.
>>
>> Let me know if there might be something else affecting the container
>> count.
>>
>> thanks,
>> hari
>>
>>
>>
>>
>>
>>
>
Re: YARN creates only 1 container
Posted by Wangda Tan <wh...@gmail.com>.
It should be yarn-site.xml not yarn.xml.
yarn.xml will not be added to $CLASSPATH
Thanks,
Wangda
On Wed, May 28, 2014 at 8:56 AM, hari <ha...@gmail.com> wrote:
> The issue was not related the configuration related to containers. Due to
> misconfiguration, the Application master was not able to contact
> resourcemanager
> causing in the 1 container problem.
>
> However, the total containers allocated still is not as expected. The
> configuration settings
> should have resulted in 16 containers per node, but it is allocating 64
> containers per node.
>
> Reiterating the config parameters here again:
>
> mapred-site.xml
> mapreduce.map.cpu.vcores = 1
> mapreduce.reduce.cpu.vcores = 1
> mapreduce.map.memory.mb = 1024
> mapreduce.reduce.memory.mb = 1024
> mapreduce.map.java.opts = -Xmx1024m
> mapreduce.reduce.java.opts = -Xmx1024m
>
> yarn.xml
> yarn.nodemanager.resource.memory-mb = 65536
> yarn.nodemanager.resource.cpu-vcores = 16
> yarn.scheduler.minimum-allocation-mb = 1024
> yarn.scheduler.maximum-allocation-mb = 2048
> yarn.scheduler.minimum-allocation-vcores = 1
> yarn.scheduler.maximum-allocation-vcores = 1
>
> Is there anything else that might be causing this problem ?
>
> thanks,
> hari
>
>
>
>
>
> On Tue, May 27, 2014 at 3:31 AM, hari <ha...@gmail.com> wrote:
>
>> Hi,
>>
>> When using YARN 2.2.0 version, only 1 container is created
>> for an application in the entire cluster.
>> The single container is created at an arbitrary node
>> for every run. This happens when running any application from
>> the examples jar (e.g., wordcount). Currently only one application is
>> run at a time. The input datasize is > 200GB.
>>
>> I am setting custom values that affect concurrent container count.
>> These config parameters were mostly taken from:
>>
>> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
>> These wasn't much description elsewhere on how the container count would
>> be
>> decided.
>>
>> The settings are:
>>
>> mapred-site.xml
>> mapreduce.map.cpu.vcores = 1
>> mapreduce.reduce.cpu.vcores = 1
>> mapreduce.map.memory.mb = 1024
>> mapreduce.reduce.memory.mb = 1024
>> mapreduce.map.java.opts = -Xmx1024m
>> mapreduce.reduce.java.opts = -Xmx1024m
>>
>> yarn.xml
>> yarn.nodemanager.resource.memory-mb = 65536
>> yarn.nodemanager.resource.cpu-vcores = 16
>>
>> From these settings, each node should be running 16 containers.
>>
>> Let me know if there might be something else affecting the container
>> count.
>>
>> thanks,
>> hari
>>
>>
>>
>>
>>
>>
>
Re: YARN creates only 1 container
Posted by hari <ha...@gmail.com>.
The issue was not related the configuration related to containers. Due to
misconfiguration, the Application master was not able to contact
resourcemanager
causing in the 1 container problem.
However, the total containers allocated still is not as expected. The
configuration settings
should have resulted in 16 containers per node, but it is allocating 64
containers per node.
Reiterating the config parameters here again:
mapred-site.xml
mapreduce.map.cpu.vcores = 1
mapreduce.reduce.cpu.vcores = 1
mapreduce.map.memory.mb = 1024
mapreduce.reduce.memory.mb = 1024
mapreduce.map.java.opts = -Xmx1024m
mapreduce.reduce.java.opts = -Xmx1024m
yarn.xml
yarn.nodemanager.resource.memory-mb = 65536
yarn.nodemanager.resource.cpu-vcores = 16
yarn.scheduler.minimum-allocation-mb = 1024
yarn.scheduler.maximum-allocation-mb = 2048
yarn.scheduler.minimum-allocation-vcores = 1
yarn.scheduler.maximum-allocation-vcores = 1
Is there anything else that might be causing this problem ?
thanks,
hari
On Tue, May 27, 2014 at 3:31 AM, hari <ha...@gmail.com> wrote:
> Hi,
>
> When using YARN 2.2.0 version, only 1 container is created
> for an application in the entire cluster.
> The single container is created at an arbitrary node
> for every run. This happens when running any application from
> the examples jar (e.g., wordcount). Currently only one application is
> run at a time. The input datasize is > 200GB.
>
> I am setting custom values that affect concurrent container count.
> These config parameters were mostly taken from:
>
> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
> These wasn't much description elsewhere on how the container count would
> be
> decided.
>
> The settings are:
>
> mapred-site.xml
> mapreduce.map.cpu.vcores = 1
> mapreduce.reduce.cpu.vcores = 1
> mapreduce.map.memory.mb = 1024
> mapreduce.reduce.memory.mb = 1024
> mapreduce.map.java.opts = -Xmx1024m
> mapreduce.reduce.java.opts = -Xmx1024m
>
> yarn.xml
> yarn.nodemanager.resource.memory-mb = 65536
> yarn.nodemanager.resource.cpu-vcores = 16
>
> From these settings, each node should be running 16 containers.
>
> Let me know if there might be something else affecting the container
> count.
>
> thanks,
> hari
>
>
>
>
>
>
Re: YARN creates only 1 container
Posted by hari <ha...@gmail.com>.
The issue was not related the configuration related to containers. Due to
misconfiguration, the Application master was not able to contact
resourcemanager
causing in the 1 container problem.
However, the total containers allocated still is not as expected. The
configuration settings
should have resulted in 16 containers per node, but it is allocating 64
containers per node.
Reiterating the config parameters here again:
mapred-site.xml
mapreduce.map.cpu.vcores = 1
mapreduce.reduce.cpu.vcores = 1
mapreduce.map.memory.mb = 1024
mapreduce.reduce.memory.mb = 1024
mapreduce.map.java.opts = -Xmx1024m
mapreduce.reduce.java.opts = -Xmx1024m
yarn.xml
yarn.nodemanager.resource.memory-mb = 65536
yarn.nodemanager.resource.cpu-vcores = 16
yarn.scheduler.minimum-allocation-mb = 1024
yarn.scheduler.maximum-allocation-mb = 2048
yarn.scheduler.minimum-allocation-vcores = 1
yarn.scheduler.maximum-allocation-vcores = 1
Is there anything else that might be causing this problem ?
thanks,
hari
On Tue, May 27, 2014 at 3:31 AM, hari <ha...@gmail.com> wrote:
> Hi,
>
> When using YARN 2.2.0 version, only 1 container is created
> for an application in the entire cluster.
> The single container is created at an arbitrary node
> for every run. This happens when running any application from
> the examples jar (e.g., wordcount). Currently only one application is
> run at a time. The input datasize is > 200GB.
>
> I am setting custom values that affect concurrent container count.
> These config parameters were mostly taken from:
>
> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
> These wasn't much description elsewhere on how the container count would
> be
> decided.
>
> The settings are:
>
> mapred-site.xml
> mapreduce.map.cpu.vcores = 1
> mapreduce.reduce.cpu.vcores = 1
> mapreduce.map.memory.mb = 1024
> mapreduce.reduce.memory.mb = 1024
> mapreduce.map.java.opts = -Xmx1024m
> mapreduce.reduce.java.opts = -Xmx1024m
>
> yarn.xml
> yarn.nodemanager.resource.memory-mb = 65536
> yarn.nodemanager.resource.cpu-vcores = 16
>
> From these settings, each node should be running 16 containers.
>
> Let me know if there might be something else affecting the container
> count.
>
> thanks,
> hari
>
>
>
>
>
>
Re: YARN creates only 1 container
Posted by hari <ha...@gmail.com>.
The issue was not related the configuration related to containers. Due to
misconfiguration, the Application master was not able to contact
resourcemanager
causing in the 1 container problem.
However, the total containers allocated still is not as expected. The
configuration settings
should have resulted in 16 containers per node, but it is allocating 64
containers per node.
Reiterating the config parameters here again:
mapred-site.xml
mapreduce.map.cpu.vcores = 1
mapreduce.reduce.cpu.vcores = 1
mapreduce.map.memory.mb = 1024
mapreduce.reduce.memory.mb = 1024
mapreduce.map.java.opts = -Xmx1024m
mapreduce.reduce.java.opts = -Xmx1024m
yarn.xml
yarn.nodemanager.resource.memory-mb = 65536
yarn.nodemanager.resource.cpu-vcores = 16
yarn.scheduler.minimum-allocation-mb = 1024
yarn.scheduler.maximum-allocation-mb = 2048
yarn.scheduler.minimum-allocation-vcores = 1
yarn.scheduler.maximum-allocation-vcores = 1
Is there anything else that might be causing this problem ?
thanks,
hari
On Tue, May 27, 2014 at 3:31 AM, hari <ha...@gmail.com> wrote:
> Hi,
>
> When using YARN 2.2.0 version, only 1 container is created
> for an application in the entire cluster.
> The single container is created at an arbitrary node
> for every run. This happens when running any application from
> the examples jar (e.g., wordcount). Currently only one application is
> run at a time. The input datasize is > 200GB.
>
> I am setting custom values that affect concurrent container count.
> These config parameters were mostly taken from:
>
> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
> These wasn't much description elsewhere on how the container count would
> be
> decided.
>
> The settings are:
>
> mapred-site.xml
> mapreduce.map.cpu.vcores = 1
> mapreduce.reduce.cpu.vcores = 1
> mapreduce.map.memory.mb = 1024
> mapreduce.reduce.memory.mb = 1024
> mapreduce.map.java.opts = -Xmx1024m
> mapreduce.reduce.java.opts = -Xmx1024m
>
> yarn.xml
> yarn.nodemanager.resource.memory-mb = 65536
> yarn.nodemanager.resource.cpu-vcores = 16
>
> From these settings, each node should be running 16 containers.
>
> Let me know if there might be something else affecting the container
> count.
>
> thanks,
> hari
>
>
>
>
>
>
Re: YARN creates only 1 container
Posted by hari <ha...@gmail.com>.
The issue was not related the configuration related to containers. Due to
misconfiguration, the Application master was not able to contact
resourcemanager
causing in the 1 container problem.
However, the total containers allocated still is not as expected. The
configuration settings
should have resulted in 16 containers per node, but it is allocating 64
containers per node.
Reiterating the config parameters here again:
mapred-site.xml
mapreduce.map.cpu.vcores = 1
mapreduce.reduce.cpu.vcores = 1
mapreduce.map.memory.mb = 1024
mapreduce.reduce.memory.mb = 1024
mapreduce.map.java.opts = -Xmx1024m
mapreduce.reduce.java.opts = -Xmx1024m
yarn.xml
yarn.nodemanager.resource.memory-mb = 65536
yarn.nodemanager.resource.cpu-vcores = 16
yarn.scheduler.minimum-allocation-mb = 1024
yarn.scheduler.maximum-allocation-mb = 2048
yarn.scheduler.minimum-allocation-vcores = 1
yarn.scheduler.maximum-allocation-vcores = 1
Is there anything else that might be causing this problem ?
thanks,
hari
On Tue, May 27, 2014 at 3:31 AM, hari <ha...@gmail.com> wrote:
> Hi,
>
> When using YARN 2.2.0 version, only 1 container is created
> for an application in the entire cluster.
> The single container is created at an arbitrary node
> for every run. This happens when running any application from
> the examples jar (e.g., wordcount). Currently only one application is
> run at a time. The input datasize is > 200GB.
>
> I am setting custom values that affect concurrent container count.
> These config parameters were mostly taken from:
>
> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/
> These wasn't much description elsewhere on how the container count would
> be
> decided.
>
> The settings are:
>
> mapred-site.xml
> mapreduce.map.cpu.vcores = 1
> mapreduce.reduce.cpu.vcores = 1
> mapreduce.map.memory.mb = 1024
> mapreduce.reduce.memory.mb = 1024
> mapreduce.map.java.opts = -Xmx1024m
> mapreduce.reduce.java.opts = -Xmx1024m
>
> yarn.xml
> yarn.nodemanager.resource.memory-mb = 65536
> yarn.nodemanager.resource.cpu-vcores = 16
>
> From these settings, each node should be running 16 containers.
>
> Let me know if there might be something else affecting the container
> count.
>
> thanks,
> hari
>
>
>
>
>
>