You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by ricky l <ri...@gmail.com> on 2014/02/10 18:32:07 UTC
change the Yarn application container memory size when it is running
Hi All,
Can I change the allocated memory size of a container after a
container has started (running)? For example, in the beginning, I want
to allocate 1GB of memory to a container A and later I want to
allocate 2GB, and later 512MB. Is it a possible scenario? thx.
Re: change the Yarn application container memory size when it is running
Posted by ricky l <ri...@gmail.com>.
Thanks for the replies. I have an application that has irregular
memory usage (sometimes very high, but most of time small) pattern,
and I didn't want to allocate the resource based on the maximum usage.
After walking through the YARN code, It seems like I have to allocate
the resource based on the maximal usage. thx.
On Mon, Feb 10, 2014 at 1:37 PM, Pradeep Gollakota <pr...@gmail.com> wrote:
> I'm not sure I understand the use case for something like that. I'm pretty
> sure the YARN API doesn't support it though. What you might be able to do is
> to tear down your existing container and request a new one.
>
>
> On Mon, Feb 10, 2014 at 10:28 AM, Thomas Bentsen <th...@bentzn.com> wrote:
>>
>> I am no Yarn expert at all - in fact I have never run it. I am still an
>> absolute beginner with Hadoop.
>> But... Provided Yarn is running in _one_ VM (java process) and does not
>> fork processes on the OS you can not change the memsettings once it's
>> started.
>> But why would you want to change it? The JVM is able to use more or less
>> memory as needed if you set the startup params Xmx and Xms correctly.
>> Just be careful not to overallocate with Xmx in different processes at
>> the same time. Can do weird things when it starts using swap.
>>
>>
>> best
>> /th
>>
>>
>>
>>
>> On Mon, 2014-02-10 at 12:32 -0500, ricky l wrote:
>> > Hi All,
>> >
>> > Can I change the allocated memory size of a container after a
>> > container has started (running)? For example, in the beginning, I want
>> > to allocate 1GB of memory to a container A and later I want to
>> > allocate 2GB, and later 512MB. Is it a possible scenario? thx.
>>
>>
>
Re: change the Yarn application container memory size when it is running
Posted by ricky l <ri...@gmail.com>.
Thanks for the replies. I have an application that has irregular
memory usage (sometimes very high, but most of time small) pattern,
and I didn't want to allocate the resource based on the maximum usage.
After walking through the YARN code, It seems like I have to allocate
the resource based on the maximal usage. thx.
On Mon, Feb 10, 2014 at 1:37 PM, Pradeep Gollakota <pr...@gmail.com> wrote:
> I'm not sure I understand the use case for something like that. I'm pretty
> sure the YARN API doesn't support it though. What you might be able to do is
> to tear down your existing container and request a new one.
>
>
> On Mon, Feb 10, 2014 at 10:28 AM, Thomas Bentsen <th...@bentzn.com> wrote:
>>
>> I am no Yarn expert at all - in fact I have never run it. I am still an
>> absolute beginner with Hadoop.
>> But... Provided Yarn is running in _one_ VM (java process) and does not
>> fork processes on the OS you can not change the memsettings once it's
>> started.
>> But why would you want to change it? The JVM is able to use more or less
>> memory as needed if you set the startup params Xmx and Xms correctly.
>> Just be careful not to overallocate with Xmx in different processes at
>> the same time. Can do weird things when it starts using swap.
>>
>>
>> best
>> /th
>>
>>
>>
>>
>> On Mon, 2014-02-10 at 12:32 -0500, ricky l wrote:
>> > Hi All,
>> >
>> > Can I change the allocated memory size of a container after a
>> > container has started (running)? For example, in the beginning, I want
>> > to allocate 1GB of memory to a container A and later I want to
>> > allocate 2GB, and later 512MB. Is it a possible scenario? thx.
>>
>>
>
Re: change the Yarn application container memory size when it is running
Posted by ricky l <ri...@gmail.com>.
Thanks for the replies. I have an application that has irregular
memory usage (sometimes very high, but most of time small) pattern,
and I didn't want to allocate the resource based on the maximum usage.
After walking through the YARN code, It seems like I have to allocate
the resource based on the maximal usage. thx.
On Mon, Feb 10, 2014 at 1:37 PM, Pradeep Gollakota <pr...@gmail.com> wrote:
> I'm not sure I understand the use case for something like that. I'm pretty
> sure the YARN API doesn't support it though. What you might be able to do is
> to tear down your existing container and request a new one.
>
>
> On Mon, Feb 10, 2014 at 10:28 AM, Thomas Bentsen <th...@bentzn.com> wrote:
>>
>> I am no Yarn expert at all - in fact I have never run it. I am still an
>> absolute beginner with Hadoop.
>> But... Provided Yarn is running in _one_ VM (java process) and does not
>> fork processes on the OS you can not change the memsettings once it's
>> started.
>> But why would you want to change it? The JVM is able to use more or less
>> memory as needed if you set the startup params Xmx and Xms correctly.
>> Just be careful not to overallocate with Xmx in different processes at
>> the same time. Can do weird things when it starts using swap.
>>
>>
>> best
>> /th
>>
>>
>>
>>
>> On Mon, 2014-02-10 at 12:32 -0500, ricky l wrote:
>> > Hi All,
>> >
>> > Can I change the allocated memory size of a container after a
>> > container has started (running)? For example, in the beginning, I want
>> > to allocate 1GB of memory to a container A and later I want to
>> > allocate 2GB, and later 512MB. Is it a possible scenario? thx.
>>
>>
>
Re: change the Yarn application container memory size when it is running
Posted by ricky l <ri...@gmail.com>.
Thanks for the replies. I have an application that has irregular
memory usage (sometimes very high, but most of time small) pattern,
and I didn't want to allocate the resource based on the maximum usage.
After walking through the YARN code, It seems like I have to allocate
the resource based on the maximal usage. thx.
On Mon, Feb 10, 2014 at 1:37 PM, Pradeep Gollakota <pr...@gmail.com> wrote:
> I'm not sure I understand the use case for something like that. I'm pretty
> sure the YARN API doesn't support it though. What you might be able to do is
> to tear down your existing container and request a new one.
>
>
> On Mon, Feb 10, 2014 at 10:28 AM, Thomas Bentsen <th...@bentzn.com> wrote:
>>
>> I am no Yarn expert at all - in fact I have never run it. I am still an
>> absolute beginner with Hadoop.
>> But... Provided Yarn is running in _one_ VM (java process) and does not
>> fork processes on the OS you can not change the memsettings once it's
>> started.
>> But why would you want to change it? The JVM is able to use more or less
>> memory as needed if you set the startup params Xmx and Xms correctly.
>> Just be careful not to overallocate with Xmx in different processes at
>> the same time. Can do weird things when it starts using swap.
>>
>>
>> best
>> /th
>>
>>
>>
>>
>> On Mon, 2014-02-10 at 12:32 -0500, ricky l wrote:
>> > Hi All,
>> >
>> > Can I change the allocated memory size of a container after a
>> > container has started (running)? For example, in the beginning, I want
>> > to allocate 1GB of memory to a container A and later I want to
>> > allocate 2GB, and later 512MB. Is it a possible scenario? thx.
>>
>>
>
Re: change the Yarn application container memory size when it is running
Posted by Pradeep Gollakota <pr...@gmail.com>.
I'm not sure I understand the use case for something like that. I'm pretty
sure the YARN API doesn't support it though. What you might be able to do
is to tear down your existing container and request a new one.
On Mon, Feb 10, 2014 at 10:28 AM, Thomas Bentsen <th...@bentzn.com> wrote:
> I am no Yarn expert at all - in fact I have never run it. I am still an
> absolute beginner with Hadoop.
> But... Provided Yarn is running in _one_ VM (java process) and does not
> fork processes on the OS you can not change the memsettings once it's
> started.
> But why would you want to change it? The JVM is able to use more or less
> memory as needed if you set the startup params Xmx and Xms correctly.
> Just be careful not to overallocate with Xmx in different processes at
> the same time. Can do weird things when it starts using swap.
>
>
> best
> /th
>
>
>
>
> On Mon, 2014-02-10 at 12:32 -0500, ricky l wrote:
> > Hi All,
> >
> > Can I change the allocated memory size of a container after a
> > container has started (running)? For example, in the beginning, I want
> > to allocate 1GB of memory to a container A and later I want to
> > allocate 2GB, and later 512MB. Is it a possible scenario? thx.
>
>
>
Re: change the Yarn application container memory size when it is running
Posted by Pradeep Gollakota <pr...@gmail.com>.
I'm not sure I understand the use case for something like that. I'm pretty
sure the YARN API doesn't support it though. What you might be able to do
is to tear down your existing container and request a new one.
On Mon, Feb 10, 2014 at 10:28 AM, Thomas Bentsen <th...@bentzn.com> wrote:
> I am no Yarn expert at all - in fact I have never run it. I am still an
> absolute beginner with Hadoop.
> But... Provided Yarn is running in _one_ VM (java process) and does not
> fork processes on the OS you can not change the memsettings once it's
> started.
> But why would you want to change it? The JVM is able to use more or less
> memory as needed if you set the startup params Xmx and Xms correctly.
> Just be careful not to overallocate with Xmx in different processes at
> the same time. Can do weird things when it starts using swap.
>
>
> best
> /th
>
>
>
>
> On Mon, 2014-02-10 at 12:32 -0500, ricky l wrote:
> > Hi All,
> >
> > Can I change the allocated memory size of a container after a
> > container has started (running)? For example, in the beginning, I want
> > to allocate 1GB of memory to a container A and later I want to
> > allocate 2GB, and later 512MB. Is it a possible scenario? thx.
>
>
>
Re: change the Yarn application container memory size when it is running
Posted by Pradeep Gollakota <pr...@gmail.com>.
I'm not sure I understand the use case for something like that. I'm pretty
sure the YARN API doesn't support it though. What you might be able to do
is to tear down your existing container and request a new one.
On Mon, Feb 10, 2014 at 10:28 AM, Thomas Bentsen <th...@bentzn.com> wrote:
> I am no Yarn expert at all - in fact I have never run it. I am still an
> absolute beginner with Hadoop.
> But... Provided Yarn is running in _one_ VM (java process) and does not
> fork processes on the OS you can not change the memsettings once it's
> started.
> But why would you want to change it? The JVM is able to use more or less
> memory as needed if you set the startup params Xmx and Xms correctly.
> Just be careful not to overallocate with Xmx in different processes at
> the same time. Can do weird things when it starts using swap.
>
>
> best
> /th
>
>
>
>
> On Mon, 2014-02-10 at 12:32 -0500, ricky l wrote:
> > Hi All,
> >
> > Can I change the allocated memory size of a container after a
> > container has started (running)? For example, in the beginning, I want
> > to allocate 1GB of memory to a container A and later I want to
> > allocate 2GB, and later 512MB. Is it a possible scenario? thx.
>
>
>
Re: change the Yarn application container memory size when it is running
Posted by Pradeep Gollakota <pr...@gmail.com>.
I'm not sure I understand the use case for something like that. I'm pretty
sure the YARN API doesn't support it though. What you might be able to do
is to tear down your existing container and request a new one.
On Mon, Feb 10, 2014 at 10:28 AM, Thomas Bentsen <th...@bentzn.com> wrote:
> I am no Yarn expert at all - in fact I have never run it. I am still an
> absolute beginner with Hadoop.
> But... Provided Yarn is running in _one_ VM (java process) and does not
> fork processes on the OS you can not change the memsettings once it's
> started.
> But why would you want to change it? The JVM is able to use more or less
> memory as needed if you set the startup params Xmx and Xms correctly.
> Just be careful not to overallocate with Xmx in different processes at
> the same time. Can do weird things when it starts using swap.
>
>
> best
> /th
>
>
>
>
> On Mon, 2014-02-10 at 12:32 -0500, ricky l wrote:
> > Hi All,
> >
> > Can I change the allocated memory size of a container after a
> > container has started (running)? For example, in the beginning, I want
> > to allocate 1GB of memory to a container A and later I want to
> > allocate 2GB, and later 512MB. Is it a possible scenario? thx.
>
>
>
Re: change the Yarn application container memory size when it is
running
Posted by Thomas Bentsen <th...@bentzn.com>.
I am no Yarn expert at all - in fact I have never run it. I am still an
absolute beginner with Hadoop.
But... Provided Yarn is running in _one_ VM (java process) and does not
fork processes on the OS you can not change the memsettings once it's
started.
But why would you want to change it? The JVM is able to use more or less
memory as needed if you set the startup params Xmx and Xms correctly.
Just be careful not to overallocate with Xmx in different processes at
the same time. Can do weird things when it starts using swap.
best
/th
On Mon, 2014-02-10 at 12:32 -0500, ricky l wrote:
> Hi All,
>
> Can I change the allocated memory size of a container after a
> container has started (running)? For example, in the beginning, I want
> to allocate 1GB of memory to a container A and later I want to
> allocate 2GB, and later 512MB. Is it a possible scenario? thx.
Re: change the Yarn application container memory size when it is
running
Posted by Thomas Bentsen <th...@bentzn.com>.
I am no Yarn expert at all - in fact I have never run it. I am still an
absolute beginner with Hadoop.
But... Provided Yarn is running in _one_ VM (java process) and does not
fork processes on the OS you can not change the memsettings once it's
started.
But why would you want to change it? The JVM is able to use more or less
memory as needed if you set the startup params Xmx and Xms correctly.
Just be careful not to overallocate with Xmx in different processes at
the same time. Can do weird things when it starts using swap.
best
/th
On Mon, 2014-02-10 at 12:32 -0500, ricky l wrote:
> Hi All,
>
> Can I change the allocated memory size of a container after a
> container has started (running)? For example, in the beginning, I want
> to allocate 1GB of memory to a container A and later I want to
> allocate 2GB, and later 512MB. Is it a possible scenario? thx.
Re: change the Yarn application container memory size when it is
running
Posted by Thomas Bentsen <th...@bentzn.com>.
I am no Yarn expert at all - in fact I have never run it. I am still an
absolute beginner with Hadoop.
But... Provided Yarn is running in _one_ VM (java process) and does not
fork processes on the OS you can not change the memsettings once it's
started.
But why would you want to change it? The JVM is able to use more or less
memory as needed if you set the startup params Xmx and Xms correctly.
Just be careful not to overallocate with Xmx in different processes at
the same time. Can do weird things when it starts using swap.
best
/th
On Mon, 2014-02-10 at 12:32 -0500, ricky l wrote:
> Hi All,
>
> Can I change the allocated memory size of a container after a
> container has started (running)? For example, in the beginning, I want
> to allocate 1GB of memory to a container A and later I want to
> allocate 2GB, and later 512MB. Is it a possible scenario? thx.
Re: change the Yarn application container memory size when it is
running
Posted by Thomas Bentsen <th...@bentzn.com>.
I am no Yarn expert at all - in fact I have never run it. I am still an
absolute beginner with Hadoop.
But... Provided Yarn is running in _one_ VM (java process) and does not
fork processes on the OS you can not change the memsettings once it's
started.
But why would you want to change it? The JVM is able to use more or less
memory as needed if you set the startup params Xmx and Xms correctly.
Just be careful not to overallocate with Xmx in different processes at
the same time. Can do weird things when it starts using swap.
best
/th
On Mon, 2014-02-10 at 12:32 -0500, ricky l wrote:
> Hi All,
>
> Can I change the allocated memory size of a container after a
> container has started (running)? For example, in the beginning, I want
> to allocate 1GB of memory to a container A and later I want to
> allocate 2GB, and later 512MB. Is it a possible scenario? thx.