Java Opts and Max Heap

[Jordan Shaw <jo...@pubnub.com>]

Hey Everyone,
This I have a question somewhat related to SAMZA-109 and this line in
run-class.sh:
# Check if a max-heap size is specified. If not - set a 768M heap [[
$JAVA_OPTS != *-Xmx* ]] && JAVA_OPTS="$JAVA_OPTS -Xmx768M"

If I were to set the container.memory.mb for yarn to 4GB (
yarn.container.memory.mb = 4096) the above JAVA_OPTS check would cause this
to be ignored right? I can understand why preventing the Heap from going
crazy is important in these long running jobs but to me this might cause
some confusion especially when trying to debug a thrashing GC and Java
isn't committing the memory amount set in yarn.container.memory.mb.

I was trying to follow why this was decided on in SAMZA-109 but nothing
stuck out to me. Would it make more sense to just not specifying a max heap
and letting Yarn kill the job if it goes over it's specified allotment
and/or let the user explicitly set these opts?  Thanks!

- Jordan

Re: Java Opts and Max Heap

[Jordan Shaw <jo...@pubnub.com>]

Thanks for clarifying Chris!

On Tue, Mar 10, 2015 at 9:16 AM, Chris Riccomini <cr...@apache.org>
wrote:

> Hey Jordan,
>
> There are two settings you need to care about when changing memory. One is
> yarn.container.memory.mb. The other is the task.opts setting, which will
> allow you to specify a custom -Xmx parameter.
>
> If you set the yarn.container.memory.mb to 4000 (4GB), as you suggest, the
> JAVA_OPTS check still takes affect, and your Xmx will remain 768MB. If you
> wish to increase your heap, you'll need to set both
> yarn.container.memory.mb *and* the task.opts setting to be something like
> task.opts=-Xmx3g. This will tell YARN to kill any container that uses > 4GB
> of physical memory on the machine. The Xmx tells the container not to use
> more than 3GB of heap.
>
> It's a bit confusing, but there's a reason for this. The problem we have is
> that we (Samza) don't know how much off-heap/non-JVM memory you're going to
> use in your container. YARN only pays attention to the amount of physical
> memory used by a process. With Java, you can set the heap, but there's also
> permgen, JVM, JNI libraries, and off-heap memory usage. All of these
> contribute to the physical memory usage that YARN cares about, but are
> outside the JVM heap. This means that we can't just use one memory setting
> for both YARN and Java. We have to have two.
>
> Cheers,
> Chris
>
> On Tue, Mar 10, 2015 at 1:13 AM, Jordan Shaw <jo...@pubnub.com> wrote:
>
> > Hey Everyone,
> > This I have a question somewhat related to SAMZA-109 and this line in
> > run-class.sh:
> > # Check if a max-heap size is specified. If not - set a 768M heap [[
> > $JAVA_OPTS != *-Xmx* ]] && JAVA_OPTS="$JAVA_OPTS -Xmx768M"
> >
> > If I were to set the container.memory.mb for yarn to 4GB (
> > yarn.container.memory.mb = 4096) the above JAVA_OPTS check would cause
> this
> > to be ignored right? I can understand why preventing the Heap from going
> > crazy is important in these long running jobs but to me this might cause
> > some confusion especially when trying to debug a thrashing GC and Java
> > isn't committing the memory amount set in yarn.container.memory.mb.
> >
> > I was trying to follow why this was decided on in SAMZA-109 but nothing
> > stuck out to me. Would it make more sense to just not specifying a max
> heap
> > and letting Yarn kill the job if it goes over it's specified allotment
> > and/or let the user explicitly set these opts?  Thanks!
> >
> > - Jordan
> >
>



-- 
Jordan Shaw
Full Stack Software Engineer
PubNub Inc
1045 17th St
San Francisco, CA 94107

Re: Java Opts and Max Heap

[Chris Riccomini <cr...@apache.org>]

Hey Jordan,

There are two settings you need to care about when changing memory. One is
yarn.container.memory.mb. The other is the task.opts setting, which will
allow you to specify a custom -Xmx parameter.

If you set the yarn.container.memory.mb to 4000 (4GB), as you suggest, the
JAVA_OPTS check still takes affect, and your Xmx will remain 768MB. If you
wish to increase your heap, you'll need to set both
yarn.container.memory.mb *and* the task.opts setting to be something like
task.opts=-Xmx3g. This will tell YARN to kill any container that uses > 4GB
of physical memory on the machine. The Xmx tells the container not to use
more than 3GB of heap.

It's a bit confusing, but there's a reason for this. The problem we have is
that we (Samza) don't know how much off-heap/non-JVM memory you're going to
use in your container. YARN only pays attention to the amount of physical
memory used by a process. With Java, you can set the heap, but there's also
permgen, JVM, JNI libraries, and off-heap memory usage. All of these
contribute to the physical memory usage that YARN cares about, but are
outside the JVM heap. This means that we can't just use one memory setting
for both YARN and Java. We have to have two.

Cheers,
Chris

On Tue, Mar 10, 2015 at 1:13 AM, Jordan Shaw <jo...@pubnub.com> wrote:

> Hey Everyone,
> This I have a question somewhat related to SAMZA-109 and this line in
> run-class.sh:
> # Check if a max-heap size is specified. If not - set a 768M heap [[
> $JAVA_OPTS != *-Xmx* ]] && JAVA_OPTS="$JAVA_OPTS -Xmx768M"
>
> If I were to set the container.memory.mb for yarn to 4GB (
> yarn.container.memory.mb = 4096) the above JAVA_OPTS check would cause this
> to be ignored right? I can understand why preventing the Heap from going
> crazy is important in these long running jobs but to me this might cause
> some confusion especially when trying to debug a thrashing GC and Java
> isn't committing the memory amount set in yarn.container.memory.mb.
>
> I was trying to follow why this was decided on in SAMZA-109 but nothing
> stuck out to me. Would it make more sense to just not specifying a max heap
> and letting Yarn kill the job if it goes over it's specified allotment
> and/or let the user explicitly set these opts?  Thanks!
>
> - Jordan
>