You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Malcolm McFarland <mm...@cavulus.com> on 2019/05/23 17:44:50 UTC
AM resource needs
Hey folks,
Are there any guidelines for how to provision an Application Master in
relation to the number of StreamTask instances it will be managing? Ie, are
there different memory, CPU, and thread-count figures for 100S StreamTasks
vs 1000, vs 10000?
Cheers,
Malcolm McFarland
Cavulus
This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
unauthorized or improper disclosure, copying, distribution, or use of the
contents of this message is prohibited. The information contained in this
message is intended only for the personal and confidential use of the
recipient(s) named above. If you have received this message in error,
please notify the sender immediately and delete the original message.
Re: AM resource needs
Posted by Jagadish Venkatraman <ja...@gmail.com>.
Yup, if that doesn't work for any reason, you can increase memory in 1G
increments.
Ensure that you have some head-room on top of your Xmx to account for native
memory
On Fri, May 24, 2019 at 10:34 AM Malcolm McFarland <mm...@cavulus.com>
wrote:
> Thanks Jagadish, just wanted to verify that this wasn't an obvious thing I
> was missing. I'm also setting the AM's heap size explicitly with the
> yarn.am.opts configuration parameter; in this case, I'm allowing 512MB for
> the JVM on top of my heap for both the AM and the container (ie,
> yarn.am.container.memory.mb=1536, yarn.am.opts=-Xmx1024m); does that sound
> reasonable?
>
> Cheers,
> Malcolm McFarland
> Cavulus
>
>
> This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
> unauthorized or improper disclosure, copying, distribution, or use of the
> contents of this message is prohibited. The information contained in this
> message is intended only for the personal and confidential use of the
> recipient(s) named above. If you have received this message in error,
> please notify the sender immediately and delete the original message.
>
> Malcolm McFarland
> Cavulus
>
>
> This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
> unauthorized or improper disclosure, copying, distribution, or use of the
> contents of this message is prohibited. The information contained in this
> message is intended only for the personal and confidential use of the
> recipient(s) named above. If you have received this message in error,
> please notify the sender immediately and delete the original message.
>
>
> On Thu, May 23, 2019 at 10:39 PM Jagadish Venkatraman <
> jagadish1989@gmail.com> wrote:
>
> > No, the default setup should be sufficient - the number of tasks should
> > have no significant impact on AM memory/resources.
> >
> > If you run out-of-memory, you can of course increase yarn.am.memory.mb.
> >
> > On Thu, May 23, 2019 at 10:45 AM Malcolm McFarland <
> mmcfarland@cavulus.com
> > >
> > wrote:
> >
> > > Hey folks,
> > >
> > > Are there any guidelines for how to provision an Application Master in
> > > relation to the number of StreamTask instances it will be managing? Ie,
> > are
> > > there different memory, CPU, and thread-count figures for 100S
> > StreamTasks
> > > vs 1000, vs 10000?
> > >
> > > Cheers,
> > > Malcolm McFarland
> > > Cavulus
> > >
> > >
> > > This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
> > > unauthorized or improper disclosure, copying, distribution, or use of
> the
> > > contents of this message is prohibited. The information contained in
> this
> > > message is intended only for the personal and confidential use of the
> > > recipient(s) named above. If you have received this message in error,
> > > please notify the sender immediately and delete the original message.
> > >
> >
> >
> > --
> > Jagadish V,
> > Graduate Student,
> > Department of Computer Science,
> > Stanford University
> >
>
--
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University
Re: AM resource needs
Posted by Malcolm McFarland <mm...@cavulus.com>.
Thanks Jagadish, just wanted to verify that this wasn't an obvious thing I
was missing. I'm also setting the AM's heap size explicitly with the
yarn.am.opts configuration parameter; in this case, I'm allowing 512MB for
the JVM on top of my heap for both the AM and the container (ie,
yarn.am.container.memory.mb=1536, yarn.am.opts=-Xmx1024m); does that sound
reasonable?
Cheers,
Malcolm McFarland
Cavulus
This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
unauthorized or improper disclosure, copying, distribution, or use of the
contents of this message is prohibited. The information contained in this
message is intended only for the personal and confidential use of the
recipient(s) named above. If you have received this message in error,
please notify the sender immediately and delete the original message.
Malcolm McFarland
Cavulus
This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
unauthorized or improper disclosure, copying, distribution, or use of the
contents of this message is prohibited. The information contained in this
message is intended only for the personal and confidential use of the
recipient(s) named above. If you have received this message in error,
please notify the sender immediately and delete the original message.
On Thu, May 23, 2019 at 10:39 PM Jagadish Venkatraman <
jagadish1989@gmail.com> wrote:
> No, the default setup should be sufficient - the number of tasks should
> have no significant impact on AM memory/resources.
>
> If you run out-of-memory, you can of course increase yarn.am.memory.mb.
>
> On Thu, May 23, 2019 at 10:45 AM Malcolm McFarland <mmcfarland@cavulus.com
> >
> wrote:
>
> > Hey folks,
> >
> > Are there any guidelines for how to provision an Application Master in
> > relation to the number of StreamTask instances it will be managing? Ie,
> are
> > there different memory, CPU, and thread-count figures for 100S
> StreamTasks
> > vs 1000, vs 10000?
> >
> > Cheers,
> > Malcolm McFarland
> > Cavulus
> >
> >
> > This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
> > unauthorized or improper disclosure, copying, distribution, or use of the
> > contents of this message is prohibited. The information contained in this
> > message is intended only for the personal and confidential use of the
> > recipient(s) named above. If you have received this message in error,
> > please notify the sender immediately and delete the original message.
> >
>
>
> --
> Jagadish V,
> Graduate Student,
> Department of Computer Science,
> Stanford University
>
Re: AM resource needs
Posted by Jagadish Venkatraman <ja...@gmail.com>.
No, the default setup should be sufficient - the number of tasks should
have no significant impact on AM memory/resources.
If you run out-of-memory, you can of course increase yarn.am.memory.mb.
On Thu, May 23, 2019 at 10:45 AM Malcolm McFarland <mm...@cavulus.com>
wrote:
> Hey folks,
>
> Are there any guidelines for how to provision an Application Master in
> relation to the number of StreamTask instances it will be managing? Ie, are
> there different memory, CPU, and thread-count figures for 100S StreamTasks
> vs 1000, vs 10000?
>
> Cheers,
> Malcolm McFarland
> Cavulus
>
>
> This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
> unauthorized or improper disclosure, copying, distribution, or use of the
> contents of this message is prohibited. The information contained in this
> message is intended only for the personal and confidential use of the
> recipient(s) named above. If you have received this message in error,
> please notify the sender immediately and delete the original message.
>
--
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University