Default App Master Container Memory Size

[Timothy Farkas <ti...@datatorrent.com>]

Hi All,

The default value for MASTER_MEMORY_MB is 1024. I've observed in multiple
demo apps and in the production apps of some of our users that 1024 MB is
not enough memory, and the app master dies. This was observed on small
applications with 5 - 20 containers. I think the default app master memory
size should be 2048.

Any feedback?

Thanks,
Tim

Re: Default App Master Container Memory Size

[Gaurav Gupta <ga...@datatorrent.com>]

Tim,

I will look into this. Can you give more details as to many operators the DAG should have and for how long long the app should run before hitting this issue?

Thanks
- Gaurav

> On Dec 28, 2015, at 11:54 AM, Timothy Farkas <ti...@datatorrent.com> wrote:
> 
> Gaurav,
> 
> If increasing the default is not an option, could you please look into the
> issue? Our users are getting burned by this and it's difficult to diagnose
> unless you've seen it before.
> 
> https://issues.apache.org/jira/browse/APEXCORE-296
> 
> Thanks,
> Tim
> 
> On Wed, Dec 23, 2015 at 10:46 PM, Aniruddha Thombare <
> aniruddha@datatorrent.com> wrote:
> 
>> Hi,
>> 
>> In my demo setups I have observed this multiple times that app master dies
>> on 1024MB memory.
>> 
>> So, I have set default app master to be at 2GB and since then haven't
>> observed the issue.
>> 
>> But, I would say that we should check why we need that much memory for app
>> master as Gaurav suggested.
>> 
>> Platform should use minimal possible memory and keep resources free for
>> actual applications.
>> 
>> Thanks,
>> 
>> Aniruddha
>> 
>> Sent from handheld device...
>> On Dec 24, 2015 11:23 AM, "Chinmay Kolhatkar" <ch...@datatorrent.com>
>> wrote:
>> 
>>> I agree with Gaurav.
>>> We should try to identify what is causing current memory limit not to
>> work.
>>> 
>>> If there is a need for a stop gap solution, we could temporarily make it
>>> higher. But I believe this is not a solution unless we answer the "why"
>>> question.
>>> 
>>> 
>>> ~ Chinmay.
>>> 
>>> On Thu, Dec 24, 2015 at 10:23 AM, Gaurav Gupta <ga...@datatorrent.com>
>>> wrote:
>>> 
>>>> Tim,
>>>> 
>>>> This setting used to work fine earlier. What did we change/add in App
>>>> master that is causing this behavior. We should know the reason before
>>>> bumping the memory. May be there is some other issue.
>>>> 
>>>> Thanks
>>>> - Gaurav
>>>> 
>>>>> On Dec 23, 2015, at 6:55 PM, Sandeep Deshmukh <
>> sandeep@datatorrent.com
>>>> 
>>>> wrote:
>>>>> 
>>>>> I am ok with 1280MB.
>>>>> On 24 Dec 2015 06:34, "Timothy Farkas" <ti...@datatorrent.com> wrote:
>>>>> 
>>>>>> If there are no objections I will open a pull request shortly to
>> bump
>>> up
>>>>>> the default App Master Memory to 1280 MB. I recently got burned by
>>> this
>>>>>> default AGAIN when the App Master randomly died in the middle of
>> demo.
>>>> The
>>>>>> demo application was small and only had 6 physical operators.
>>>>>> 
>>>>>> Thanks,
>>>>>> Tim
>>>>>> 
>>>>>> On Mon, Dec 14, 2015 at 2:01 PM, Munagala Ramanath <
>>> ram@datatorrent.com
>>>>> 
>>>>>> wrote:
>>>>>> 
>>>>>>> Sorry, meant to say 4-6GB, not MB.
>>>>>>> 
>>>>>>> Ram
>>>>>>> 
>>>>>>> On Mon, Dec 14, 2015 at 2:00 PM, Munagala Ramanath <
>>>> ram@datatorrent.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> I'm OK with increasing it to 1280MB.
>>>>>>>> 
>>>>>>>> I agree with Sandeep about new people trying out Apex,
>>>>>>>> often on limited environments like VMs limited to 4-6MB. With a
>>>> default
>>>>>>> of
>>>>>>>> 0.5GB per Buffer Server
>>>>>>>> and 1GB per operator, things start to fail when the user has more
>>> than
>>>>>> a
>>>>>>>> handful of operators.
>>>>>>>> 
>>>>>>>> I'm guessing it's not just the number of the operators that puts
>>>> memory
>>>>>>>> pressure on master memory
>>>>>>>> but the size of the jars that get pulled in by all the operator
>>>> classes
>>>>>>>> and their dependencies. If this is
>>>>>>>> true, one could cause failure even with a single, sufficiently
>>> complex
>>>>>>>> operator.
>>>>>>>> 
>>>>>>>> Ram
>>>>>>>> 
>>>>>>>> On Mon, Dec 14, 2015 at 1:41 PM, Timothy Farkas <
>>> tim@datatorrent.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hi Sandeep,
>>>>>>>>> 
>>>>>>>>> I have seen the issue even with small apps like Twitter Demo. It
>>>>>> happens
>>>>>>>>> less frequently in very small apps but the issue is still there.
>> It
>>>>>>>>> happens
>>>>>>>>> frequently in medium size apps (10 containers). Could we increase
>>> the
>>>>>>>>> MASTER_MEMORY_MB to 1280 instead of 2048? That way we can get
>>> greater
>>>>>>>>> stability with the defaults settings without allocating too much
>>>>>>>>> additional
>>>>>>>>> memory?
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Tim
>>>>>>>>> 
>>>>>>>>> On Sun, Dec 13, 2015 at 1:22 AM, Sandeep Deshmukh <
>>>>>>>>> sandeep@datatorrent.com>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> With large number of containers, the problem becomes severe and
>> I
>>>>>> too
>>>>>>>>> have
>>>>>>>>>> faced it.
>>>>>>>>>> 
>>>>>>>>>> Not sure if default should be increased as that should work in
>>> most
>>>>>> of
>>>>>>>>> the
>>>>>>>>>> cases and even when new people are trying out Apex for the first
>>>>>> time.
>>>>>>>>>> They may not have enough memory.
>>>>>>>>>> 
>>>>>>>>>> Regards
>>>>>>>>>> Sandeep
>>>>>>>>>> On 13 Dec 2015 11:30, "Ganelin, Ilya" <
>>> Ilya.Ganelin@capitalone.com>
>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Tim - great point, we reduce the memory consumption for
>> operators
>>>>>>> and
>>>>>>>>> do
>>>>>>>>>>> actually increase it for the app master, especially when we
>> have
>>>>>>> more
>>>>>>>>>>> operators. I misread your original email.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Thank you,
>>>>>>>>>>> Ilya Ganelin
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>> From: Timothy Farkas [tim@datatorrent.com<mailto:
>>>>>>> tim@datatorrent.com
>>>>>>>>>> ]
>>>>>>>>>>> Sent: Saturday, December 12, 2015 11:30 PM Eastern Standard
>> Time
>>>>>>>>>>> To: dev@apex.incubator.apache.org
>>>>>>>>>>> Subject: Re: Default App Master Container Memory Size
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Hi Ilya,
>>>>>>>>>>> 
>>>>>>>>>>> Dou you reduce the memory to 256 mb for operators only or for
>> the
>>>>>>> App
>>>>>>>>>>> Master as well?
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Tim
>>>>>>>>>>> 
>>>>>>>>>>> On Sat, Dec 12, 2015 at 8:06 PM, Ganelin, Ilya <
>>>>>>>>>>> Ilya.Ganelin@capitalone.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Tim - in our applications we've actually tried to go with
>>>>>> smaller
>>>>>>>>>>>> containers, as small as 256 MB, simply because many operators
>>>>>>> don't
>>>>>>>>>>> demand
>>>>>>>>>>>> any memory and this allows us to run more partitions.
>>>>>>>>>>>> 
>>>>>>>>>>>> We haven't noticed performance degradation.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Thank you,
>>>>>>>>>>>> Ilya Ganelin
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Timothy Farkas [tim@datatorrent.com<mailto:
>>>>>>>>> tim@datatorrent.com>]
>>>>>>>>>>>> Sent: Saturday, December 12, 2015 11:02 PM Eastern Standard
>> Time
>>>>>>>>>>>> To: dev@apex.incubator.apache.org
>>>>>>>>>>>> Subject: Default App Master Container Memory Size
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi All,
>>>>>>>>>>>> 
>>>>>>>>>>>> The default value for MASTER_MEMORY_MB is 1024. I've observed
>> in
>>>>>>>>>> multiple
>>>>>>>>>>>> demo apps and in the production apps of some of our users that
>>>>>>> 1024
>>>>>>>>> MB
>>>>>>>>>> is
>>>>>>>>>>>> not enough memory, and the app master dies. This was observed
>> on
>>>>>>>>> small
>>>>>>>>>>>> applications with 5 - 20 containers. I think the default app
>>>>>>> master
>>>>>>>>>>> memory
>>>>>>>>>>>> size should be 2048.
>>>>>>>>>>>> 
>>>>>>>>>>>> Any feedback?
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Tim
>>>>>>>>>>>> ________________________________________________________
>>>>>>>>>>>> 
>>>>>>>>>>>> The information contained in this e-mail is confidential
>> and/or
>>>>>>>>>>>> proprietary to Capital One and/or its affiliates and may only
>> be
>>>>>>>>> used
>>>>>>>>>>>> solely in performance of work or services for Capital One. The
>>>>>>>>>>> information
>>>>>>>>>>>> transmitted herewith is intended only for use by the
>> individual
>>>>>> or
>>>>>>>>>> entity
>>>>>>>>>>>> to which it is addressed. If the reader of this message is not
>>>>>> the
>>>>>>>>>>> intended
>>>>>>>>>>>> recipient, you are hereby notified that any review,
>>>>>>> retransmission,
>>>>>>>>>>>> dissemination, distribution, copying or other use of, or
>> taking
>>>>>> of
>>>>>>>>> any
>>>>>>>>>>>> action in reliance upon this information is strictly
>> prohibited.
>>>>>>> If
>>>>>>>>> you
>>>>>>>>>>>> have received this communication in error, please contact the
>>>>>>> sender
>>>>>>>>>> and
>>>>>>>>>>>> delete the material from your computer.
>>>>>>>>>>>> 
>>>>>>>>>>> ________________________________________________________
>>>>>>>>>>> 
>>>>>>>>>>> The information contained in this e-mail is confidential and/or
>>>>>>>>>>> proprietary to Capital One and/or its affiliates and may only
>> be
>>>>>>> used
>>>>>>>>>>> solely in performance of work or services for Capital One. The
>>>>>>>>>> information
>>>>>>>>>>> transmitted herewith is intended only for use by the individual
>>> or
>>>>>>>>> entity
>>>>>>>>>>> to which it is addressed. If the reader of this message is not
>>> the
>>>>>>>>>> intended
>>>>>>>>>>> recipient, you are hereby notified that any review,
>>>>>> retransmission,
>>>>>>>>>>> dissemination, distribution, copying or other use of, or taking
>>> of
>>>>>>> any
>>>>>>>>>>> action in reliance upon this information is strictly
>> prohibited.
>>>>>> If
>>>>>>>>> you
>>>>>>>>>>> have received this communication in error, please contact the
>>>>>> sender
>>>>>>>>> and
>>>>>>>>>>> delete the material from your computer.
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>> 
>> 

Re: Default App Master Container Memory Size

[Timothy Farkas <ti...@datatorrent.com>]

Gaurav,

If increasing the default is not an option, could you please look into the
issue? Our users are getting burned by this and it's difficult to diagnose
unless you've seen it before.

https://issues.apache.org/jira/browse/APEXCORE-296

Thanks,
Tim

On Wed, Dec 23, 2015 at 10:46 PM, Aniruddha Thombare <
aniruddha@datatorrent.com> wrote:

> Hi,
>
> In my demo setups I have observed this multiple times that app master dies
> on 1024MB memory.
>
> So, I have set default app master to be at 2GB and since then haven't
> observed the issue.
>
> But, I would say that we should check why we need that much memory for app
> master as Gaurav suggested.
>
> Platform should use minimal possible memory and keep resources free for
> actual applications.
>
> Thanks,
>
> Aniruddha
>
> Sent from handheld device...
> On Dec 24, 2015 11:23 AM, "Chinmay Kolhatkar" <ch...@datatorrent.com>
> wrote:
>
> > I agree with Gaurav.
> > We should try to identify what is causing current memory limit not to
> work.
> >
> > If there is a need for a stop gap solution, we could temporarily make it
> > higher. But I believe this is not a solution unless we answer the "why"
> > question.
> >
> >
> > ~ Chinmay.
> >
> > On Thu, Dec 24, 2015 at 10:23 AM, Gaurav Gupta <ga...@datatorrent.com>
> > wrote:
> >
> > > Tim,
> > >
> > > This setting used to work fine earlier. What did we change/add in App
> > > master that is causing this behavior. We should know the reason before
> > > bumping the memory. May be there is some other issue.
> > >
> > > Thanks
> > > - Gaurav
> > >
> > > > On Dec 23, 2015, at 6:55 PM, Sandeep Deshmukh <
> sandeep@datatorrent.com
> > >
> > > wrote:
> > > >
> > > > I am ok with 1280MB.
> > > > On 24 Dec 2015 06:34, "Timothy Farkas" <ti...@datatorrent.com> wrote:
> > > >
> > > >> If there are no objections I will open a pull request shortly to
> bump
> > up
> > > >> the default App Master Memory to 1280 MB. I recently got burned by
> > this
> > > >> default AGAIN when the App Master randomly died in the middle of
> demo.
> > > The
> > > >> demo application was small and only had 6 physical operators.
> > > >>
> > > >> Thanks,
> > > >> Tim
> > > >>
> > > >> On Mon, Dec 14, 2015 at 2:01 PM, Munagala Ramanath <
> > ram@datatorrent.com
> > > >
> > > >> wrote:
> > > >>
> > > >>> Sorry, meant to say 4-6GB, not MB.
> > > >>>
> > > >>> Ram
> > > >>>
> > > >>> On Mon, Dec 14, 2015 at 2:00 PM, Munagala Ramanath <
> > > ram@datatorrent.com>
> > > >>> wrote:
> > > >>>
> > > >>>> I'm OK with increasing it to 1280MB.
> > > >>>>
> > > >>>> I agree with Sandeep about new people trying out Apex,
> > > >>>> often on limited environments like VMs limited to 4-6MB. With a
> > > default
> > > >>> of
> > > >>>> 0.5GB per Buffer Server
> > > >>>> and 1GB per operator, things start to fail when the user has more
> > than
> > > >> a
> > > >>>> handful of operators.
> > > >>>>
> > > >>>> I'm guessing it's not just the number of the operators that puts
> > > memory
> > > >>>> pressure on master memory
> > > >>>> but the size of the jars that get pulled in by all the operator
> > > classes
> > > >>>> and their dependencies. If this is
> > > >>>> true, one could cause failure even with a single, sufficiently
> > complex
> > > >>>> operator.
> > > >>>>
> > > >>>> Ram
> > > >>>>
> > > >>>> On Mon, Dec 14, 2015 at 1:41 PM, Timothy Farkas <
> > tim@datatorrent.com>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Hi Sandeep,
> > > >>>>>
> > > >>>>> I have seen the issue even with small apps like Twitter Demo. It
> > > >> happens
> > > >>>>> less frequently in very small apps but the issue is still there.
> It
> > > >>>>> happens
> > > >>>>> frequently in medium size apps (10 containers). Could we increase
> > the
> > > >>>>> MASTER_MEMORY_MB to 1280 instead of 2048? That way we can get
> > greater
> > > >>>>> stability with the defaults settings without allocating too much
> > > >>>>> additional
> > > >>>>> memory?
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>> Tim
> > > >>>>>
> > > >>>>> On Sun, Dec 13, 2015 at 1:22 AM, Sandeep Deshmukh <
> > > >>>>> sandeep@datatorrent.com>
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> With large number of containers, the problem becomes severe and
> I
> > > >> too
> > > >>>>> have
> > > >>>>>> faced it.
> > > >>>>>>
> > > >>>>>> Not sure if default should be increased as that should work in
> > most
> > > >> of
> > > >>>>> the
> > > >>>>>> cases and even when new people are trying out Apex for the first
> > > >> time.
> > > >>>>>> They may not have enough memory.
> > > >>>>>>
> > > >>>>>> Regards
> > > >>>>>> Sandeep
> > > >>>>>> On 13 Dec 2015 11:30, "Ganelin, Ilya" <
> > Ilya.Ganelin@capitalone.com>
> > > >>>>> wrote:
> > > >>>>>>
> > > >>>>>>> Tim - great point, we reduce the memory consumption for
> operators
> > > >>> and
> > > >>>>> do
> > > >>>>>>> actually increase it for the app master, especially when we
> have
> > > >>> more
> > > >>>>>>> operators. I misread your original email.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> Thank you,
> > > >>>>>>> Ilya Ganelin
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> -----Original Message-----
> > > >>>>>>> From: Timothy Farkas [tim@datatorrent.com<mailto:
> > > >>> tim@datatorrent.com
> > > >>>>>> ]
> > > >>>>>>> Sent: Saturday, December 12, 2015 11:30 PM Eastern Standard
> Time
> > > >>>>>>> To: dev@apex.incubator.apache.org
> > > >>>>>>> Subject: Re: Default App Master Container Memory Size
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> Hi Ilya,
> > > >>>>>>>
> > > >>>>>>> Dou you reduce the memory to 256 mb for operators only or for
> the
> > > >>> App
> > > >>>>>>> Master as well?
> > > >>>>>>>
> > > >>>>>>> Thanks,
> > > >>>>>>> Tim
> > > >>>>>>>
> > > >>>>>>> On Sat, Dec 12, 2015 at 8:06 PM, Ganelin, Ilya <
> > > >>>>>>> Ilya.Ganelin@capitalone.com>
> > > >>>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> Tim - in our applications we've actually tried to go with
> > > >> smaller
> > > >>>>>>>> containers, as small as 256 MB, simply because many operators
> > > >>> don't
> > > >>>>>>> demand
> > > >>>>>>>> any memory and this allows us to run more partitions.
> > > >>>>>>>>
> > > >>>>>>>> We haven't noticed performance degradation.
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> Thank you,
> > > >>>>>>>> Ilya Ganelin
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> -----Original Message-----
> > > >>>>>>>> From: Timothy Farkas [tim@datatorrent.com<mailto:
> > > >>>>> tim@datatorrent.com>]
> > > >>>>>>>> Sent: Saturday, December 12, 2015 11:02 PM Eastern Standard
> Time
> > > >>>>>>>> To: dev@apex.incubator.apache.org
> > > >>>>>>>> Subject: Default App Master Container Memory Size
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> Hi All,
> > > >>>>>>>>
> > > >>>>>>>> The default value for MASTER_MEMORY_MB is 1024. I've observed
> in
> > > >>>>>> multiple
> > > >>>>>>>> demo apps and in the production apps of some of our users that
> > > >>> 1024
> > > >>>>> MB
> > > >>>>>> is
> > > >>>>>>>> not enough memory, and the app master dies. This was observed
> on
> > > >>>>> small
> > > >>>>>>>> applications with 5 - 20 containers. I think the default app
> > > >>> master
> > > >>>>>>> memory
> > > >>>>>>>> size should be 2048.
> > > >>>>>>>>
> > > >>>>>>>> Any feedback?
> > > >>>>>>>>
> > > >>>>>>>> Thanks,
> > > >>>>>>>> Tim
> > > >>>>>>>> ________________________________________________________
> > > >>>>>>>>
> > > >>>>>>>> The information contained in this e-mail is confidential
> and/or
> > > >>>>>>>> proprietary to Capital One and/or its affiliates and may only
> be
> > > >>>>> used
> > > >>>>>>>> solely in performance of work or services for Capital One. The
> > > >>>>>>> information
> > > >>>>>>>> transmitted herewith is intended only for use by the
> individual
> > > >> or
> > > >>>>>> entity
> > > >>>>>>>> to which it is addressed. If the reader of this message is not
> > > >> the
> > > >>>>>>> intended
> > > >>>>>>>> recipient, you are hereby notified that any review,
> > > >>> retransmission,
> > > >>>>>>>> dissemination, distribution, copying or other use of, or
> taking
> > > >> of
> > > >>>>> any
> > > >>>>>>>> action in reliance upon this information is strictly
> prohibited.
> > > >>> If
> > > >>>>> you
> > > >>>>>>>> have received this communication in error, please contact the
> > > >>> sender
> > > >>>>>> and
> > > >>>>>>>> delete the material from your computer.
> > > >>>>>>>>
> > > >>>>>>> ________________________________________________________
> > > >>>>>>>
> > > >>>>>>> The information contained in this e-mail is confidential and/or
> > > >>>>>>> proprietary to Capital One and/or its affiliates and may only
> be
> > > >>> used
> > > >>>>>>> solely in performance of work or services for Capital One. The
> > > >>>>>> information
> > > >>>>>>> transmitted herewith is intended only for use by the individual
> > or
> > > >>>>> entity
> > > >>>>>>> to which it is addressed. If the reader of this message is not
> > the
> > > >>>>>> intended
> > > >>>>>>> recipient, you are hereby notified that any review,
> > > >> retransmission,
> > > >>>>>>> dissemination, distribution, copying or other use of, or taking
> > of
> > > >>> any
> > > >>>>>>> action in reliance upon this information is strictly
> prohibited.
> > > >> If
> > > >>>>> you
> > > >>>>>>> have received this communication in error, please contact the
> > > >> sender
> > > >>>>> and
> > > >>>>>>> delete the material from your computer.
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> > >
> >
>

Re: Default App Master Container Memory Size

[Aniruddha Thombare <an...@datatorrent.com>]

Hi,

In my demo setups I have observed this multiple times that app master dies
on 1024MB memory.

So, I have set default app master to be at 2GB and since then haven't
observed the issue.

But, I would say that we should check why we need that much memory for app
master as Gaurav suggested.

Platform should use minimal possible memory and keep resources free for
actual applications.

Thanks,

Aniruddha

Sent from handheld device...
On Dec 24, 2015 11:23 AM, "Chinmay Kolhatkar" <ch...@datatorrent.com>
wrote:

> I agree with Gaurav.
> We should try to identify what is causing current memory limit not to work.
>
> If there is a need for a stop gap solution, we could temporarily make it
> higher. But I believe this is not a solution unless we answer the "why"
> question.
>
>
> ~ Chinmay.
>
> On Thu, Dec 24, 2015 at 10:23 AM, Gaurav Gupta <ga...@datatorrent.com>
> wrote:
>
> > Tim,
> >
> > This setting used to work fine earlier. What did we change/add in App
> > master that is causing this behavior. We should know the reason before
> > bumping the memory. May be there is some other issue.
> >
> > Thanks
> > - Gaurav
> >
> > > On Dec 23, 2015, at 6:55 PM, Sandeep Deshmukh <sandeep@datatorrent.com
> >
> > wrote:
> > >
> > > I am ok with 1280MB.
> > > On 24 Dec 2015 06:34, "Timothy Farkas" <ti...@datatorrent.com> wrote:
> > >
> > >> If there are no objections I will open a pull request shortly to bump
> up
> > >> the default App Master Memory to 1280 MB. I recently got burned by
> this
> > >> default AGAIN when the App Master randomly died in the middle of demo.
> > The
> > >> demo application was small and only had 6 physical operators.
> > >>
> > >> Thanks,
> > >> Tim
> > >>
> > >> On Mon, Dec 14, 2015 at 2:01 PM, Munagala Ramanath <
> ram@datatorrent.com
> > >
> > >> wrote:
> > >>
> > >>> Sorry, meant to say 4-6GB, not MB.
> > >>>
> > >>> Ram
> > >>>
> > >>> On Mon, Dec 14, 2015 at 2:00 PM, Munagala Ramanath <
> > ram@datatorrent.com>
> > >>> wrote:
> > >>>
> > >>>> I'm OK with increasing it to 1280MB.
> > >>>>
> > >>>> I agree with Sandeep about new people trying out Apex,
> > >>>> often on limited environments like VMs limited to 4-6MB. With a
> > default
> > >>> of
> > >>>> 0.5GB per Buffer Server
> > >>>> and 1GB per operator, things start to fail when the user has more
> than
> > >> a
> > >>>> handful of operators.
> > >>>>
> > >>>> I'm guessing it's not just the number of the operators that puts
> > memory
> > >>>> pressure on master memory
> > >>>> but the size of the jars that get pulled in by all the operator
> > classes
> > >>>> and their dependencies. If this is
> > >>>> true, one could cause failure even with a single, sufficiently
> complex
> > >>>> operator.
> > >>>>
> > >>>> Ram
> > >>>>
> > >>>> On Mon, Dec 14, 2015 at 1:41 PM, Timothy Farkas <
> tim@datatorrent.com>
> > >>>> wrote:
> > >>>>
> > >>>>> Hi Sandeep,
> > >>>>>
> > >>>>> I have seen the issue even with small apps like Twitter Demo. It
> > >> happens
> > >>>>> less frequently in very small apps but the issue is still there. It
> > >>>>> happens
> > >>>>> frequently in medium size apps (10 containers). Could we increase
> the
> > >>>>> MASTER_MEMORY_MB to 1280 instead of 2048? That way we can get
> greater
> > >>>>> stability with the defaults settings without allocating too much
> > >>>>> additional
> > >>>>> memory?
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Tim
> > >>>>>
> > >>>>> On Sun, Dec 13, 2015 at 1:22 AM, Sandeep Deshmukh <
> > >>>>> sandeep@datatorrent.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> With large number of containers, the problem becomes severe and I
> > >> too
> > >>>>> have
> > >>>>>> faced it.
> > >>>>>>
> > >>>>>> Not sure if default should be increased as that should work in
> most
> > >> of
> > >>>>> the
> > >>>>>> cases and even when new people are trying out Apex for the first
> > >> time.
> > >>>>>> They may not have enough memory.
> > >>>>>>
> > >>>>>> Regards
> > >>>>>> Sandeep
> > >>>>>> On 13 Dec 2015 11:30, "Ganelin, Ilya" <
> Ilya.Ganelin@capitalone.com>
> > >>>>> wrote:
> > >>>>>>
> > >>>>>>> Tim - great point, we reduce the memory consumption for operators
> > >>> and
> > >>>>> do
> > >>>>>>> actually increase it for the app master, especially when we have
> > >>> more
> > >>>>>>> operators. I misread your original email.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Thank you,
> > >>>>>>> Ilya Ganelin
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> -----Original Message-----
> > >>>>>>> From: Timothy Farkas [tim@datatorrent.com<mailto:
> > >>> tim@datatorrent.com
> > >>>>>> ]
> > >>>>>>> Sent: Saturday, December 12, 2015 11:30 PM Eastern Standard Time
> > >>>>>>> To: dev@apex.incubator.apache.org
> > >>>>>>> Subject: Re: Default App Master Container Memory Size
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Hi Ilya,
> > >>>>>>>
> > >>>>>>> Dou you reduce the memory to 256 mb for operators only or for the
> > >>> App
> > >>>>>>> Master as well?
> > >>>>>>>
> > >>>>>>> Thanks,
> > >>>>>>> Tim
> > >>>>>>>
> > >>>>>>> On Sat, Dec 12, 2015 at 8:06 PM, Ganelin, Ilya <
> > >>>>>>> Ilya.Ganelin@capitalone.com>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Tim - in our applications we've actually tried to go with
> > >> smaller
> > >>>>>>>> containers, as small as 256 MB, simply because many operators
> > >>> don't
> > >>>>>>> demand
> > >>>>>>>> any memory and this allows us to run more partitions.
> > >>>>>>>>
> > >>>>>>>> We haven't noticed performance degradation.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> Thank you,
> > >>>>>>>> Ilya Ganelin
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> -----Original Message-----
> > >>>>>>>> From: Timothy Farkas [tim@datatorrent.com<mailto:
> > >>>>> tim@datatorrent.com>]
> > >>>>>>>> Sent: Saturday, December 12, 2015 11:02 PM Eastern Standard Time
> > >>>>>>>> To: dev@apex.incubator.apache.org
> > >>>>>>>> Subject: Default App Master Container Memory Size
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> Hi All,
> > >>>>>>>>
> > >>>>>>>> The default value for MASTER_MEMORY_MB is 1024. I've observed in
> > >>>>>> multiple
> > >>>>>>>> demo apps and in the production apps of some of our users that
> > >>> 1024
> > >>>>> MB
> > >>>>>> is
> > >>>>>>>> not enough memory, and the app master dies. This was observed on
> > >>>>> small
> > >>>>>>>> applications with 5 - 20 containers. I think the default app
> > >>> master
> > >>>>>>> memory
> > >>>>>>>> size should be 2048.
> > >>>>>>>>
> > >>>>>>>> Any feedback?
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> Tim
> > >>>>>>>> ________________________________________________________
> > >>>>>>>>
> > >>>>>>>> The information contained in this e-mail is confidential and/or
> > >>>>>>>> proprietary to Capital One and/or its affiliates and may only be
> > >>>>> used
> > >>>>>>>> solely in performance of work or services for Capital One. The
> > >>>>>>> information
> > >>>>>>>> transmitted herewith is intended only for use by the individual
> > >> or
> > >>>>>> entity
> > >>>>>>>> to which it is addressed. If the reader of this message is not
> > >> the
> > >>>>>>> intended
> > >>>>>>>> recipient, you are hereby notified that any review,
> > >>> retransmission,
> > >>>>>>>> dissemination, distribution, copying or other use of, or taking
> > >> of
> > >>>>> any
> > >>>>>>>> action in reliance upon this information is strictly prohibited.
> > >>> If
> > >>>>> you
> > >>>>>>>> have received this communication in error, please contact the
> > >>> sender
> > >>>>>> and
> > >>>>>>>> delete the material from your computer.
> > >>>>>>>>
> > >>>>>>> ________________________________________________________
> > >>>>>>>
> > >>>>>>> The information contained in this e-mail is confidential and/or
> > >>>>>>> proprietary to Capital One and/or its affiliates and may only be
> > >>> used
> > >>>>>>> solely in performance of work or services for Capital One. The
> > >>>>>> information
> > >>>>>>> transmitted herewith is intended only for use by the individual
> or
> > >>>>> entity
> > >>>>>>> to which it is addressed. If the reader of this message is not
> the
> > >>>>>> intended
> > >>>>>>> recipient, you are hereby notified that any review,
> > >> retransmission,
> > >>>>>>> dissemination, distribution, copying or other use of, or taking
> of
> > >>> any
> > >>>>>>> action in reliance upon this information is strictly prohibited.
> > >> If
> > >>>>> you
> > >>>>>>> have received this communication in error, please contact the
> > >> sender
> > >>>>> and
> > >>>>>>> delete the material from your computer.
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> >
> >
>

Re: Default App Master Container Memory Size

[Chinmay Kolhatkar <ch...@datatorrent.com>]

I agree with Gaurav.
We should try to identify what is causing current memory limit not to work.

If there is a need for a stop gap solution, we could temporarily make it
higher. But I believe this is not a solution unless we answer the "why"
question.


~ Chinmay.

On Thu, Dec 24, 2015 at 10:23 AM, Gaurav Gupta <ga...@datatorrent.com>
wrote:

> Tim,
>
> This setting used to work fine earlier. What did we change/add in App
> master that is causing this behavior. We should know the reason before
> bumping the memory. May be there is some other issue.
>
> Thanks
> - Gaurav
>
> > On Dec 23, 2015, at 6:55 PM, Sandeep Deshmukh <sa...@datatorrent.com>
> wrote:
> >
> > I am ok with 1280MB.
> > On 24 Dec 2015 06:34, "Timothy Farkas" <ti...@datatorrent.com> wrote:
> >
> >> If there are no objections I will open a pull request shortly to bump up
> >> the default App Master Memory to 1280 MB. I recently got burned by this
> >> default AGAIN when the App Master randomly died in the middle of demo.
> The
> >> demo application was small and only had 6 physical operators.
> >>
> >> Thanks,
> >> Tim
> >>
> >> On Mon, Dec 14, 2015 at 2:01 PM, Munagala Ramanath <ram@datatorrent.com
> >
> >> wrote:
> >>
> >>> Sorry, meant to say 4-6GB, not MB.
> >>>
> >>> Ram
> >>>
> >>> On Mon, Dec 14, 2015 at 2:00 PM, Munagala Ramanath <
> ram@datatorrent.com>
> >>> wrote:
> >>>
> >>>> I'm OK with increasing it to 1280MB.
> >>>>
> >>>> I agree with Sandeep about new people trying out Apex,
> >>>> often on limited environments like VMs limited to 4-6MB. With a
> default
> >>> of
> >>>> 0.5GB per Buffer Server
> >>>> and 1GB per operator, things start to fail when the user has more than
> >> a
> >>>> handful of operators.
> >>>>
> >>>> I'm guessing it's not just the number of the operators that puts
> memory
> >>>> pressure on master memory
> >>>> but the size of the jars that get pulled in by all the operator
> classes
> >>>> and their dependencies. If this is
> >>>> true, one could cause failure even with a single, sufficiently complex
> >>>> operator.
> >>>>
> >>>> Ram
> >>>>
> >>>> On Mon, Dec 14, 2015 at 1:41 PM, Timothy Farkas <ti...@datatorrent.com>
> >>>> wrote:
> >>>>
> >>>>> Hi Sandeep,
> >>>>>
> >>>>> I have seen the issue even with small apps like Twitter Demo. It
> >> happens
> >>>>> less frequently in very small apps but the issue is still there. It
> >>>>> happens
> >>>>> frequently in medium size apps (10 containers). Could we increase the
> >>>>> MASTER_MEMORY_MB to 1280 instead of 2048? That way we can get greater
> >>>>> stability with the defaults settings without allocating too much
> >>>>> additional
> >>>>> memory?
> >>>>>
> >>>>> Thanks,
> >>>>> Tim
> >>>>>
> >>>>> On Sun, Dec 13, 2015 at 1:22 AM, Sandeep Deshmukh <
> >>>>> sandeep@datatorrent.com>
> >>>>> wrote:
> >>>>>
> >>>>>> With large number of containers, the problem becomes severe and I
> >> too
> >>>>> have
> >>>>>> faced it.
> >>>>>>
> >>>>>> Not sure if default should be increased as that should work in most
> >> of
> >>>>> the
> >>>>>> cases and even when new people are trying out Apex for the first
> >> time.
> >>>>>> They may not have enough memory.
> >>>>>>
> >>>>>> Regards
> >>>>>> Sandeep
> >>>>>> On 13 Dec 2015 11:30, "Ganelin, Ilya" <Il...@capitalone.com>
> >>>>> wrote:
> >>>>>>
> >>>>>>> Tim - great point, we reduce the memory consumption for operators
> >>> and
> >>>>> do
> >>>>>>> actually increase it for the app master, especially when we have
> >>> more
> >>>>>>> operators. I misread your original email.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Thank you,
> >>>>>>> Ilya Ganelin
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: Timothy Farkas [tim@datatorrent.com<mailto:
> >>> tim@datatorrent.com
> >>>>>> ]
> >>>>>>> Sent: Saturday, December 12, 2015 11:30 PM Eastern Standard Time
> >>>>>>> To: dev@apex.incubator.apache.org
> >>>>>>> Subject: Re: Default App Master Container Memory Size
> >>>>>>>
> >>>>>>>
> >>>>>>> Hi Ilya,
> >>>>>>>
> >>>>>>> Dou you reduce the memory to 256 mb for operators only or for the
> >>> App
> >>>>>>> Master as well?
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Tim
> >>>>>>>
> >>>>>>> On Sat, Dec 12, 2015 at 8:06 PM, Ganelin, Ilya <
> >>>>>>> Ilya.Ganelin@capitalone.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Tim - in our applications we've actually tried to go with
> >> smaller
> >>>>>>>> containers, as small as 256 MB, simply because many operators
> >>> don't
> >>>>>>> demand
> >>>>>>>> any memory and this allows us to run more partitions.
> >>>>>>>>
> >>>>>>>> We haven't noticed performance degradation.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Thank you,
> >>>>>>>> Ilya Ganelin
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> -----Original Message-----
> >>>>>>>> From: Timothy Farkas [tim@datatorrent.com<mailto:
> >>>>> tim@datatorrent.com>]
> >>>>>>>> Sent: Saturday, December 12, 2015 11:02 PM Eastern Standard Time
> >>>>>>>> To: dev@apex.incubator.apache.org
> >>>>>>>> Subject: Default App Master Container Memory Size
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Hi All,
> >>>>>>>>
> >>>>>>>> The default value for MASTER_MEMORY_MB is 1024. I've observed in
> >>>>>> multiple
> >>>>>>>> demo apps and in the production apps of some of our users that
> >>> 1024
> >>>>> MB
> >>>>>> is
> >>>>>>>> not enough memory, and the app master dies. This was observed on
> >>>>> small
> >>>>>>>> applications with 5 - 20 containers. I think the default app
> >>> master
> >>>>>>> memory
> >>>>>>>> size should be 2048.
> >>>>>>>>
> >>>>>>>> Any feedback?
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Tim
> >>>>>>>> ________________________________________________________
> >>>>>>>>
> >>>>>>>> The information contained in this e-mail is confidential and/or
> >>>>>>>> proprietary to Capital One and/or its affiliates and may only be
> >>>>> used
> >>>>>>>> solely in performance of work or services for Capital One. The
> >>>>>>> information
> >>>>>>>> transmitted herewith is intended only for use by the individual
> >> or
> >>>>>> entity
> >>>>>>>> to which it is addressed. If the reader of this message is not
> >> the
> >>>>>>> intended
> >>>>>>>> recipient, you are hereby notified that any review,
> >>> retransmission,
> >>>>>>>> dissemination, distribution, copying or other use of, or taking
> >> of
> >>>>> any
> >>>>>>>> action in reliance upon this information is strictly prohibited.
> >>> If
> >>>>> you
> >>>>>>>> have received this communication in error, please contact the
> >>> sender
> >>>>>> and
> >>>>>>>> delete the material from your computer.
> >>>>>>>>
> >>>>>>> ________________________________________________________
> >>>>>>>
> >>>>>>> The information contained in this e-mail is confidential and/or
> >>>>>>> proprietary to Capital One and/or its affiliates and may only be
> >>> used
> >>>>>>> solely in performance of work or services for Capital One. The
> >>>>>> information
> >>>>>>> transmitted herewith is intended only for use by the individual or
> >>>>> entity
> >>>>>>> to which it is addressed. If the reader of this message is not the
> >>>>>> intended
> >>>>>>> recipient, you are hereby notified that any review,
> >> retransmission,
> >>>>>>> dissemination, distribution, copying or other use of, or taking of
> >>> any
> >>>>>>> action in reliance upon this information is strictly prohibited.
> >> If
> >>>>> you
> >>>>>>> have received this communication in error, please contact the
> >> sender
> >>>>> and
> >>>>>>> delete the material from your computer.
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
>
>

Re: Default App Master Container Memory Size

[Gaurav Gupta <ga...@datatorrent.com>]

Tim,

This setting used to work fine earlier. What did we change/add in App master that is causing this behavior. We should know the reason before bumping the memory. May be there is some other issue.

Thanks
- Gaurav

> On Dec 23, 2015, at 6:55 PM, Sandeep Deshmukh <sa...@datatorrent.com> wrote:
> 
> I am ok with 1280MB.
> On 24 Dec 2015 06:34, "Timothy Farkas" <ti...@datatorrent.com> wrote:
> 
>> If there are no objections I will open a pull request shortly to bump up
>> the default App Master Memory to 1280 MB. I recently got burned by this
>> default AGAIN when the App Master randomly died in the middle of demo. The
>> demo application was small and only had 6 physical operators.
>> 
>> Thanks,
>> Tim
>> 
>> On Mon, Dec 14, 2015 at 2:01 PM, Munagala Ramanath <ra...@datatorrent.com>
>> wrote:
>> 
>>> Sorry, meant to say 4-6GB, not MB.
>>> 
>>> Ram
>>> 
>>> On Mon, Dec 14, 2015 at 2:00 PM, Munagala Ramanath <ra...@datatorrent.com>
>>> wrote:
>>> 
>>>> I'm OK with increasing it to 1280MB.
>>>> 
>>>> I agree with Sandeep about new people trying out Apex,
>>>> often on limited environments like VMs limited to 4-6MB. With a default
>>> of
>>>> 0.5GB per Buffer Server
>>>> and 1GB per operator, things start to fail when the user has more than
>> a
>>>> handful of operators.
>>>> 
>>>> I'm guessing it's not just the number of the operators that puts memory
>>>> pressure on master memory
>>>> but the size of the jars that get pulled in by all the operator classes
>>>> and their dependencies. If this is
>>>> true, one could cause failure even with a single, sufficiently complex
>>>> operator.
>>>> 
>>>> Ram
>>>> 
>>>> On Mon, Dec 14, 2015 at 1:41 PM, Timothy Farkas <ti...@datatorrent.com>
>>>> wrote:
>>>> 
>>>>> Hi Sandeep,
>>>>> 
>>>>> I have seen the issue even with small apps like Twitter Demo. It
>> happens
>>>>> less frequently in very small apps but the issue is still there. It
>>>>> happens
>>>>> frequently in medium size apps (10 containers). Could we increase the
>>>>> MASTER_MEMORY_MB to 1280 instead of 2048? That way we can get greater
>>>>> stability with the defaults settings without allocating too much
>>>>> additional
>>>>> memory?
>>>>> 
>>>>> Thanks,
>>>>> Tim
>>>>> 
>>>>> On Sun, Dec 13, 2015 at 1:22 AM, Sandeep Deshmukh <
>>>>> sandeep@datatorrent.com>
>>>>> wrote:
>>>>> 
>>>>>> With large number of containers, the problem becomes severe and I
>> too
>>>>> have
>>>>>> faced it.
>>>>>> 
>>>>>> Not sure if default should be increased as that should work in most
>> of
>>>>> the
>>>>>> cases and even when new people are trying out Apex for the first
>> time.
>>>>>> They may not have enough memory.
>>>>>> 
>>>>>> Regards
>>>>>> Sandeep
>>>>>> On 13 Dec 2015 11:30, "Ganelin, Ilya" <Il...@capitalone.com>
>>>>> wrote:
>>>>>> 
>>>>>>> Tim - great point, we reduce the memory consumption for operators
>>> and
>>>>> do
>>>>>>> actually increase it for the app master, especially when we have
>>> more
>>>>>>> operators. I misread your original email.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Thank you,
>>>>>>> Ilya Ganelin
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> -----Original Message-----
>>>>>>> From: Timothy Farkas [tim@datatorrent.com<mailto:
>>> tim@datatorrent.com
>>>>>> ]
>>>>>>> Sent: Saturday, December 12, 2015 11:30 PM Eastern Standard Time
>>>>>>> To: dev@apex.incubator.apache.org
>>>>>>> Subject: Re: Default App Master Container Memory Size
>>>>>>> 
>>>>>>> 
>>>>>>> Hi Ilya,
>>>>>>> 
>>>>>>> Dou you reduce the memory to 256 mb for operators only or for the
>>> App
>>>>>>> Master as well?
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Tim
>>>>>>> 
>>>>>>> On Sat, Dec 12, 2015 at 8:06 PM, Ganelin, Ilya <
>>>>>>> Ilya.Ganelin@capitalone.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Tim - in our applications we've actually tried to go with
>> smaller
>>>>>>>> containers, as small as 256 MB, simply because many operators
>>> don't
>>>>>>> demand
>>>>>>>> any memory and this allows us to run more partitions.
>>>>>>>> 
>>>>>>>> We haven't noticed performance degradation.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Thank you,
>>>>>>>> Ilya Ganelin
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> -----Original Message-----
>>>>>>>> From: Timothy Farkas [tim@datatorrent.com<mailto:
>>>>> tim@datatorrent.com>]
>>>>>>>> Sent: Saturday, December 12, 2015 11:02 PM Eastern Standard Time
>>>>>>>> To: dev@apex.incubator.apache.org
>>>>>>>> Subject: Default App Master Container Memory Size
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Hi All,
>>>>>>>> 
>>>>>>>> The default value for MASTER_MEMORY_MB is 1024. I've observed in
>>>>>> multiple
>>>>>>>> demo apps and in the production apps of some of our users that
>>> 1024
>>>>> MB
>>>>>> is
>>>>>>>> not enough memory, and the app master dies. This was observed on
>>>>> small
>>>>>>>> applications with 5 - 20 containers. I think the default app
>>> master
>>>>>>> memory
>>>>>>>> size should be 2048.
>>>>>>>> 
>>>>>>>> Any feedback?
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Tim
>>>>>>>> ________________________________________________________
>>>>>>>> 
>>>>>>>> The information contained in this e-mail is confidential and/or
>>>>>>>> proprietary to Capital One and/or its affiliates and may only be
>>>>> used
>>>>>>>> solely in performance of work or services for Capital One. The
>>>>>>> information
>>>>>>>> transmitted herewith is intended only for use by the individual
>> or
>>>>>> entity
>>>>>>>> to which it is addressed. If the reader of this message is not
>> the
>>>>>>> intended
>>>>>>>> recipient, you are hereby notified that any review,
>>> retransmission,
>>>>>>>> dissemination, distribution, copying or other use of, or taking
>> of
>>>>> any
>>>>>>>> action in reliance upon this information is strictly prohibited.
>>> If
>>>>> you
>>>>>>>> have received this communication in error, please contact the
>>> sender
>>>>>> and
>>>>>>>> delete the material from your computer.
>>>>>>>> 
>>>>>>> ________________________________________________________
>>>>>>> 
>>>>>>> The information contained in this e-mail is confidential and/or
>>>>>>> proprietary to Capital One and/or its affiliates and may only be
>>> used
>>>>>>> solely in performance of work or services for Capital One. The
>>>>>> information
>>>>>>> transmitted herewith is intended only for use by the individual or
>>>>> entity
>>>>>>> to which it is addressed. If the reader of this message is not the
>>>>>> intended
>>>>>>> recipient, you are hereby notified that any review,
>> retransmission,
>>>>>>> dissemination, distribution, copying or other use of, or taking of
>>> any
>>>>>>> action in reliance upon this information is strictly prohibited.
>> If
>>>>> you
>>>>>>> have received this communication in error, please contact the
>> sender
>>>>> and
>>>>>>> delete the material from your computer.
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>> 

Re: Default App Master Container Memory Size

[Sandeep Deshmukh <sa...@datatorrent.com>]

I am ok with 1280MB.
On 24 Dec 2015 06:34, "Timothy Farkas" <ti...@datatorrent.com> wrote:

> If there are no objections I will open a pull request shortly to bump up
> the default App Master Memory to 1280 MB. I recently got burned by this
> default AGAIN when the App Master randomly died in the middle of demo. The
> demo application was small and only had 6 physical operators.
>
> Thanks,
> Tim
>
> On Mon, Dec 14, 2015 at 2:01 PM, Munagala Ramanath <ra...@datatorrent.com>
> wrote:
>
> > Sorry, meant to say 4-6GB, not MB.
> >
> > Ram
> >
> > On Mon, Dec 14, 2015 at 2:00 PM, Munagala Ramanath <ra...@datatorrent.com>
> > wrote:
> >
> > > I'm OK with increasing it to 1280MB.
> > >
> > > I agree with Sandeep about new people trying out Apex,
> > > often on limited environments like VMs limited to 4-6MB. With a default
> > of
> > > 0.5GB per Buffer Server
> > > and 1GB per operator, things start to fail when the user has more than
> a
> > > handful of operators.
> > >
> > > I'm guessing it's not just the number of the operators that puts memory
> > > pressure on master memory
> > > but the size of the jars that get pulled in by all the operator classes
> > > and their dependencies. If this is
> > > true, one could cause failure even with a single, sufficiently complex
> > > operator.
> > >
> > > Ram
> > >
> > > On Mon, Dec 14, 2015 at 1:41 PM, Timothy Farkas <ti...@datatorrent.com>
> > > wrote:
> > >
> > >> Hi Sandeep,
> > >>
> > >> I have seen the issue even with small apps like Twitter Demo. It
> happens
> > >> less frequently in very small apps but the issue is still there. It
> > >> happens
> > >> frequently in medium size apps (10 containers). Could we increase the
> > >> MASTER_MEMORY_MB to 1280 instead of 2048? That way we can get greater
> > >> stability with the defaults settings without allocating too much
> > >> additional
> > >> memory?
> > >>
> > >> Thanks,
> > >> Tim
> > >>
> > >> On Sun, Dec 13, 2015 at 1:22 AM, Sandeep Deshmukh <
> > >> sandeep@datatorrent.com>
> > >> wrote:
> > >>
> > >> > With large number of containers, the problem becomes severe and I
> too
> > >> have
> > >> > faced it.
> > >> >
> > >> > Not sure if default should be increased as that should work in most
> of
> > >> the
> > >> > cases and even when new people are trying out Apex for the first
> time.
> > >> > They may not have enough memory.
> > >> >
> > >> > Regards
> > >> > Sandeep
> > >> > On 13 Dec 2015 11:30, "Ganelin, Ilya" <Il...@capitalone.com>
> > >> wrote:
> > >> >
> > >> > > Tim - great point, we reduce the memory consumption for operators
> > and
> > >> do
> > >> > > actually increase it for the app master, especially when we have
> > more
> > >> > > operators. I misread your original email.
> > >> > >
> > >> > >
> > >> > >
> > >> > > Thank you,
> > >> > > Ilya Ganelin
> > >> > >
> > >> > >
> > >> > >
> > >> > > -----Original Message-----
> > >> > > From: Timothy Farkas [tim@datatorrent.com<mailto:
> > tim@datatorrent.com
> > >> >]
> > >> > > Sent: Saturday, December 12, 2015 11:30 PM Eastern Standard Time
> > >> > > To: dev@apex.incubator.apache.org
> > >> > > Subject: Re: Default App Master Container Memory Size
> > >> > >
> > >> > >
> > >> > > Hi Ilya,
> > >> > >
> > >> > > Dou you reduce the memory to 256 mb for operators only or for the
> > App
> > >> > > Master as well?
> > >> > >
> > >> > > Thanks,
> > >> > > Tim
> > >> > >
> > >> > > On Sat, Dec 12, 2015 at 8:06 PM, Ganelin, Ilya <
> > >> > > Ilya.Ganelin@capitalone.com>
> > >> > > wrote:
> > >> > >
> > >> > > > Tim - in our applications we've actually tried to go with
> smaller
> > >> > > > containers, as small as 256 MB, simply because many operators
> > don't
> > >> > > demand
> > >> > > > any memory and this allows us to run more partitions.
> > >> > > >
> > >> > > > We haven't noticed performance degradation.
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > Thank you,
> > >> > > > Ilya Ganelin
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > -----Original Message-----
> > >> > > > From: Timothy Farkas [tim@datatorrent.com<mailto:
> > >> tim@datatorrent.com>]
> > >> > > > Sent: Saturday, December 12, 2015 11:02 PM Eastern Standard Time
> > >> > > > To: dev@apex.incubator.apache.org
> > >> > > > Subject: Default App Master Container Memory Size
> > >> > > >
> > >> > > >
> > >> > > > Hi All,
> > >> > > >
> > >> > > > The default value for MASTER_MEMORY_MB is 1024. I've observed in
> > >> > multiple
> > >> > > > demo apps and in the production apps of some of our users that
> > 1024
> > >> MB
> > >> > is
> > >> > > > not enough memory, and the app master dies. This was observed on
> > >> small
> > >> > > > applications with 5 - 20 containers. I think the default app
> > master
> > >> > > memory
> > >> > > > size should be 2048.
> > >> > > >
> > >> > > > Any feedback?
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Tim
> > >> > > > ________________________________________________________
> > >> > > >
> > >> > > > The information contained in this e-mail is confidential and/or
> > >> > > > proprietary to Capital One and/or its affiliates and may only be
> > >> used
> > >> > > > solely in performance of work or services for Capital One. The
> > >> > > information
> > >> > > > transmitted herewith is intended only for use by the individual
> or
> > >> > entity
> > >> > > > to which it is addressed. If the reader of this message is not
> the
> > >> > > intended
> > >> > > > recipient, you are hereby notified that any review,
> > retransmission,
> > >> > > > dissemination, distribution, copying or other use of, or taking
> of
> > >> any
> > >> > > > action in reliance upon this information is strictly prohibited.
> > If
> > >> you
> > >> > > > have received this communication in error, please contact the
> > sender
> > >> > and
> > >> > > > delete the material from your computer.
> > >> > > >
> > >> > > ________________________________________________________
> > >> > >
> > >> > > The information contained in this e-mail is confidential and/or
> > >> > > proprietary to Capital One and/or its affiliates and may only be
> > used
> > >> > > solely in performance of work or services for Capital One. The
> > >> > information
> > >> > > transmitted herewith is intended only for use by the individual or
> > >> entity
> > >> > > to which it is addressed. If the reader of this message is not the
> > >> > intended
> > >> > > recipient, you are hereby notified that any review,
> retransmission,
> > >> > > dissemination, distribution, copying or other use of, or taking of
> > any
> > >> > > action in reliance upon this information is strictly prohibited.
> If
> > >> you
> > >> > > have received this communication in error, please contact the
> sender
> > >> and
> > >> > > delete the material from your computer.
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Re: Default App Master Container Memory Size

[Timothy Farkas <ti...@datatorrent.com>]

If there are no objections I will open a pull request shortly to bump up
the default App Master Memory to 1280 MB. I recently got burned by this
default AGAIN when the App Master randomly died in the middle of demo. The
demo application was small and only had 6 physical operators.

Thanks,
Tim

On Mon, Dec 14, 2015 at 2:01 PM, Munagala Ramanath <ra...@datatorrent.com>
wrote:

> Sorry, meant to say 4-6GB, not MB.
>
> Ram
>
> On Mon, Dec 14, 2015 at 2:00 PM, Munagala Ramanath <ra...@datatorrent.com>
> wrote:
>
> > I'm OK with increasing it to 1280MB.
> >
> > I agree with Sandeep about new people trying out Apex,
> > often on limited environments like VMs limited to 4-6MB. With a default
> of
> > 0.5GB per Buffer Server
> > and 1GB per operator, things start to fail when the user has more than a
> > handful of operators.
> >
> > I'm guessing it's not just the number of the operators that puts memory
> > pressure on master memory
> > but the size of the jars that get pulled in by all the operator classes
> > and their dependencies. If this is
> > true, one could cause failure even with a single, sufficiently complex
> > operator.
> >
> > Ram
> >
> > On Mon, Dec 14, 2015 at 1:41 PM, Timothy Farkas <ti...@datatorrent.com>
> > wrote:
> >
> >> Hi Sandeep,
> >>
> >> I have seen the issue even with small apps like Twitter Demo. It happens
> >> less frequently in very small apps but the issue is still there. It
> >> happens
> >> frequently in medium size apps (10 containers). Could we increase the
> >> MASTER_MEMORY_MB to 1280 instead of 2048? That way we can get greater
> >> stability with the defaults settings without allocating too much
> >> additional
> >> memory?
> >>
> >> Thanks,
> >> Tim
> >>
> >> On Sun, Dec 13, 2015 at 1:22 AM, Sandeep Deshmukh <
> >> sandeep@datatorrent.com>
> >> wrote:
> >>
> >> > With large number of containers, the problem becomes severe and I too
> >> have
> >> > faced it.
> >> >
> >> > Not sure if default should be increased as that should work in most of
> >> the
> >> > cases and even when new people are trying out Apex for the first time.
> >> > They may not have enough memory.
> >> >
> >> > Regards
> >> > Sandeep
> >> > On 13 Dec 2015 11:30, "Ganelin, Ilya" <Il...@capitalone.com>
> >> wrote:
> >> >
> >> > > Tim - great point, we reduce the memory consumption for operators
> and
> >> do
> >> > > actually increase it for the app master, especially when we have
> more
> >> > > operators. I misread your original email.
> >> > >
> >> > >
> >> > >
> >> > > Thank you,
> >> > > Ilya Ganelin
> >> > >
> >> > >
> >> > >
> >> > > -----Original Message-----
> >> > > From: Timothy Farkas [tim@datatorrent.com<mailto:
> tim@datatorrent.com
> >> >]
> >> > > Sent: Saturday, December 12, 2015 11:30 PM Eastern Standard Time
> >> > > To: dev@apex.incubator.apache.org
> >> > > Subject: Re: Default App Master Container Memory Size
> >> > >
> >> > >
> >> > > Hi Ilya,
> >> > >
> >> > > Dou you reduce the memory to 256 mb for operators only or for the
> App
> >> > > Master as well?
> >> > >
> >> > > Thanks,
> >> > > Tim
> >> > >
> >> > > On Sat, Dec 12, 2015 at 8:06 PM, Ganelin, Ilya <
> >> > > Ilya.Ganelin@capitalone.com>
> >> > > wrote:
> >> > >
> >> > > > Tim - in our applications we've actually tried to go with smaller
> >> > > > containers, as small as 256 MB, simply because many operators
> don't
> >> > > demand
> >> > > > any memory and this allows us to run more partitions.
> >> > > >
> >> > > > We haven't noticed performance degradation.
> >> > > >
> >> > > >
> >> > > >
> >> > > > Thank you,
> >> > > > Ilya Ganelin
> >> > > >
> >> > > >
> >> > > >
> >> > > > -----Original Message-----
> >> > > > From: Timothy Farkas [tim@datatorrent.com<mailto:
> >> tim@datatorrent.com>]
> >> > > > Sent: Saturday, December 12, 2015 11:02 PM Eastern Standard Time
> >> > > > To: dev@apex.incubator.apache.org
> >> > > > Subject: Default App Master Container Memory Size
> >> > > >
> >> > > >
> >> > > > Hi All,
> >> > > >
> >> > > > The default value for MASTER_MEMORY_MB is 1024. I've observed in
> >> > multiple
> >> > > > demo apps and in the production apps of some of our users that
> 1024
> >> MB
> >> > is
> >> > > > not enough memory, and the app master dies. This was observed on
> >> small
> >> > > > applications with 5 - 20 containers. I think the default app
> master
> >> > > memory
> >> > > > size should be 2048.
> >> > > >
> >> > > > Any feedback?
> >> > > >
> >> > > > Thanks,
> >> > > > Tim
> >> > > > ________________________________________________________
> >> > > >
> >> > > > The information contained in this e-mail is confidential and/or
> >> > > > proprietary to Capital One and/or its affiliates and may only be
> >> used
> >> > > > solely in performance of work or services for Capital One. The
> >> > > information
> >> > > > transmitted herewith is intended only for use by the individual or
> >> > entity
> >> > > > to which it is addressed. If the reader of this message is not the
> >> > > intended
> >> > > > recipient, you are hereby notified that any review,
> retransmission,
> >> > > > dissemination, distribution, copying or other use of, or taking of
> >> any
> >> > > > action in reliance upon this information is strictly prohibited.
> If
> >> you
> >> > > > have received this communication in error, please contact the
> sender
> >> > and
> >> > > > delete the material from your computer.
> >> > > >
> >> > > ________________________________________________________
> >> > >
> >> > > The information contained in this e-mail is confidential and/or
> >> > > proprietary to Capital One and/or its affiliates and may only be
> used
> >> > > solely in performance of work or services for Capital One. The
> >> > information
> >> > > transmitted herewith is intended only for use by the individual or
> >> entity
> >> > > to which it is addressed. If the reader of this message is not the
> >> > intended
> >> > > recipient, you are hereby notified that any review, retransmission,
> >> > > dissemination, distribution, copying or other use of, or taking of
> any
> >> > > action in reliance upon this information is strictly prohibited. If
> >> you
> >> > > have received this communication in error, please contact the sender
> >> and
> >> > > delete the material from your computer.
> >> > >
> >> >
> >>
> >
> >
>

Re: Default App Master Container Memory Size

[Munagala Ramanath <ra...@datatorrent.com>]

Sorry, meant to say 4-6GB, not MB.

Ram

On Mon, Dec 14, 2015 at 2:00 PM, Munagala Ramanath <ra...@datatorrent.com>
wrote:

> I'm OK with increasing it to 1280MB.
>
> I agree with Sandeep about new people trying out Apex,
> often on limited environments like VMs limited to 4-6MB. With a default of
> 0.5GB per Buffer Server
> and 1GB per operator, things start to fail when the user has more than a
> handful of operators.
>
> I'm guessing it's not just the number of the operators that puts memory
> pressure on master memory
> but the size of the jars that get pulled in by all the operator classes
> and their dependencies. If this is
> true, one could cause failure even with a single, sufficiently complex
> operator.
>
> Ram
>
> On Mon, Dec 14, 2015 at 1:41 PM, Timothy Farkas <ti...@datatorrent.com>
> wrote:
>
>> Hi Sandeep,
>>
>> I have seen the issue even with small apps like Twitter Demo. It happens
>> less frequently in very small apps but the issue is still there. It
>> happens
>> frequently in medium size apps (10 containers). Could we increase the
>> MASTER_MEMORY_MB to 1280 instead of 2048? That way we can get greater
>> stability with the defaults settings without allocating too much
>> additional
>> memory?
>>
>> Thanks,
>> Tim
>>
>> On Sun, Dec 13, 2015 at 1:22 AM, Sandeep Deshmukh <
>> sandeep@datatorrent.com>
>> wrote:
>>
>> > With large number of containers, the problem becomes severe and I too
>> have
>> > faced it.
>> >
>> > Not sure if default should be increased as that should work in most of
>> the
>> > cases and even when new people are trying out Apex for the first time.
>> > They may not have enough memory.
>> >
>> > Regards
>> > Sandeep
>> > On 13 Dec 2015 11:30, "Ganelin, Ilya" <Il...@capitalone.com>
>> wrote:
>> >
>> > > Tim - great point, we reduce the memory consumption for operators and
>> do
>> > > actually increase it for the app master, especially when we have more
>> > > operators. I misread your original email.
>> > >
>> > >
>> > >
>> > > Thank you,
>> > > Ilya Ganelin
>> > >
>> > >
>> > >
>> > > -----Original Message-----
>> > > From: Timothy Farkas [tim@datatorrent.com<mailto:tim@datatorrent.com
>> >]
>> > > Sent: Saturday, December 12, 2015 11:30 PM Eastern Standard Time
>> > > To: dev@apex.incubator.apache.org
>> > > Subject: Re: Default App Master Container Memory Size
>> > >
>> > >
>> > > Hi Ilya,
>> > >
>> > > Dou you reduce the memory to 256 mb for operators only or for the App
>> > > Master as well?
>> > >
>> > > Thanks,
>> > > Tim
>> > >
>> > > On Sat, Dec 12, 2015 at 8:06 PM, Ganelin, Ilya <
>> > > Ilya.Ganelin@capitalone.com>
>> > > wrote:
>> > >
>> > > > Tim - in our applications we've actually tried to go with smaller
>> > > > containers, as small as 256 MB, simply because many operators don't
>> > > demand
>> > > > any memory and this allows us to run more partitions.
>> > > >
>> > > > We haven't noticed performance degradation.
>> > > >
>> > > >
>> > > >
>> > > > Thank you,
>> > > > Ilya Ganelin
>> > > >
>> > > >
>> > > >
>> > > > -----Original Message-----
>> > > > From: Timothy Farkas [tim@datatorrent.com<mailto:
>> tim@datatorrent.com>]
>> > > > Sent: Saturday, December 12, 2015 11:02 PM Eastern Standard Time
>> > > > To: dev@apex.incubator.apache.org
>> > > > Subject: Default App Master Container Memory Size
>> > > >
>> > > >
>> > > > Hi All,
>> > > >
>> > > > The default value for MASTER_MEMORY_MB is 1024. I've observed in
>> > multiple
>> > > > demo apps and in the production apps of some of our users that 1024
>> MB
>> > is
>> > > > not enough memory, and the app master dies. This was observed on
>> small
>> > > > applications with 5 - 20 containers. I think the default app master
>> > > memory
>> > > > size should be 2048.
>> > > >
>> > > > Any feedback?
>> > > >
>> > > > Thanks,
>> > > > Tim
>> > > > ________________________________________________________
>> > > >
>> > > > The information contained in this e-mail is confidential and/or
>> > > > proprietary to Capital One and/or its affiliates and may only be
>> used
>> > > > solely in performance of work or services for Capital One. The
>> > > information
>> > > > transmitted herewith is intended only for use by the individual or
>> > entity
>> > > > to which it is addressed. If the reader of this message is not the
>> > > intended
>> > > > recipient, you are hereby notified that any review, retransmission,
>> > > > dissemination, distribution, copying or other use of, or taking of
>> any
>> > > > action in reliance upon this information is strictly prohibited. If
>> you
>> > > > have received this communication in error, please contact the sender
>> > and
>> > > > delete the material from your computer.
>> > > >
>> > > ________________________________________________________
>> > >
>> > > The information contained in this e-mail is confidential and/or
>> > > proprietary to Capital One and/or its affiliates and may only be used
>> > > solely in performance of work or services for Capital One. The
>> > information
>> > > transmitted herewith is intended only for use by the individual or
>> entity
>> > > to which it is addressed. If the reader of this message is not the
>> > intended
>> > > recipient, you are hereby notified that any review, retransmission,
>> > > dissemination, distribution, copying or other use of, or taking of any
>> > > action in reliance upon this information is strictly prohibited. If
>> you
>> > > have received this communication in error, please contact the sender
>> and
>> > > delete the material from your computer.
>> > >
>> >
>>
>
>

Re: Default App Master Container Memory Size

[Munagala Ramanath <ra...@datatorrent.com>]

I'm OK with increasing it to 1280MB.

I agree with Sandeep about new people trying out Apex,
often on limited environments like VMs limited to 4-6MB. With a default of
0.5GB per Buffer Server
and 1GB per operator, things start to fail when the user has more than a
handful of operators.

I'm guessing it's not just the number of the operators that puts memory
pressure on master memory
but the size of the jars that get pulled in by all the operator classes and
their dependencies. If this is
true, one could cause failure even with a single, sufficiently complex
operator.

Ram

On Mon, Dec 14, 2015 at 1:41 PM, Timothy Farkas <ti...@datatorrent.com> wrote:

> Hi Sandeep,
>
> I have seen the issue even with small apps like Twitter Demo. It happens
> less frequently in very small apps but the issue is still there. It happens
> frequently in medium size apps (10 containers). Could we increase the
> MASTER_MEMORY_MB to 1280 instead of 2048? That way we can get greater
> stability with the defaults settings without allocating too much additional
> memory?
>
> Thanks,
> Tim
>
> On Sun, Dec 13, 2015 at 1:22 AM, Sandeep Deshmukh <sandeep@datatorrent.com
> >
> wrote:
>
> > With large number of containers, the problem becomes severe and I too
> have
> > faced it.
> >
> > Not sure if default should be increased as that should work in most of
> the
> > cases and even when new people are trying out Apex for the first time.
> > They may not have enough memory.
> >
> > Regards
> > Sandeep
> > On 13 Dec 2015 11:30, "Ganelin, Ilya" <Il...@capitalone.com>
> wrote:
> >
> > > Tim - great point, we reduce the memory consumption for operators and
> do
> > > actually increase it for the app master, especially when we have more
> > > operators. I misread your original email.
> > >
> > >
> > >
> > > Thank you,
> > > Ilya Ganelin
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Timothy Farkas [tim@datatorrent.com<ma...@datatorrent.com>]
> > > Sent: Saturday, December 12, 2015 11:30 PM Eastern Standard Time
> > > To: dev@apex.incubator.apache.org
> > > Subject: Re: Default App Master Container Memory Size
> > >
> > >
> > > Hi Ilya,
> > >
> > > Dou you reduce the memory to 256 mb for operators only or for the App
> > > Master as well?
> > >
> > > Thanks,
> > > Tim
> > >
> > > On Sat, Dec 12, 2015 at 8:06 PM, Ganelin, Ilya <
> > > Ilya.Ganelin@capitalone.com>
> > > wrote:
> > >
> > > > Tim - in our applications we've actually tried to go with smaller
> > > > containers, as small as 256 MB, simply because many operators don't
> > > demand
> > > > any memory and this allows us to run more partitions.
> > > >
> > > > We haven't noticed performance degradation.
> > > >
> > > >
> > > >
> > > > Thank you,
> > > > Ilya Ganelin
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Timothy Farkas [tim@datatorrent.com<mailto:tim@datatorrent.com
> >]
> > > > Sent: Saturday, December 12, 2015 11:02 PM Eastern Standard Time
> > > > To: dev@apex.incubator.apache.org
> > > > Subject: Default App Master Container Memory Size
> > > >
> > > >
> > > > Hi All,
> > > >
> > > > The default value for MASTER_MEMORY_MB is 1024. I've observed in
> > multiple
> > > > demo apps and in the production apps of some of our users that 1024
> MB
> > is
> > > > not enough memory, and the app master dies. This was observed on
> small
> > > > applications with 5 - 20 containers. I think the default app master
> > > memory
> > > > size should be 2048.
> > > >
> > > > Any feedback?
> > > >
> > > > Thanks,
> > > > Tim
> > > > ________________________________________________________
> > > >
> > > > The information contained in this e-mail is confidential and/or
> > > > proprietary to Capital One and/or its affiliates and may only be used
> > > > solely in performance of work or services for Capital One. The
> > > information
> > > > transmitted herewith is intended only for use by the individual or
> > entity
> > > > to which it is addressed. If the reader of this message is not the
> > > intended
> > > > recipient, you are hereby notified that any review, retransmission,
> > > > dissemination, distribution, copying or other use of, or taking of
> any
> > > > action in reliance upon this information is strictly prohibited. If
> you
> > > > have received this communication in error, please contact the sender
> > and
> > > > delete the material from your computer.
> > > >
> > > ________________________________________________________
> > >
> > > The information contained in this e-mail is confidential and/or
> > > proprietary to Capital One and/or its affiliates and may only be used
> > > solely in performance of work or services for Capital One. The
> > information
> > > transmitted herewith is intended only for use by the individual or
> entity
> > > to which it is addressed. If the reader of this message is not the
> > intended
> > > recipient, you are hereby notified that any review, retransmission,
> > > dissemination, distribution, copying or other use of, or taking of any
> > > action in reliance upon this information is strictly prohibited. If you
> > > have received this communication in error, please contact the sender
> and
> > > delete the material from your computer.
> > >
> >
>

Re: Default App Master Container Memory Size

[Timothy Farkas <ti...@datatorrent.com>]

Hi Sandeep,

I have seen the issue even with small apps like Twitter Demo. It happens
less frequently in very small apps but the issue is still there. It happens
frequently in medium size apps (10 containers). Could we increase the
MASTER_MEMORY_MB to 1280 instead of 2048? That way we can get greater
stability with the defaults settings without allocating too much additional
memory?

Thanks,
Tim

On Sun, Dec 13, 2015 at 1:22 AM, Sandeep Deshmukh <sa...@datatorrent.com>
wrote:

> With large number of containers, the problem becomes severe and I too have
> faced it.
>
> Not sure if default should be increased as that should work in most of the
> cases and even when new people are trying out Apex for the first time.
> They may not have enough memory.
>
> Regards
> Sandeep
> On 13 Dec 2015 11:30, "Ganelin, Ilya" <Il...@capitalone.com> wrote:
>
> > Tim - great point, we reduce the memory consumption for operators and do
> > actually increase it for the app master, especially when we have more
> > operators. I misread your original email.
> >
> >
> >
> > Thank you,
> > Ilya Ganelin
> >
> >
> >
> > -----Original Message-----
> > From: Timothy Farkas [tim@datatorrent.com<ma...@datatorrent.com>]
> > Sent: Saturday, December 12, 2015 11:30 PM Eastern Standard Time
> > To: dev@apex.incubator.apache.org
> > Subject: Re: Default App Master Container Memory Size
> >
> >
> > Hi Ilya,
> >
> > Dou you reduce the memory to 256 mb for operators only or for the App
> > Master as well?
> >
> > Thanks,
> > Tim
> >
> > On Sat, Dec 12, 2015 at 8:06 PM, Ganelin, Ilya <
> > Ilya.Ganelin@capitalone.com>
> > wrote:
> >
> > > Tim - in our applications we've actually tried to go with smaller
> > > containers, as small as 256 MB, simply because many operators don't
> > demand
> > > any memory and this allows us to run more partitions.
> > >
> > > We haven't noticed performance degradation.
> > >
> > >
> > >
> > > Thank you,
> > > Ilya Ganelin
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Timothy Farkas [tim@datatorrent.com<ma...@datatorrent.com>]
> > > Sent: Saturday, December 12, 2015 11:02 PM Eastern Standard Time
> > > To: dev@apex.incubator.apache.org
> > > Subject: Default App Master Container Memory Size
> > >
> > >
> > > Hi All,
> > >
> > > The default value for MASTER_MEMORY_MB is 1024. I've observed in
> multiple
> > > demo apps and in the production apps of some of our users that 1024 MB
> is
> > > not enough memory, and the app master dies. This was observed on small
> > > applications with 5 - 20 containers. I think the default app master
> > memory
> > > size should be 2048.
> > >
> > > Any feedback?
> > >
> > > Thanks,
> > > Tim
> > > ________________________________________________________
> > >
> > > The information contained in this e-mail is confidential and/or
> > > proprietary to Capital One and/or its affiliates and may only be used
> > > solely in performance of work or services for Capital One. The
> > information
> > > transmitted herewith is intended only for use by the individual or
> entity
> > > to which it is addressed. If the reader of this message is not the
> > intended
> > > recipient, you are hereby notified that any review, retransmission,
> > > dissemination, distribution, copying or other use of, or taking of any
> > > action in reliance upon this information is strictly prohibited. If you
> > > have received this communication in error, please contact the sender
> and
> > > delete the material from your computer.
> > >
> > ________________________________________________________
> >
> > The information contained in this e-mail is confidential and/or
> > proprietary to Capital One and/or its affiliates and may only be used
> > solely in performance of work or services for Capital One. The
> information
> > transmitted herewith is intended only for use by the individual or entity
> > to which it is addressed. If the reader of this message is not the
> intended
> > recipient, you are hereby notified that any review, retransmission,
> > dissemination, distribution, copying or other use of, or taking of any
> > action in reliance upon this information is strictly prohibited. If you
> > have received this communication in error, please contact the sender and
> > delete the material from your computer.
> >
>

RE: Default App Master Container Memory Size

[Sandeep Deshmukh <sa...@datatorrent.com>]

With large number of containers, the problem becomes severe and I too have
faced it.

Not sure if default should be increased as that should work in most of the
cases and even when new people are trying out Apex for the first time.
They may not have enough memory.

Regards
Sandeep
On 13 Dec 2015 11:30, "Ganelin, Ilya" <Il...@capitalone.com> wrote:

> Tim - great point, we reduce the memory consumption for operators and do
> actually increase it for the app master, especially when we have more
> operators. I misread your original email.
>
>
>
> Thank you,
> Ilya Ganelin
>
>
>
> -----Original Message-----
> From: Timothy Farkas [tim@datatorrent.com<ma...@datatorrent.com>]
> Sent: Saturday, December 12, 2015 11:30 PM Eastern Standard Time
> To: dev@apex.incubator.apache.org
> Subject: Re: Default App Master Container Memory Size
>
>
> Hi Ilya,
>
> Dou you reduce the memory to 256 mb for operators only or for the App
> Master as well?
>
> Thanks,
> Tim
>
> On Sat, Dec 12, 2015 at 8:06 PM, Ganelin, Ilya <
> Ilya.Ganelin@capitalone.com>
> wrote:
>
> > Tim - in our applications we've actually tried to go with smaller
> > containers, as small as 256 MB, simply because many operators don't
> demand
> > any memory and this allows us to run more partitions.
> >
> > We haven't noticed performance degradation.
> >
> >
> >
> > Thank you,
> > Ilya Ganelin
> >
> >
> >
> > -----Original Message-----
> > From: Timothy Farkas [tim@datatorrent.com<ma...@datatorrent.com>]
> > Sent: Saturday, December 12, 2015 11:02 PM Eastern Standard Time
> > To: dev@apex.incubator.apache.org
> > Subject: Default App Master Container Memory Size
> >
> >
> > Hi All,
> >
> > The default value for MASTER_MEMORY_MB is 1024. I've observed in multiple
> > demo apps and in the production apps of some of our users that 1024 MB is
> > not enough memory, and the app master dies. This was observed on small
> > applications with 5 - 20 containers. I think the default app master
> memory
> > size should be 2048.
> >
> > Any feedback?
> >
> > Thanks,
> > Tim
> > ________________________________________________________
> >
> > The information contained in this e-mail is confidential and/or
> > proprietary to Capital One and/or its affiliates and may only be used
> > solely in performance of work or services for Capital One. The
> information
> > transmitted herewith is intended only for use by the individual or entity
> > to which it is addressed. If the reader of this message is not the
> intended
> > recipient, you are hereby notified that any review, retransmission,
> > dissemination, distribution, copying or other use of, or taking of any
> > action in reliance upon this information is strictly prohibited. If you
> > have received this communication in error, please contact the sender and
> > delete the material from your computer.
> >
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>

RE: Default App Master Container Memory Size

["Ganelin, Ilya" <Il...@capitalone.com>]

Tim - great point, we reduce the memory consumption for operators and do actually increase it for the app master, especially when we have more operators. I misread your original email.



Thank you,
Ilya Ganelin



-----Original Message-----
From: Timothy Farkas [tim@datatorrent.com<ma...@datatorrent.com>]
Sent: Saturday, December 12, 2015 11:30 PM Eastern Standard Time
To: dev@apex.incubator.apache.org
Subject: Re: Default App Master Container Memory Size


Hi Ilya,

Dou you reduce the memory to 256 mb for operators only or for the App
Master as well?

Thanks,
Tim

On Sat, Dec 12, 2015 at 8:06 PM, Ganelin, Ilya <Il...@capitalone.com>
wrote:

> Tim - in our applications we've actually tried to go with smaller
> containers, as small as 256 MB, simply because many operators don't demand
> any memory and this allows us to run more partitions.
>
> We haven't noticed performance degradation.
>
>
>
> Thank you,
> Ilya Ganelin
>
>
>
> -----Original Message-----
> From: Timothy Farkas [tim@datatorrent.com<ma...@datatorrent.com>]
> Sent: Saturday, December 12, 2015 11:02 PM Eastern Standard Time
> To: dev@apex.incubator.apache.org
> Subject: Default App Master Container Memory Size
>
>
> Hi All,
>
> The default value for MASTER_MEMORY_MB is 1024. I've observed in multiple
> demo apps and in the production apps of some of our users that 1024 MB is
> not enough memory, and the app master dies. This was observed on small
> applications with 5 - 20 containers. I think the default app master memory
> size should be 2048.
>
> Any feedback?
>
> Thanks,
> Tim
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
________________________________________________________

The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

Re: Default App Master Container Memory Size

[Timothy Farkas <ti...@datatorrent.com>]

Hi Ilya,

Dou you reduce the memory to 256 mb for operators only or for the App
Master as well?

Thanks,
Tim

On Sat, Dec 12, 2015 at 8:06 PM, Ganelin, Ilya <Il...@capitalone.com>
wrote:

> Tim - in our applications we've actually tried to go with smaller
> containers, as small as 256 MB, simply because many operators don't demand
> any memory and this allows us to run more partitions.
>
> We haven't noticed performance degradation.
>
>
>
> Thank you,
> Ilya Ganelin
>
>
>
> -----Original Message-----
> From: Timothy Farkas [tim@datatorrent.com<ma...@datatorrent.com>]
> Sent: Saturday, December 12, 2015 11:02 PM Eastern Standard Time
> To: dev@apex.incubator.apache.org
> Subject: Default App Master Container Memory Size
>
>
> Hi All,
>
> The default value for MASTER_MEMORY_MB is 1024. I've observed in multiple
> demo apps and in the production apps of some of our users that 1024 MB is
> not enough memory, and the app master dies. This was observed on small
> applications with 5 - 20 containers. I think the default app master memory
> size should be 2048.
>
> Any feedback?
>
> Thanks,
> Tim
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>

RE: Default App Master Container Memory Size

["Ganelin, Ilya" <Il...@capitalone.com>]

Tim - in our applications we've actually tried to go with smaller containers, as small as 256 MB, simply because many operators don't demand any memory and this allows us to run more partitions.

We haven't noticed performance degradation.



Thank you,
Ilya Ganelin



-----Original Message-----
From: Timothy Farkas [tim@datatorrent.com<ma...@datatorrent.com>]
Sent: Saturday, December 12, 2015 11:02 PM Eastern Standard Time
To: dev@apex.incubator.apache.org
Subject: Default App Master Container Memory Size


Hi All,

The default value for MASTER_MEMORY_MB is 1024. I've observed in multiple
demo apps and in the production apps of some of our users that 1024 MB is
not enough memory, and the app master dies. This was observed on small
applications with 5 - 20 containers. I think the default app master memory
size should be 2048.

Any feedback?

Thanks,
Tim
________________________________________________________

The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.