Correlation between number of operators and Job manager memory requirements

[Shailesh Jain <sh...@stellapps.com>]

Hi,

I have flink job with almost 300 operators, and every time I'm trying to
submit the job, the cluster crashes with OutOfMemory exception.

I have 1 job manager and 1 task manager with 2 GB heap space allocated to
both.

In the configuration section of the documentation

Re: Correlation between number of operators and Job manager memory requirements

[Till Rohrmann <tr...@apache.org>]

Hi Shailesh,

my question would be where do you see the OOM happening? Does it happen on
the JM or the TM.

The memory requirements for each operator strongly depend on the operator
and it is hard to give a general formula for that. It mostly depends on the
user function. Flink itself should not need too much extra memory for the
framework specific code.

CEP, however, can easily add a couple of hundred megabytes to your memory
requirements. This depends strongly on the pattern you're matching and
which state backend you're using.

Concerning your question one big job vs. multiple jobs, I could see that
this helps if not all jobs are executed at the same time. Especially if you
only have a single TM with a limited number of slots, I think that you
effectively queue up jobs. That should reduce the required amount of
resources for each individual job.

Cheers,
Till

On Mon, Feb 19, 2018 at 11:35 AM, Shailesh Jain <shailesh.jain@stellapps.com
> wrote:

> Actually, there are too many hyperparameters to experiment with, that is
> why I'm trying to understand if there is any particular way in which a
> cluster could be benchmarked.
>
> Another strange behaviour I am observing is: Delaying the operator
> creation (by distributing the operators across jobs, and submitting
> multiple jobs to the same cluster instead of one) is helping in creating
> more operators. Any ideas on why that is happening?
>
> Shailesh
>
>
> On Sun, Feb 18, 2018 at 11:16 PM, Pawel Bartoszek <
> pawelbartoszek89@gmail.com> wrote:
>
>> Hi,
>>
>> You could definitely try to find formula for heap size, but isnt's it
>> easier just to try out different memory settings and see which works best
>> for you?
>>
>> Thanks,
>> Pawel
>>
>> 17 lut 2018 12:26 "Shailesh Jain" <sh...@stellapps.com>
>> napisał(a):
>>
>> Oops, hit send by mistake.
>>
>> In the configuration section, it is mentioned that for "many operators"
>> heap size should be increased.
>>
>> "JVM heap size (in megabytes) for the JobManager. You may have to
>> increase the heap size for the JobManager if you are running very large
>> applications (with many operators), or if you are keeping a long history of
>> them."
>>
>> Is there any recommendation on the heap space required when there are
>> around 200 CEP operators, and close 80 Filter operators?
>>
>> Any other leads on calculating the expected heap space allocation to
>> start the job would be really helpful.
>>
>> Thanks,
>> Shailesh
>>
>>
>>
>> On Sat, Feb 17, 2018 at 5:53 PM, Shailesh Jain <
>> shailesh.jain@stellapps.com> wrote:
>>
>>> Hi,
>>>
>>> I have flink job with almost 300 operators, and every time I'm trying to
>>> submit the job, the cluster crashes with OutOfMemory exception.
>>>
>>> I have 1 job manager and 1 task manager with 2 GB heap space allocated
>>> to both.
>>>
>>> In the configuration section of the documentation
>>>
>>>
>>>
>>>
>>
>>
>

Re: Correlation between number of operators and Job manager memory requirements

[Shailesh Jain <sh...@stellapps.com>]

Actually, there are too many hyperparameters to experiment with, that is
why I'm trying to understand if there is any particular way in which a
cluster could be benchmarked.

Another strange behaviour I am observing is: Delaying the operator creation
(by distributing the operators across jobs, and submitting multiple jobs to
the same cluster instead of one) is helping in creating more operators. Any
ideas on why that is happening?

Shailesh

On Sun, Feb 18, 2018 at 11:16 PM, Pawel Bartoszek <
pawelbartoszek89@gmail.com> wrote:

> Hi,
>
> You could definitely try to find formula for heap size, but isnt's it
> easier just to try out different memory settings and see which works best
> for you?
>
> Thanks,
> Pawel
>
> 17 lut 2018 12:26 "Shailesh Jain" <sh...@stellapps.com>
> napisał(a):
>
> Oops, hit send by mistake.
>
> In the configuration section, it is mentioned that for "many operators"
> heap size should be increased.
>
> "JVM heap size (in megabytes) for the JobManager. You may have to increase
> the heap size for the JobManager if you are running very large applications
> (with many operators), or if you are keeping a long history of them."
>
> Is there any recommendation on the heap space required when there are
> around 200 CEP operators, and close 80 Filter operators?
>
> Any other leads on calculating the expected heap space allocation to start
> the job would be really helpful.
>
> Thanks,
> Shailesh
>
>
>
> On Sat, Feb 17, 2018 at 5:53 PM, Shailesh Jain <
> shailesh.jain@stellapps.com> wrote:
>
>> Hi,
>>
>> I have flink job with almost 300 operators, and every time I'm trying to
>> submit the job, the cluster crashes with OutOfMemory exception.
>>
>> I have 1 job manager and 1 task manager with 2 GB heap space allocated to
>> both.
>>
>> In the configuration section of the documentation
>>
>>
>>
>>
>
>

Re: Correlation between number of operators and Job manager memory requirements

[Pawel Bartoszek <pa...@gmail.com>]

Hi,

You could definitely try to find formula for heap size, but isnt's it
easier just to try out different memory settings and see which works best
for you?

Thanks,
Pawel

17 lut 2018 12:26 "Shailesh Jain" <sh...@stellapps.com> napisał(a):

Oops, hit send by mistake.

In the configuration section, it is mentioned that for "many operators"
heap size should be increased.

"JVM heap size (in megabytes) for the JobManager. You may have to increase
the heap size for the JobManager if you are running very large applications
(with many operators), or if you are keeping a long history of them."

Is there any recommendation on the heap space required when there are
around 200 CEP operators, and close 80 Filter operators?

Any other leads on calculating the expected heap space allocation to start
the job would be really helpful.

Thanks,
Shailesh



On Sat, Feb 17, 2018 at 5:53 PM, Shailesh Jain <sh...@stellapps.com>
wrote:

> Hi,
>
> I have flink job with almost 300 operators, and every time I'm trying to
> submit the job, the cluster crashes with OutOfMemory exception.
>
> I have 1 job manager and 1 task manager with 2 GB heap space allocated to
> both.
>
> In the configuration section of the documentation
>
>
>
>

Re: Correlation between number of operators and Job manager memory requirements

[Shailesh Jain <sh...@stellapps.com>]

Oops, hit send by mistake.

In the configuration section, it is mentioned that for "many operators"
heap size should be increased.

"JVM heap size (in megabytes) for the JobManager. You may have to increase
the heap size for the JobManager if you are running very large applications
(with many operators), or if you are keeping a long history of them."

Is there any recommendation on the heap space required when there are
around 200 CEP operators, and close 80 Filter operators?

Any other leads on calculating the expected heap space allocation to start
the job would be really helpful.

Thanks,
Shailesh


On Sat, Feb 17, 2018 at 5:53 PM, Shailesh Jain <sh...@stellapps.com>
wrote:

> Hi,
>
> I have flink job with almost 300 operators, and every time I'm trying to
> submit the job, the cluster crashes with OutOfMemory exception.
>
> I have 1 job manager and 1 task manager with 2 GB heap space allocated to
> both.
>
> In the configuration section of the documentation
>
>
>
>