You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Bariša <ba...@gmail.com> on 2021/02/24 23:25:43 UTC
Is it possible to specify max process memory in flink 1.8.2, similar
to what is possible in flink 1.11
I'm running flink 1.8.2 in a container, and under heavy load, container
gets OOM from the kernel.
I'm guessing that that reason for the kernel OOM is large size of the
off-heap memory. Is there a way I can limit it in flink 1.8.2?
I can see that newer version of flink has a config param, checking here is
it possible to do something similar in flink 1.8.2, without a flink upgrade?
Cheers,
Barisa
Re: Is it possible to specify max process memory in flink 1.8.2,
similar to what is possible in flink 1.11
Posted by Matthias Pohl <ma...@ververica.com>.
Hi Bariša,
have you had the chance to analyze the memory usage in more detail? An
OutOfMemoryError might be an indication for some memory leak which should
be solved instead of lowering some memory configuration parameters. Or is
it that the off-heap memory is not actually used but blocks the JVM from
using the allocated memory for other things?
Best,
Matthias
On Fri, Feb 26, 2021 at 10:05 AM Timo Walther <tw...@apache.org> wrote:
> Hi Barisa,
>
> by looking at the 1.8 documentation [1] it was possible to configure the
> off heap memory as well. Also other memory options were already present.
> So I don't think that you need an upgrade to 1.11 immediately. Please
> let us know if you could fix your problem, otherwise we can try to loop
> in other people that should know better.
>
> Regards,
> Timo
>
>
> [1]
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/config.html#taskmanager-memory-off-heap
>
>
>
> On 25.02.21 15:50, Bariša wrote:
> > Small update:
> > we believe that the off heap memory is used by the parquet writer (
> > used in sink to write to S3 )
> >
> > On Wed, 24 Feb 2021 at 23:25, Bariša <barisa.obradovic@gmail.com
> > <ma...@gmail.com>> wrote:
> >
> > I'm running flink 1.8.2 in a container, and under heavy load,
> > container gets OOM from the kernel.
> > I'm guessing that that reason for the kernel OOM is large size of
> > the off-heap memory. Is there a way I can limit it in flink 1.8.2?
> >
> > I can see that newer version of flink has a config param, checking
> > here is it possible to do something similar in flink 1.8.2, without
> > a flink upgrade?
> >
> > Cheers,
> > Barisa
> >
>
Re: Is it possible to specify max process memory in flink 1.8.2,
similar to what is possible in flink 1.11
Posted by Timo Walther <tw...@apache.org>.
Hi Barisa,
by looking at the 1.8 documentation [1] it was possible to configure the
off heap memory as well. Also other memory options were already present.
So I don't think that you need an upgrade to 1.11 immediately. Please
let us know if you could fix your problem, otherwise we can try to loop
in other people that should know better.
Regards,
Timo
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/config.html#taskmanager-memory-off-heap
On 25.02.21 15:50, Bariša wrote:
> Small update:
> we believe that the off heap memory is used by the parquet writer (
> used in sink to write to S3 )
>
> On Wed, 24 Feb 2021 at 23:25, Bariša <barisa.obradovic@gmail.com
> <ma...@gmail.com>> wrote:
>
> I'm running flink 1.8.2 in a container, and under heavy load,
> container gets OOM from the kernel.
> I'm guessing that that reason for the kernel OOM is large size of
> the off-heap memory. Is there a way I can limit it in flink 1.8.2?
>
> I can see that newer version of flink has a config param, checking
> here is it possible to do something similar in flink 1.8.2, without
> a flink upgrade?
>
> Cheers,
> Barisa
>
Re: Is it possible to specify max process memory in flink 1.8.2,
similar to what is possible in flink 1.11
Posted by Bariša <ba...@gmail.com>.
Small update:
we believe that the off heap memory is used by the parquet writer ( used
in sink to write to S3 )
On Wed, 24 Feb 2021 at 23:25, Bariša <ba...@gmail.com> wrote:
> I'm running flink 1.8.2 in a container, and under heavy load, container
> gets OOM from the kernel.
> I'm guessing that that reason for the kernel OOM is large size of the
> off-heap memory. Is there a way I can limit it in flink 1.8.2?
>
> I can see that newer version of flink has a config param, checking here is
> it possible to do something similar in flink 1.8.2, without a flink upgrade?
>
> Cheers,
> Barisa
>