You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by on 2018/09/05 22:03:06 UTC
Out of Memory errors
Hi
I am doing some processing and creating local list inside a Pardo function. Total data size is 2 GB which is executing inside a single instance. I am running this code on highmem-64 machine
and It gives this error
Shutting down JVM after 8 consecutive periods of measured GC thrashing. Memory is used/total/max = 24239/26759/260864 MB, GC last/max = 67.00/99.00 %, #pushbacks=0, gc thrashing=true. Heap dump written to '/var/log/dataflow/heap_dump.hprof'.
It seems looking at this error - even though Max heap size is much more but 'total memory' can't reach 'max possible' memory and it gives heap issues.
Can you please advise how to make total memory go upto max possible memory.
Re: Out of Memory errors
Posted by .
On 2018/09/06 05:31:45, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> Hi,
>
> AFAIU you are using dataflow runner right ?
>
> Regards
> JB
>
> On 06/09/2018 00:03, asharma.gd@gmail.com wrote:
> > Hi
> >
> > I am doing some processing and creating local list inside a Pardo function. Total data size is 2 GB which is executing inside a single instance. I am running this code on highmem-64 machine
> >
> > and It gives this error
> > Shutting down JVM after 8 consecutive periods of measured GC thrashing. Memory is used/total/max = 24239/26759/260864 MB, GC last/max = 67.00/99.00 %, #pushbacks=0, gc thrashing=true. Heap dump written to '/var/log/dataflow/heap_dump.hprof'.
> >
> > It seems looking at this error - even though Max heap size is much more but 'total memory' can't reach 'max possible' memory and it gives heap issues.
> >
> > Can you please advise how to make total memory go upto max possible memory.
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
yes runner is DataFlow.
Thanks
Aniruddh
Re: Out of Memory errors
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi,
AFAIU you are using dataflow runner right ?
Regards
JB
On 06/09/2018 00:03, asharma.gd@gmail.com wrote:
> Hi
>
> I am doing some processing and creating local list inside a Pardo function. Total data size is 2 GB which is executing inside a single instance. I am running this code on highmem-64 machine
>
> and It gives this error
> Shutting down JVM after 8 consecutive periods of measured GC thrashing. Memory is used/total/max = 24239/26759/260864 MB, GC last/max = 67.00/99.00 %, #pushbacks=0, gc thrashing=true. Heap dump written to '/var/log/dataflow/heap_dump.hprof'.
>
> It seems looking at this error - even though Max heap size is much more but 'total memory' can't reach 'max possible' memory and it gives heap issues.
>
> Can you please advise how to make total memory go upto max possible memory.
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com