You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Younes Naguib <Yo...@tritondigital.com> on 2015/10/30 14:46:14 UTC

heap memory

Hi all,

I'm running a spark shell: bin/spark-shell --executor-memory 32G --driver-memory 8G
I keep getting :
                15/10/30 13:41:59 WARN MemoryManager: Total allocation exceeds 95.00% (2,147,483,647 bytes) of heap memory

Any help ?

Thanks,
Younes Naguib
Triton Digital | 1440 Ste-Catherine W., Suite 1200 | Montreal, QC  H3G 1R8
Tel.: +1 514 448 4037 x2688 | Tel.: +1 866 448 4037 x2688 | younes.naguib@tritondigital.com <ma...@streamtheworld.com>

Re: heap memory

Posted by Akhil Das <ak...@sigmoidanalytics.com>.

Its coming from parquet
<https://github.com/Parquet/parquet-mr/blob/master/parquet-hadoop/src/main/java/parquet/hadoop/MemoryManager.java#L112>,
you can try increasing your driver memory and see if its still coming.

Thanks
Best Regards

On Fri, Oct 30, 2015 at 7:16 PM, Younes Naguib <
Younes.Naguib@tritondigital.com> wrote:

> Hi all,
>
>
>
> I’m running a spark shell: bin/spark-shell --executor-memory 32G
> --driver-memory 8G
>
> I keep getting :
>
>                 15/10/30 13:41:59 WARN MemoryManager: Total allocation
> exceeds 95.00% (2,147,483,647 bytes) of heap memory
>
>
>
> Any help ?
>
>
>
> *Thanks,*
>
> *Younes Naguib*
>
> Triton Digital | 1440 Ste-Catherine W., Suite 1200 | Montreal, QC  H3G 1R8
>
> Tel.: +1 514 448 4037 x2688 | Tel.: +1 866 448 4037 x2688 | younes.naguib
> @tritondigital.com <yo...@streamtheworld.com>
>
>
>