You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2019/03/03 19:43:00 UTC
[jira] [Resolved] (SPARK-25552) Upgrade from Spark 1.6.3 to 2.3.0
seems to make jobs use about 50% more memory
[ https://issues.apache.org/jira/browse/SPARK-25552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-25552.
-------------------------------
Resolution: Invalid
This is too broad. Literally 10000 things change from 1.6 to 2.3. You'd have to at least narrow this down to what's taking so much memory.
> Upgrade from Spark 1.6.3 to 2.3.0 seems to make jobs use about 50% more memory
> ------------------------------------------------------------------------------
>
> Key: SPARK-25552
> URL: https://issues.apache.org/jira/browse/SPARK-25552
> Project: Spark
> Issue Type: Bug
> Components: Java API, Spark Core
> Affects Versions: 2.3.0
> Environment: Originally found in an AWS Kubernetes environment with Spark Embedded.
> Also happens in a small scale with Spark Embedded both in Linux and MacOS.
> Reporter: Nuno Azevedo
> Priority: Major
> Attachments: Spark1.6-50GB.png, Spark2.3-50GB.png, Spark2.3-70GB.png
>
>
> After upgrading from Spark 1.6.3 to 2.3.0 our jobs started to need about 50% more memory to run. The Spark properties used were the defaults in both versions.
>
> For instance, before we were running a job with Spark 1.6.3 and it was running fine with 50 GB of memory.
> !Spark1.6-50GB.png|width=800,height=456!
>
> After upgrading to Spark 2.3.0, when running the same job again with the same 50 GB of memory it failed due to out of memory.
> !Spark2.3-50GB.png|width=800,height=366!
>
> Then, we started incrementing the memory until we were able to run the job, which was with 70 GB.
> !Spark2.3-70GB.png|width=800,height=366!
>
> The Spark upgrade was the only change in our environment. After taking a look at what seems to be causing this we noticed that Kryo Serializer is the main culprit for the raise in memory consumption.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org