You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by "Kali.tummala@gmail.com" <Ka...@gmail.com> on 2016/06/01 15:41:35 UTC

Saprk 1.6 Driver Memory Issue

Hi All , 

I am getting spark driver memory issue even after overriding the conf by
using --conf spark.driver.maxResultSize=20g and I also mentioned in my sql
script (set spark.driver.maxResultSize =16;) but still the same error
happening.

Job aborted due to stage failure: Total size of serialized results of 79
tasks (1035.2 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)

Any thoughts ?

Thanks
Sri 



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Saprk-1-6-Driver-Memory-Issue-tp27063.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Saprk 1.6 Driver Memory Issue

Posted by "Kali.tummala@gmail.com" <Ka...@gmail.com>.

Hi , 

I am using spark-sql shell wile launching I am running it as spark-sql
--conf spark.driver.maxResultSize=20g


I tried using spark-sql --conf "spark.driver.maxResults"="20g" but still no
luck do I need to use set command something like 


spark-sql --conf set "spark.driver.maxReults"="20g"

is it --conf or -conf ?

Thanks







--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Saprk-1-6-Driver-Memory-Issue-tp27063p27066.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Saprk 1.6 Driver Memory Issue

Posted by ashesh_28 <as...@gmail.com>.

Hi Karthik , 

You must set the value before the SparkContext (sc) is created. Also don't
assign too much overhead like 20g for maxResultSize , You can set it to 2G
maximum as per your error message.

Also if you are using Java 1.8 , Please add the below section in your
Yarn-site.xml

 <property>
       <name>yarn.nodemanager.vmem-pmem-ratio</name>
       <value>5</value>
 </property>



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Saprk-1-6-Driver-Memory-Issue-tp27063p27064.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Saprk 1.6 Driver Memory Issue

Posted by Kishoore MV <mv...@outlook.com>.

Hi Kali,
In the shuffle stage maximum memory is 2GB(1024 MB).in your error it is expecting more memory.can you let me know your cluster config details.

Thanks & Regards
Kishore M


> On 01-Jun-2016, at 9:11 PM, "Kali.tummala@gmail.com" <Ka...@gmail.com> wrote:
> 
> Hi All , 
> 
> I am getting spark driver memory issue even after overriding the conf by
> using --conf spark.driver.maxResultSize=20g and I also mentioned in my sql
> script (set spark.driver.maxResultSize =16;) but still the same error
> happening.
> 
> Job aborted due to stage failure: Total size of serialized results of 79
> tasks (1035.2 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
> 
> Any thoughts ?
> 
> Thanks
> Sri 
> 
> 
> 
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Saprk-1-6-Driver-Memory-Issue-tp27063.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org