You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by AlexModestov <Al...@gmail.com> on 2016/02/20 15:40:56 UTC
spark.driver.maxResultSize doesn't work in conf-file
I have a string spark.driver.maxResultSize=0 in the spark-defaults.conf.
But I get an error:
"org.apache.spark.SparkException: Job aborted due to stage failure: Total
size of serialized results of 18 tasks (1070.5 MB) is bigger than
spark.driver.maxResultSize (1024.0 MB)"
But if I write --conf spark.driver.maxResultSize=0 in pyspark-shell it works
fine.
Could anyone know how to fix it?
Thank you
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-driver-maxResultSize-doesn-t-work-in-conf-file-tp26279.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: spark.driver.maxResultSize doesn't work in conf-file
Posted by Mohannad Ali <ma...@gmail.com>.
In spark-defaults you put the values like "spark.driver.maxResultSize
0" instead of "spark.driver.maxResultSize=0" I think.
On Sat, Feb 20, 2016 at 3:40 PM, AlexModestov <Al...@gmail.com>
wrote:
> I have a string spark.driver.maxResultSize=0 in the spark-defaults.conf.
> But I get an error:
>
> "org.apache.spark.SparkException: Job aborted due to stage failure: Total
> size of serialized results of 18 tasks (1070.5 MB) is bigger than
> spark.driver.maxResultSize (1024.0 MB)"
>
> But if I write --conf spark.driver.maxResultSize=0 in pyspark-shell it
> works
> fine.
>
> Could anyone know how to fix it?
> Thank you
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/spark-driver-maxResultSize-doesn-t-work-in-conf-file-tp26279.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>