You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by rajat kumar <ku...@gmail.com> on 2022/09/12 04:15:41 UTC

Long running task in spark

Hello Users,

My 2 tasks are running forever. One of them gave a java heap space error.
I have 10 Joins , all tables are big. I understand this is data skewness.
Apart from changes at code level , any property which can be used in Spark
Config?


I am using Spark2 hence AQE can not be used.


Thanks
Rajat

Re: Long running task in spark

Posted by Sid <fl...@gmail.com>.

Try spark.driver.maxResultsSize =0

On Mon, 12 Sep 2022, 09:46 rajat kumar, <ku...@gmail.com> wrote:

> Hello Users,
>
> My 2 tasks are running forever. One of them gave a java heap space error.
> I have 10 Joins , all tables are big. I understand this is data skewness.
> Apart from changes at code level , any property which can be used in Spark
> Config?
>
>
> I am using Spark2 hence AQE can not be used.
>
>
> Thanks
> Rajat
>