You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by ankit tyagi <an...@gmail.com> on 2015/09/02 08:03:50 UTC
OOM in spark driver
Hi All,
I am using spark-sql 1.3.1 with hadoop 2.4.0 version. I am running sql
query against parquet files and wanted to save result on s3 but looks like
https://issues.apache.org/jira/browse/SPARK-2984 problem still coming while
saving data to s3.
Hence Now i am saving result on hdfs and with the help
of JavaSparkListener, copying file from hdfs to s3 with hadoop fileUtil
in onApplicationEnd method. But my job is getting failed with OOM in spark
driver.
*5/09/02 04:17:57 INFO cluster.YarnClusterSchedulerBackend: Asking each
executor to shut down*
*15/09/02 04:17:59 INFO
scheduler.OutputCommitCoordinator$OutputCommitCoordinatorActor:
OutputCommitCoordinator stopped!*
*Exception in thread "Reporter" *
*Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "Reporter"*
*Exception in thread "SparkListenerBus" *
*Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "SparkListenerBus"*
*Exception in thread "Driver" *
*Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "Driver"*
Strage part is, result is getting saved on HDFS but while copying file job
is getting failed. size of file is under 1MB.
Any help or leads would be appreciated.
Re: OOM in spark driver
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Or you can increase the driver heap space (export _JAVA_OPTIONS="-Xmx5g")
Thanks
Best Regards
On Wed, Sep 2, 2015 at 11:57 PM, Mike Hynes <91...@gmail.com> wrote:
> Just a thought; this has worked for me before on standalone client
> with a similar OOM error in a driver thread. Try setting:
> export SPARK_DAEMON_MEMORY=4G #or whatever size you can afford on your
> machine
> in your environment/spark-env.sh before running spark-submit.
> Mike
>
> On 9/2/15, ankit tyagi <an...@gmail.com> wrote:
> > Hi All,
> >
> > I am using spark-sql 1.3.1 with hadoop 2.4.0 version. I am running sql
> > query against parquet files and wanted to save result on s3 but looks
> like
> > https://issues.apache.org/jira/browse/SPARK-2984 problem still coming
> while
> > saving data to s3.
> >
> > Hence Now i am saving result on hdfs and with the help
> > of JavaSparkListener, copying file from hdfs to s3 with hadoop fileUtil
> > in onApplicationEnd method. But my job is getting failed with OOM in
> spark
> > driver.
> >
> > *5/09/02 04:17:57 INFO cluster.YarnClusterSchedulerBackend: Asking each
> > executor to shut down*
> > *15/09/02 04:17:59 INFO
> > scheduler.OutputCommitCoordinator$OutputCommitCoordinatorActor:
> > OutputCommitCoordinator stopped!*
> > *Exception in thread "Reporter" *
> > *Exception: java.lang.OutOfMemoryError thrown from the
> > UncaughtExceptionHandler in thread "Reporter"*
> > *Exception in thread "SparkListenerBus" *
> > *Exception: java.lang.OutOfMemoryError thrown from the
> > UncaughtExceptionHandler in thread "SparkListenerBus"*
> > *Exception in thread "Driver" *
> > *Exception: java.lang.OutOfMemoryError thrown from the
> > UncaughtExceptionHandler in thread "Driver"*
> >
> >
> > Strage part is, result is getting saved on HDFS but while copying file
> job
> > is getting failed. size of file is under 1MB.
> >
> > Any help or leads would be appreciated.
> >
>
>
> --
> Thanks,
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
Re: OOM in spark driver
Posted by Mike Hynes <91...@gmail.com>.
Just a thought; this has worked for me before on standalone client
with a similar OOM error in a driver thread. Try setting:
export SPARK_DAEMON_MEMORY=4G #or whatever size you can afford on your machine
in your environment/spark-env.sh before running spark-submit.
Mike
On 9/2/15, ankit tyagi <an...@gmail.com> wrote:
> Hi All,
>
> I am using spark-sql 1.3.1 with hadoop 2.4.0 version. I am running sql
> query against parquet files and wanted to save result on s3 but looks like
> https://issues.apache.org/jira/browse/SPARK-2984 problem still coming while
> saving data to s3.
>
> Hence Now i am saving result on hdfs and with the help
> of JavaSparkListener, copying file from hdfs to s3 with hadoop fileUtil
> in onApplicationEnd method. But my job is getting failed with OOM in spark
> driver.
>
> *5/09/02 04:17:57 INFO cluster.YarnClusterSchedulerBackend: Asking each
> executor to shut down*
> *15/09/02 04:17:59 INFO
> scheduler.OutputCommitCoordinator$OutputCommitCoordinatorActor:
> OutputCommitCoordinator stopped!*
> *Exception in thread "Reporter" *
> *Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "Reporter"*
> *Exception in thread "SparkListenerBus" *
> *Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "SparkListenerBus"*
> *Exception in thread "Driver" *
> *Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "Driver"*
>
>
> Strage part is, result is getting saved on HDFS but while copying file job
> is getting failed. size of file is under 1MB.
>
> Any help or leads would be appreciated.
>
--
Thanks,
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org