You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Priya Ch <le...@gmail.com> on 2015/09/08 19:15:11 UTC

foreachRDD causing executor lost failure

Hello All,

 I am using foreachRDD in my code as -

  dstream.foreachRDD { rdd => rdd.foreach { record => // look up with
cassandra table
// save updated rows to cassandra table.
}
}
 This foreachRDD is causing executor lost failure. what is the behavior of
this foreachRDD ???

Thanks,
Padma Ch

Re: foreachRDD causing executor lost failure

Posted by Akhil Das <ak...@sigmoidanalytics.com>.

If you can look a bit in the executor logs, you would see the exact reason
(mostly a OOM/GC etc). Instead of using foreach, try to use mapPartitions
or foreachPartitions.

Thanks
Best Regards

On Tue, Sep 8, 2015 at 10:45 PM, Priya Ch <le...@gmail.com>
wrote:

> Hello All,
>
>  I am using foreachRDD in my code as -
>
>   dstream.foreachRDD { rdd => rdd.foreach { record => // look up with
> cassandra table
> // save updated rows to cassandra table.
> }
> }
>  This foreachRDD is causing executor lost failure. what is the behavior of
> this foreachRDD ???
>
> Thanks,
> Padma Ch
>