RDD recomputation

[souri datta <so...@gmail.com>]

Hi,
 Currently I am trying to optimize my spark application and in that
process, I am trying to figure out if at any stage in the code, I am
recomputing a large RDD (so that I can optimize it by
persisting/checkpointing it).

Is there any indication in the event logs that tells us about an RDD being
computed?
If anyone has done similar analysis, can you please share how you went
about it.

Thanks in advance,
Souri

Re: RDD recomputation

[Kevin Mellott <ke...@gmail.com>]

I've had very good success troubleshooting this type of thing by using the
Spark Web UI, which will depict a breakdown of all tasks. This also
includes the RDDs being used, as well as any cached data. Additional
information about this tool can be found at
http://spark.apache.org/docs/latest/monitoring.html.

On Thu, Mar 10, 2016 at 1:31 PM, souri datta <so...@gmail.com>
wrote:

> Hi,
>  Currently I am trying to optimize my spark application and in that
> process, I am trying to figure out if at any stage in the code, I am
> recomputing a large RDD (so that I can optimize it by
> persisting/checkpointing it).
>
> Is there any indication in the event logs that tells us about an RDD being
> computed?
> If anyone has done similar analysis, can you please share how you went
> about it.
>
> Thanks in advance,
> Souri
>