Spark map performance question

[Yadid Ayzenberg <ya...@media.mit.edu>]

Hi All,

I'm trying to understand the performance results I'm getting for the 
following:

rdd = sc.newAPIHadoopRDD( ... )
rdd1 = rdd.keyBy( func1() )
rdd1.count()
rdd1.cache()

rdd2= rdd1.map(func2())
rdd2.count()
rdd3 = rdd2.map(func2())
rdd3.count()

I would expect the 2 maps to be more or less equivalent in terms of 
runtime, however im seeing a more than 2x improvement.
My assumption is that after the first count() operation all of the data 
is already loaded in the RAM and therefor subsequent transformations 
would have equivalent performance. I guess my assumption may be flawed.

Yadid

Re: Spark map performance question

[Yadid Ayzenberg <ya...@media.mit.edu>]

Thanks Mark, that cleared things up for me.
I applied the cache() before the count() and now its behaving as expected.

I really appreciate the fast response.

Yadid




On 12/10/13 12:20 PM, Mark Hamstra wrote:
> You're not marking rdd1 as cached (actually, 
> to-be-cached-after-next-evaluation) until after rdd1.count; so when 
> you hit rdd2.count, rdd1 is not yet cached (no action has been 
> performed on it since it was marked as cached) and has to be 
> completely re-evaluated.  On the other hand, by the time you hit 
> rdd3.count, there has been an action forcing evaluation of rdd1 after 
> it was marked as cached, so rdd1 does not need to be re-evaluated 
> within the rdd3.count job.
>
>
> On Tue, Dec 10, 2013 at 9:11 AM, Yadid Ayzenberg <yadid@media.mit.edu 
> <ma...@media.mit.edu>> wrote:
>
>
>     Hi All,
>
>     I'm trying to understand the performance results I'm getting for
>     the following:
>
>     rdd = sc.newAPIHadoopRDD( ... )
>     rdd1 = rdd.keyBy( func1() )
>     rdd1.count()
>     rdd1.cache()
>
>     rdd2= rdd1.map(func2())
>     rdd2.count()
>     rdd3 = rdd2.map(func2())
>     rdd3.count()
>
>     I would expect the 2 maps to be more or less equivalent in terms
>     of runtime, however im seeing a more than 2x improvement.
>     My assumption is that after the first count() operation all of the
>     data is already loaded in the RAM and therefor subsequent
>     transformations would have equivalent performance. I guess my
>     assumption may be flawed.
>
>     Yadid
>
>

Re: Spark map performance question

[Mark Hamstra <ma...@clearstorydata.com>]

You're not marking rdd1 as cached (actually,
to-be-cached-after-next-evaluation) until after rdd1.count; so when you hit
rdd2.count, rdd1 is not yet cached (no action has been performed on it
since it was marked as cached) and has to be completely re-evaluated.  On
the other hand, by the time you hit rdd3.count, there has been an action
forcing evaluation of rdd1 after it was marked as cached, so rdd1 does not
need to be re-evaluated within the rdd3.count job.


On Tue, Dec 10, 2013 at 9:11 AM, Yadid Ayzenberg <ya...@media.mit.edu>wrote:

>
> Hi All,
>
> I'm trying to understand the performance results I'm getting for the
> following:
>
> rdd = sc.newAPIHadoopRDD( ... )
> rdd1 = rdd.keyBy( func1() )
> rdd1.count()
> rdd1.cache()
>
> rdd2= rdd1.map(func2())
> rdd2.count()
> rdd3 = rdd2.map(func2())
> rdd3.count()
>
> I would expect the 2 maps to be more or less equivalent in terms of
> runtime, however im seeing a more than 2x improvement.
> My assumption is that after the first count() operation all of the data is
> already loaded in the RAM and therefor subsequent transformations would
> have equivalent performance. I guess my assumption may be flawed.
>
> Yadid
>
>