You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by shahab <sh...@gmail.com> on 2014/12/03 10:52:00 UTC
How to enforce RDD to be cached?
Hi,
I noticed that rdd.cache() is not happening immediately rather due to lazy
feature of Spark, it is happening just at the moment you perform some
map/reduce actions. Is this true?
If this is the case, how can I enforce Spark to cache immediately at its
cache() statement? I need this to perform some benchmarking and I need to
separate rdd caching and rdd transformation/action processing time.
best,
/Shahab
Re: How to enforce RDD to be cached?
Posted by dsiegel <de...@gmail.com>.
shahabm wrote
> I noticed that rdd.cache() is not happening immediately rather due to lazy
> feature of Spark, it is happening just at the moment you perform some
> map/reduce actions. Is this true?
Yes, .cache() is a transformation (lazy evaluation)
shahabm wrote
> If this is the case, how can I enforce Spark to cache immediately at its
> cache() statement? I need this to perform some benchmarking and I need to
> separate rdd caching and rdd transformation/action processing time.
put an action immediately after .cache()
.cache().first() may be low impact, as it only returns the first element of
the RDD, rather than iterating.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-enforce-RDD-to-be-cached-tp20230p20284.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: How to enforce RDD to be cached?
Posted by shahab <sh...@gmail.com>.
Daniel and Paolo, thanks for the comments.
best,
/Shahab
On Wed, Dec 3, 2014 at 3:12 PM, Paolo Platter <pa...@agilelab.it>
wrote:
> Yes,
>
> otherwise you can try:
>
> rdd.cache().count()
>
> and then run your benchmark
>
> Paolo
>
> *Da:* Daniel Darabos <da...@lynxanalytics.com>
> *Data invio:* mercoledì 3 dicembre 2014 12:28
> *A:* shahab <sh...@gmail.com>
> *Cc:* user@spark.apache.org
>
>
>
> On Wed, Dec 3, 2014 at 10:52 AM, shahab <sh...@gmail.com> wrote:
>
>> Hi,
>>
>> I noticed that rdd.cache() is not happening immediately rather due to
>> lazy feature of Spark, it is happening just at the moment you perform some
>> map/reduce actions. Is this true?
>>
>
> Yes, this is correct.
>
> If this is the case, how can I enforce Spark to cache immediately at
>> its cache() statement? I need this to perform some benchmarking and I need
>> to separate rdd caching and rdd transformation/action processing time.
>>
>
> The typical solution I think is to run rdd.foreach(_ => ()) to trigger a
> calculation.
>
Re: How to enforce RDD to be cached?
Posted by Paolo Platter <pa...@agilelab.it>.
Yes,
otherwise you can try:
rdd.cache().count()
and then run your benchmark
Paolo
Da: Daniel Darabos<ma...@lynxanalytics.com>
Data invio: ?mercoled?? ?3? ?dicembre? ?2014 ?12?:?28
A: shahab<ma...@gmail.com>
Cc: user@spark.apache.org<ma...@spark.apache.org>
On Wed, Dec 3, 2014 at 10:52 AM, shahab <sh...@gmail.com>> wrote:
Hi,
I noticed that rdd.cache() is not happening immediately rather due to lazy feature of Spark, it is happening just at the moment you perform some map/reduce actions. Is this true?
Yes, this is correct.
If this is the case, how can I enforce Spark to cache immediately at its cache() statement? I need this to perform some benchmarking and I need to separate rdd caching and rdd transformation/action processing time.
The typical solution I think is to run rdd.foreach(_ => ()) to trigger a calculation.
Re: How to enforce RDD to be cached?
Posted by Daniel Darabos <da...@lynxanalytics.com>.
On Wed, Dec 3, 2014 at 10:52 AM, shahab <sh...@gmail.com> wrote:
> Hi,
>
> I noticed that rdd.cache() is not happening immediately rather due to lazy
> feature of Spark, it is happening just at the moment you perform some
> map/reduce actions. Is this true?
>
Yes, this is correct.
If this is the case, how can I enforce Spark to cache immediately at its
> cache() statement? I need this to perform some benchmarking and I need to
> separate rdd caching and rdd transformation/action processing time.
>
The typical solution I think is to run rdd.foreach(_ => ()) to trigger a
calculation.