You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by marylucy <qa...@hotmail.com> on 2014/10/20 15:19:39 UTC
How to show RDD size
in spark-shell,I do in follows
val input = sc.textfile("hdfs://192.168.1.10/people/testinput/")
input.cache()
In webui,I cannot see any rdd in storage tab.can anyone tell me how to show rdd size?thank you
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: How to show RDD size
Posted by Nicholas Chammas <ni...@gmail.com>.
No, I believe unpersist acts immediately.
On Mon, Oct 20, 2014 at 10:13 AM, marylucy <qa...@hotmail.com>
wrote:
> thank you for your reply!
> is unpersist operation lazy?if yes,how to decrease memory size as quickly
> as possible
>
> 在 Oct 20, 2014，21:26，"Nicholas Chammas" <ni...@gmail.com> 写道：
>
> I believe it won't show up there until you trigger an action that causes
> the RDD to actually be cached. Remember that certain operations in Spark
> are *lazy*, and caching is one of them.
>
> Nick
>
> On Mon, Oct 20, 2014 at 9:19 AM, marylucy <qa...@hotmail.com>
> wrote:
>
>> in spark-shell,I do in follows
>> val input = sc.textfile("hdfs://192.168.1.10/people/testinput/")
>> input.cache()
>>
>> In webui,I cannot see any rdd in storage tab.can anyone tell me how to
>> show rdd size?thank you
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: How to show RDD size
Posted by marylucy <qa...@hotmail.com>.
thank you for your reply!
is unpersist operation lazy?if yes,how to decrease memory size as quickly as possible
> 在 Oct 20, 2014，21:26，"Nicholas Chammas" <ni...@gmail.com> 写道：
>
> I believe it won't show up there until you trigger an action that causes the RDD to actually be cached. Remember that certain operations in Spark are lazy, and caching is one of them.
>
> Nick
>
>> On Mon, Oct 20, 2014 at 9:19 AM, marylucy <qa...@hotmail.com> wrote:
>> in spark-shell,I do in follows
>> val input = sc.textfile("hdfs://192.168.1.10/people/testinput/")
>> input.cache()
>>
>> In webui,I cannot see any rdd in storage tab.can anyone tell me how to show rdd size?thank you
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>
Re: How to show RDD size
Posted by Nicholas Chammas <ni...@gmail.com>.
I believe it won't show up there until you trigger an action that causes
the RDD to actually be cached. Remember that certain operations in Spark
are *lazy*, and caching is one of them.
Nick
On Mon, Oct 20, 2014 at 9:19 AM, marylucy <qa...@hotmail.com> wrote:
> in spark-shell,I do in follows
> val input = sc.textfile("hdfs://192.168.1.10/people/testinput/")
> input.cache()
>
> In webui,I cannot see any rdd in storage tab.can anyone tell me how to
> show rdd size?thank you
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>