You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by binbinbin915 <bi...@live.cn> on 2014/07/21 12:46:25 UTC
Can't see any thing one the storage panel of application UI
Hi,
I'm running LogisticRegression of mllib. But I can't see the rdd information
from storage panel.
<http://apache-spark-user-list.1001560.n3.nabble.com/file/n10296/a.png>
<http://apache-spark-user-list.1001560.n3.nabble.com/file/n10296/b.png>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Can-t-see-any-thing-one-the-storage-panel-of-application-UI-tp10296.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Can't see any thing one the storage panel of application UI
Posted by Andrew Or <an...@databricks.com>.
Ah yes, Spark doesn't cache all of your RDDs by default. It turns out that
caching things too aggressively can lead to suboptimal performance because
there might be a lot of churn. If you don't call persist or cache then your
RDDs won't actually be cached. Note that even once they're cached they can
still be kicked out by LRU, however.
-Andrew
2014-08-05 0:13 GMT-07:00 Akhil Das <ak...@sigmoidanalytics.com>:
> You need to use persist or cache those rdds to appear in the Storage.
> Unless you do it, those rdds will be computed again.
>
> Thanks
> Best Regards
>
>
> On Tue, Aug 5, 2014 at 8:03 AM, binbinbin915 <bi...@live.cn> wrote:
>
>> Actually, if you don’t use method like persist or cache, it even not
>> store the rdd to the disk. Every time you use this rdd, they just compute
>> it from the original one.
>>
>> In logistic regression from mllib, they don't persist the changed input ,
>> so I can't see the rdd from the web gui.
>>
>> I have changed the code and gained a 10x speed up.
>>
>> --
>> binbinbin915
>> Sent with Airmail
>>
>>
>> ------------------------------
>> View this message in context: Re: Can't see any thing one the storage
>> panel of application UI
>> <http://apache-spark-user-list.1001560.n3.nabble.com/Can-t-see-any-thing-one-the-storage-panel-of-application-UI-tp10296p11403.html>
>> Sent from the Apache Spark User List mailing list archive
>> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>>
>
>
Re: Can't see any thing one the storage panel of application UI
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
You need to use persist or cache those rdds to appear in the Storage.
Unless you do it, those rdds will be computed again.
Thanks
Best Regards
On Tue, Aug 5, 2014 at 8:03 AM, binbinbin915 <bi...@live.cn> wrote:
> Actually, if you don’t use method like persist or cache, it even not
> store the rdd to the disk. Every time you use this rdd, they just compute
> it from the original one.
>
> In logistic regression from mllib, they don't persist the changed input ,
> so I can't see the rdd from the web gui.
>
> I have changed the code and gained a 10x speed up.
>
> --
> binbinbin915
> Sent with Airmail
>
>
> ------------------------------
> View this message in context: Re: Can't see any thing one the storage
> panel of application UI
> <http://apache-spark-user-list.1001560.n3.nabble.com/Can-t-see-any-thing-one-the-storage-panel-of-application-UI-tp10296p11403.html>
> Sent from the Apache Spark User List mailing list archive
> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>
Re: Can't see any thing one the storage panel of application UI
Posted by binbinbin915 <bi...@live.cn>.
Actually, if you don’t use method like persist or cache, it even not store the rdd to the disk. Every time you use this rdd, they just compute it from the original one.
In logistic regression from mllib, they don't persist the changed input , so I can't see the rdd from the web gui.
I have changed the code and gained a 10x speed up.
--
binbinbin915
Sent with Airmail
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Can-t-see-any-thing-one-the-storage-panel-of-application-UI-tp10296p11403.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Can't see any thing one the storage panel of application UI
Posted by "anthonyjschulte@gmail.com" <an...@gmail.com>.
Good idea Andrew... Using this feature allowed me to debug that my app
wasn't caching properly-- the UI is working as designed for me in 1.0. It
might be a good idea to say "no cached blocks" instead of an empty page...
just a thought...
On Mon, Aug 4, 2014 at 1:17 PM, Andrew Or-2 [via Apache Spark User List] <
ml-node+s1001560n11362h91@n3.nabble.com> wrote:
> Hi all,
>
> Could you check with `sc.getExecutorStorageStatus` to see if the blocks
> are in fact present? This returns a list of StorageStatus objects, and you
> can check whether each status' `blocks` is non-empty. If the blocks do
> exist, then this is likely a bug in the UI. There have been a couple of UI
> fixes since 1.0. Could you check if this is still a problem on the latest
> master: https://github.com/apache/spark
>
> Andrew
>
>
> 2014-08-04 12:10 GMT-07:00 [hidden email]
> <http://user/SendEmail.jtp?type=node&node=11362&i=0> <[hidden email]
> <http://user/SendEmail.jtp?type=node&node=11362&i=1>>:
>
>> I am (not) seeing this also... No items in the storage UI page. using 1.0
>> with HDFS...
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Can-t-see-any-thing-one-the-storage-panel-of-application-UI-tp10296p11361.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> <http://user/SendEmail.jtp?type=node&node=11362&i=2>
>> For additional commands, e-mail: [hidden email]
>> <http://user/SendEmail.jtp?type=node&node=11362&i=3>
>>
>>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Can-t-see-any-thing-one-the-storage-panel-of-application-UI-tp10296p11362.html
> To unsubscribe from Can't see any thing one the storage panel of
> application UI, click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=10296&code=YW50aG9ueWpzY2h1bHRlQGdtYWlsLmNvbXwxMDI5Nnw2ODA5NTUxMDA=>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
A N T H O N Y Ⓙ S C H U L T E
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Can-t-see-any-thing-one-the-storage-panel-of-application-UI-tp10296p11363.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Can't see any thing one the storage panel of application UI
Posted by Andrew Or <an...@databricks.com>.
Hi all,
Could you check with `sc.getExecutorStorageStatus` to see if the blocks are
in fact present? This returns a list of StorageStatus objects, and you can
check whether each status' `blocks` is non-empty. If the blocks do exist,
then this is likely a bug in the UI. There have been a couple of UI fixes
since 1.0. Could you check if this is still a problem on the latest master:
https://github.com/apache/spark
Andrew
2014-08-04 12:10 GMT-07:00 anthonyjschulte@gmail.com <
anthonyjschulte@gmail.com>:
> I am (not) seeing this also... No items in the storage UI page. using 1.0
> with HDFS...
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Can-t-see-any-thing-one-the-storage-panel-of-application-UI-tp10296p11361.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Can't see any thing one the storage panel of application UI
Posted by "anthonyjschulte@gmail.com" <an...@gmail.com>.
I am (not) seeing this also... No items in the storage UI page. using 1.0
with HDFS...
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Can-t-see-any-thing-one-the-storage-panel-of-application-UI-tp10296p11361.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Can't see any thing one the storage panel of application UI
Posted by Preeti Khurana <Pr...@guavus.com>.
Am getting the same issue .
Spark version : 1.0
On 21/07/14 4:16 PM, "binbinbin915" <bi...@live.cn> wrote:
>Hi,
>
>I'm running LogisticRegression of mllib. But I can't see the rdd
>information
>from storage panel.
>
><http://apache-spark-user-list.1001560.n3.nabble.com/file/n10296/a.png>
><http://apache-spark-user-list.1001560.n3.nabble.com/file/n10296/b.png>
>
>
>
>
>
>
>
>--
>View this message in context:
>http://apache-spark-user-list.1001560.n3.nabble.com/Can-t-see-any-thing-on
>e-the-storage-panel-of-application-UI-tp10296.html
>Sent from the Apache Spark User List mailing list archive at Nabble.com.