You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Parag Chaudhari <pa...@gmail.com> on 2017/02/23 01:51:39 UTC
Why spark history server does not show RDD even if it is persisted?
Hi,
I am running spark shell in spark version 2.0.2. Here is my program,
var myrdd = sc.parallelize(Array.range(1, 10))
myrdd.setName("test")
myrdd.cache
myrdd.collect
But I am not able to see any RDD info in "storage" tab in spark history
server.
I looked at this
<https://forums.databricks.com/questions/117/why-is-my-rdd-not-showing-up-in-the-storage-tab-of.html>
but it is not helping as I have exact similar program mentioned there. Can
anyone help?
*Thanks,*
*Parag*
Re: Why spark history server does not show RDD even if it is persisted?
Posted by Parag Chaudhari <pa...@gmail.com>.
Thanks!
*Thanks,Parag Chaudhari,**USC Alumnus (Fight On!)*
*Mobile : (213)-572-7858*
*Profile: http://www.linkedin.com/pub/parag-chaudhari/28/a55/254
<http://www.linkedin.com/pub/parag-chaudhari/28/a55/254>*
On Tue, Feb 28, 2017 at 12:53 PM, Shixiong(Ryan) Zhu <
shixiong@databricks.com> wrote:
> The REST APIs are not just for Spark history server. When an application
> is running, you can use the REST APIs to talk to Spark UI HTTP server as
> well.
>
> On Tue, Feb 28, 2017 at 10:46 AM, Parag Chaudhari <pa...@gmail.com>
> wrote:
>
>> ping...
>>
>>
>>
>> *Thanks,Parag Chaudhari,**USC Alumnus (Fight On!)*
>> *Mobile : (213)-572-7858 <(213)%20572-7858>*
>> *Profile: http://www.linkedin.com/pub/parag-chaudhari/28/a55/254
>> <http://www.linkedin.com/pub/parag-chaudhari/28/a55/254>*
>>
>>
>> On Wed, Feb 22, 2017 at 7:54 PM, Parag Chaudhari <pa...@gmail.com>
>> wrote:
>>
>>> Thanks!
>>>
>>> If spark does not log these events in event log then why spark history
>>> server provides an API to get RDD information?
>>>
>>> From the documentation,
>>>
>>> /applications/[app-id]/storage/rdd A list of stored RDDs for the
>>> given application.
>>>
>>> /applications/[app-id]/storage/rdd/[rdd-id] Details for the storage
>>> status of a given RDD.
>>>
>>>
>>>
>>>
>>> *Thanks,Parag Chaudhari,**USC Alumnus (Fight On!)*
>>> *Mobile : (213)-572-7858 <(213)%20572-7858>*
>>> *Profile: http://www.linkedin.com/pub/parag-chaudhari/28/a55/254
>>> <http://www.linkedin.com/pub/parag-chaudhari/28/a55/254>*
>>>
>>>
>>> On Wed, Feb 22, 2017 at 7:44 PM, Saisai Shao <sa...@gmail.com>
>>> wrote:
>>>
>>>> It is too verbose, and will significantly increase the size event log.
>>>>
>>>> Here is the comment in the code:
>>>>
>>>> // No-op because logging every update would be overkill
>>>>> override def onBlockUpdated(event: SparkListenerBlockUpdated): Unit = {}
>>>>>
>>>>>
>>>> On Thu, Feb 23, 2017 at 11:42 AM, Parag Chaudhari <pa...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks a lot the information!
>>>>>
>>>>> Is there any reason why EventLoggingListener ignore this event?
>>>>>
>>>>> *Thanks,*
>>>>>
>>>>>
>>>>> *Parag*
>>>>>
>>>>> On Wed, Feb 22, 2017 at 7:11 PM, Saisai Shao <sa...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> AFAIK, Spark's EventLoggingListerner ignores BlockUpdate event, so it
>>>>>> will not be written into event-log, I think that's why you cannot get such
>>>>>> info in history server.
>>>>>>
>>>>>> On Thu, Feb 23, 2017 at 9:51 AM, Parag Chaudhari <paragpc89@gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am running spark shell in spark version 2.0.2. Here is my program,
>>>>>>>
>>>>>>> var myrdd = sc.parallelize(Array.range(1, 10))
>>>>>>> myrdd.setName("test")
>>>>>>> myrdd.cache
>>>>>>> myrdd.collect
>>>>>>>
>>>>>>> But I am not able to see any RDD info in "storage" tab in spark
>>>>>>> history server.
>>>>>>>
>>>>>>> I looked at this
>>>>>>> <https://forums.databricks.com/questions/117/why-is-my-rdd-not-showing-up-in-the-storage-tab-of.html>
>>>>>>> but it is not helping as I have exact similar program mentioned there. Can
>>>>>>> anyone help?
>>>>>>>
>>>>>>>
>>>>>>> *Thanks,*
>>>>>>>
>>>>>>> *Parag*
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Why spark history server does not show RDD even if it is persisted?
Posted by "Shixiong(Ryan) Zhu" <sh...@databricks.com>.
The REST APIs are not just for Spark history server. When an application is
running, you can use the REST APIs to talk to Spark UI HTTP server as well.
On Tue, Feb 28, 2017 at 10:46 AM, Parag Chaudhari <pa...@gmail.com>
wrote:
> ping...
>
>
>
> *Thanks,Parag Chaudhari,**USC Alumnus (Fight On!)*
> *Mobile : (213)-572-7858 <(213)%20572-7858>*
> *Profile: http://www.linkedin.com/pub/parag-chaudhari/28/a55/254
> <http://www.linkedin.com/pub/parag-chaudhari/28/a55/254>*
>
>
> On Wed, Feb 22, 2017 at 7:54 PM, Parag Chaudhari <pa...@gmail.com>
> wrote:
>
>> Thanks!
>>
>> If spark does not log these events in event log then why spark history
>> server provides an API to get RDD information?
>>
>> From the documentation,
>>
>> /applications/[app-id]/storage/rdd A list of stored RDDs for the given
>> application.
>>
>> /applications/[app-id]/storage/rdd/[rdd-id] Details for the storage
>> status of a given RDD.
>>
>>
>>
>>
>> *Thanks,Parag Chaudhari,**USC Alumnus (Fight On!)*
>> *Mobile : (213)-572-7858 <(213)%20572-7858>*
>> *Profile: http://www.linkedin.com/pub/parag-chaudhari/28/a55/254
>> <http://www.linkedin.com/pub/parag-chaudhari/28/a55/254>*
>>
>>
>> On Wed, Feb 22, 2017 at 7:44 PM, Saisai Shao <sa...@gmail.com>
>> wrote:
>>
>>> It is too verbose, and will significantly increase the size event log.
>>>
>>> Here is the comment in the code:
>>>
>>> // No-op because logging every update would be overkill
>>>> override def onBlockUpdated(event: SparkListenerBlockUpdated): Unit = {}
>>>>
>>>>
>>> On Thu, Feb 23, 2017 at 11:42 AM, Parag Chaudhari <pa...@gmail.com>
>>> wrote:
>>>
>>>> Thanks a lot the information!
>>>>
>>>> Is there any reason why EventLoggingListener ignore this event?
>>>>
>>>> *Thanks,*
>>>>
>>>>
>>>> *Parag*
>>>>
>>>> On Wed, Feb 22, 2017 at 7:11 PM, Saisai Shao <sa...@gmail.com>
>>>> wrote:
>>>>
>>>>> AFAIK, Spark's EventLoggingListerner ignores BlockUpdate event, so it
>>>>> will not be written into event-log, I think that's why you cannot get such
>>>>> info in history server.
>>>>>
>>>>> On Thu, Feb 23, 2017 at 9:51 AM, Parag Chaudhari <pa...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am running spark shell in spark version 2.0.2. Here is my program,
>>>>>>
>>>>>> var myrdd = sc.parallelize(Array.range(1, 10))
>>>>>> myrdd.setName("test")
>>>>>> myrdd.cache
>>>>>> myrdd.collect
>>>>>>
>>>>>> But I am not able to see any RDD info in "storage" tab in spark
>>>>>> history server.
>>>>>>
>>>>>> I looked at this
>>>>>> <https://forums.databricks.com/questions/117/why-is-my-rdd-not-showing-up-in-the-storage-tab-of.html>
>>>>>> but it is not helping as I have exact similar program mentioned there. Can
>>>>>> anyone help?
>>>>>>
>>>>>>
>>>>>> *Thanks,*
>>>>>>
>>>>>> *Parag*
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Why spark history server does not show RDD even if it is persisted?
Posted by Parag Chaudhari <pa...@gmail.com>.
ping...
*Thanks,Parag Chaudhari,**USC Alumnus (Fight On!)*
*Mobile : (213)-572-7858*
*Profile: http://www.linkedin.com/pub/parag-chaudhari/28/a55/254
<http://www.linkedin.com/pub/parag-chaudhari/28/a55/254>*
On Wed, Feb 22, 2017 at 7:54 PM, Parag Chaudhari <pa...@gmail.com>
wrote:
> Thanks!
>
> If spark does not log these events in event log then why spark history
> server provides an API to get RDD information?
>
> From the documentation,
>
> /applications/[app-id]/storage/rdd A list of stored RDDs for the given
> application.
>
> /applications/[app-id]/storage/rdd/[rdd-id] Details for the storage
> status of a given RDD.
>
>
>
>
> *Thanks,Parag Chaudhari,**USC Alumnus (Fight On!)*
> *Mobile : (213)-572-7858 <(213)%20572-7858>*
> *Profile: http://www.linkedin.com/pub/parag-chaudhari/28/a55/254
> <http://www.linkedin.com/pub/parag-chaudhari/28/a55/254>*
>
>
> On Wed, Feb 22, 2017 at 7:44 PM, Saisai Shao <sa...@gmail.com>
> wrote:
>
>> It is too verbose, and will significantly increase the size event log.
>>
>> Here is the comment in the code:
>>
>> // No-op because logging every update would be overkill
>>> override def onBlockUpdated(event: SparkListenerBlockUpdated): Unit = {}
>>>
>>>
>> On Thu, Feb 23, 2017 at 11:42 AM, Parag Chaudhari <pa...@gmail.com>
>> wrote:
>>
>>> Thanks a lot the information!
>>>
>>> Is there any reason why EventLoggingListener ignore this event?
>>>
>>> *Thanks,*
>>>
>>>
>>> *Parag*
>>>
>>> On Wed, Feb 22, 2017 at 7:11 PM, Saisai Shao <sa...@gmail.com>
>>> wrote:
>>>
>>>> AFAIK, Spark's EventLoggingListerner ignores BlockUpdate event, so it
>>>> will not be written into event-log, I think that's why you cannot get such
>>>> info in history server.
>>>>
>>>> On Thu, Feb 23, 2017 at 9:51 AM, Parag Chaudhari <pa...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am running spark shell in spark version 2.0.2. Here is my program,
>>>>>
>>>>> var myrdd = sc.parallelize(Array.range(1, 10))
>>>>> myrdd.setName("test")
>>>>> myrdd.cache
>>>>> myrdd.collect
>>>>>
>>>>> But I am not able to see any RDD info in "storage" tab in spark
>>>>> history server.
>>>>>
>>>>> I looked at this
>>>>> <https://forums.databricks.com/questions/117/why-is-my-rdd-not-showing-up-in-the-storage-tab-of.html>
>>>>> but it is not helping as I have exact similar program mentioned there. Can
>>>>> anyone help?
>>>>>
>>>>>
>>>>> *Thanks,*
>>>>>
>>>>> *Parag*
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Why spark history server does not show RDD even if it is persisted?
Posted by Parag Chaudhari <pa...@gmail.com>.
Thanks!
If spark does not log these events in event log then why spark history
server provides an API to get RDD information?
From the documentation,
/applications/[app-id]/storage/rdd A list of stored RDDs for the given
application.
/applications/[app-id]/storage/rdd/[rdd-id] Details for the storage
status of a given RDD.
*Thanks,Parag Chaudhari,**USC Alumnus (Fight On!)*
*Mobile : (213)-572-7858*
*Profile: http://www.linkedin.com/pub/parag-chaudhari/28/a55/254
<http://www.linkedin.com/pub/parag-chaudhari/28/a55/254>*
On Wed, Feb 22, 2017 at 7:44 PM, Saisai Shao <sa...@gmail.com> wrote:
> It is too verbose, and will significantly increase the size event log.
>
> Here is the comment in the code:
>
> // No-op because logging every update would be overkill
>> override def onBlockUpdated(event: SparkListenerBlockUpdated): Unit = {}
>>
>>
> On Thu, Feb 23, 2017 at 11:42 AM, Parag Chaudhari <pa...@gmail.com>
> wrote:
>
>> Thanks a lot the information!
>>
>> Is there any reason why EventLoggingListener ignore this event?
>>
>> *Thanks,*
>>
>>
>> *Parag*
>>
>> On Wed, Feb 22, 2017 at 7:11 PM, Saisai Shao <sa...@gmail.com>
>> wrote:
>>
>>> AFAIK, Spark's EventLoggingListerner ignores BlockUpdate event, so it
>>> will not be written into event-log, I think that's why you cannot get such
>>> info in history server.
>>>
>>> On Thu, Feb 23, 2017 at 9:51 AM, Parag Chaudhari <pa...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am running spark shell in spark version 2.0.2. Here is my program,
>>>>
>>>> var myrdd = sc.parallelize(Array.range(1, 10))
>>>> myrdd.setName("test")
>>>> myrdd.cache
>>>> myrdd.collect
>>>>
>>>> But I am not able to see any RDD info in "storage" tab in spark history
>>>> server.
>>>>
>>>> I looked at this
>>>> <https://forums.databricks.com/questions/117/why-is-my-rdd-not-showing-up-in-the-storage-tab-of.html>
>>>> but it is not helping as I have exact similar program mentioned there. Can
>>>> anyone help?
>>>>
>>>>
>>>> *Thanks,*
>>>>
>>>> *Parag*
>>>>
>>>
>>>
>>
>
Re: Why spark history server does not show RDD even if it is persisted?
Posted by Saisai Shao <sa...@gmail.com>.
It is too verbose, and will significantly increase the size event log.
Here is the comment in the code:
// No-op because logging every update would be overkill
> override def onBlockUpdated(event: SparkListenerBlockUpdated): Unit = {}
>
>
On Thu, Feb 23, 2017 at 11:42 AM, Parag Chaudhari <pa...@gmail.com>
wrote:
> Thanks a lot the information!
>
> Is there any reason why EventLoggingListener ignore this event?
>
> *Thanks,*
>
>
> *Parag*
>
> On Wed, Feb 22, 2017 at 7:11 PM, Saisai Shao <sa...@gmail.com>
> wrote:
>
>> AFAIK, Spark's EventLoggingListerner ignores BlockUpdate event, so it
>> will not be written into event-log, I think that's why you cannot get such
>> info in history server.
>>
>> On Thu, Feb 23, 2017 at 9:51 AM, Parag Chaudhari <pa...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I am running spark shell in spark version 2.0.2. Here is my program,
>>>
>>> var myrdd = sc.parallelize(Array.range(1, 10))
>>> myrdd.setName("test")
>>> myrdd.cache
>>> myrdd.collect
>>>
>>> But I am not able to see any RDD info in "storage" tab in spark history
>>> server.
>>>
>>> I looked at this
>>> <https://forums.databricks.com/questions/117/why-is-my-rdd-not-showing-up-in-the-storage-tab-of.html>
>>> but it is not helping as I have exact similar program mentioned there. Can
>>> anyone help?
>>>
>>>
>>> *Thanks,*
>>>
>>> *Parag*
>>>
>>
>>
>
Re: Why spark history server does not show RDD even if it is persisted?
Posted by Parag Chaudhari <pa...@gmail.com>.
Thanks a lot the information!
Is there any reason why EventLoggingListener ignore this event?
*Thanks,*
*Parag*
On Wed, Feb 22, 2017 at 7:11 PM, Saisai Shao <sa...@gmail.com> wrote:
> AFAIK, Spark's EventLoggingListerner ignores BlockUpdate event, so it will
> not be written into event-log, I think that's why you cannot get such info
> in history server.
>
> On Thu, Feb 23, 2017 at 9:51 AM, Parag Chaudhari <pa...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am running spark shell in spark version 2.0.2. Here is my program,
>>
>> var myrdd = sc.parallelize(Array.range(1, 10))
>> myrdd.setName("test")
>> myrdd.cache
>> myrdd.collect
>>
>> But I am not able to see any RDD info in "storage" tab in spark history
>> server.
>>
>> I looked at this
>> <https://forums.databricks.com/questions/117/why-is-my-rdd-not-showing-up-in-the-storage-tab-of.html>
>> but it is not helping as I have exact similar program mentioned there. Can
>> anyone help?
>>
>>
>> *Thanks,*
>>
>> *Parag*
>>
>
>
Re: Why spark history server does not show RDD even if it is persisted?
Posted by Saisai Shao <sa...@gmail.com>.
AFAIK, Spark's EventLoggingListerner ignores BlockUpdate event, so it will
not be written into event-log, I think that's why you cannot get such info
in history server.
On Thu, Feb 23, 2017 at 9:51 AM, Parag Chaudhari <pa...@gmail.com>
wrote:
> Hi,
>
> I am running spark shell in spark version 2.0.2. Here is my program,
>
> var myrdd = sc.parallelize(Array.range(1, 10))
> myrdd.setName("test")
> myrdd.cache
> myrdd.collect
>
> But I am not able to see any RDD info in "storage" tab in spark history
> server.
>
> I looked at this
> <https://forums.databricks.com/questions/117/why-is-my-rdd-not-showing-up-in-the-storage-tab-of.html>
> but it is not helping as I have exact similar program mentioned there. Can
> anyone help?
>
>
> *Thanks,*
>
> *Parag*
>