Re: Why spark history server does not show RDD even if it is persisted?

[Parag Chaudhari <pa...@gmail.com>]

Thanks!



*Thanks,Parag Chaudhari,**USC Alumnus (Fight On!)*
*Mobile : (213)-572-7858*
*Profile: http://www.linkedin.com/pub/parag-chaudhari/28/a55/254
<http://www.linkedin.com/pub/parag-chaudhari/28/a55/254>*


On Tue, Feb 28, 2017 at 12:53 PM, Shixiong(Ryan) Zhu <
shixiong@databricks.com> wrote:

> The REST APIs are not just for Spark history server. When an application
> is running, you can use the REST APIs to talk to Spark UI HTTP server as
> well.
>
> On Tue, Feb 28, 2017 at 10:46 AM, Parag Chaudhari <pa...@gmail.com>
> wrote:
>
>> ping...
>>
>>
>>
>> *Thanks,Parag Chaudhari,**USC Alumnus (Fight On!)*
>> *Mobile : (213)-572-7858 <(213)%20572-7858>*
>> *Profile: http://www.linkedin.com/pub/parag-chaudhari/28/a55/254
>> <http://www.linkedin.com/pub/parag-chaudhari/28/a55/254>*
>>
>>
>> On Wed, Feb 22, 2017 at 7:54 PM, Parag Chaudhari <pa...@gmail.com>
>> wrote:
>>
>>> Thanks!
>>>
>>> If spark does not log these events in event log then why spark history
>>> server provides an API to get RDD information?
>>>
>>> From the documentation,
>>>
>>> /applications/[app-id]/storage/rdd   A list of stored RDDs for the
>>> given application.
>>>
>>> /applications/[app-id]/storage/rdd/[rdd-id]   Details for the storage
>>> status of a given RDD.
>>>
>>>
>>>
>>>
>>> *Thanks,Parag Chaudhari,**USC Alumnus (Fight On!)*
>>> *Mobile : (213)-572-7858 <(213)%20572-7858>*
>>> *Profile: http://www.linkedin.com/pub/parag-chaudhari/28/a55/254
>>> <http://www.linkedin.com/pub/parag-chaudhari/28/a55/254>*
>>>
>>>
>>> On Wed, Feb 22, 2017 at 7:44 PM, Saisai Shao <sa...@gmail.com>
>>> wrote:
>>>
>>>> It is too verbose, and will significantly increase the size event log.
>>>>
>>>> Here is the comment in the code:
>>>>
>>>> // No-op because logging every update would be overkill
>>>>> override def onBlockUpdated(event: SparkListenerBlockUpdated): Unit = {}
>>>>>
>>>>>
>>>> On Thu, Feb 23, 2017 at 11:42 AM, Parag Chaudhari <pa...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks a lot the information!
>>>>>
>>>>> Is there any reason why EventLoggingListener ignore this event?
>>>>>
>>>>> *Thanks,*
>>>>>
>>>>>
>>>>> *​Parag​*
>>>>>
>>>>> On Wed, Feb 22, 2017 at 7:11 PM, Saisai Shao <sa...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> AFAIK, Spark's EventLoggingListerner ignores BlockUpdate event, so it
>>>>>> will not be written into event-log, I think that's why you cannot get such
>>>>>> info in history server.
>>>>>>
>>>>>> On Thu, Feb 23, 2017 at 9:51 AM, Parag Chaudhari <paragpc89@gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am running spark shell in spark version 2.0.2. Here is my program,
>>>>>>>
>>>>>>> var myrdd = sc.parallelize(Array.range(1, 10))
>>>>>>> myrdd.setName("test")
>>>>>>> myrdd.cache
>>>>>>> myrdd.collect
>>>>>>>
>>>>>>> But I am not able to see any RDD info in "storage" tab in spark
>>>>>>> history server.
>>>>>>>
>>>>>>> I looked at this
>>>>>>> <https://forums.databricks.com/questions/117/why-is-my-rdd-not-showing-up-in-the-storage-tab-of.html>
>>>>>>> but it is not helping as I have exact similar program mentioned there. Can
>>>>>>> anyone help?
>>>>>>>
>>>>>>>
>>>>>>> *Thanks,*
>>>>>>>
>>>>>>> *​Parag​*
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>