You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Harsha HN <99...@gmail.com> on 2014/09/25 21:55:02 UTC
SPARK UI - Details post job processiong
Hi,
Details laid out in Spark UI for the job in progress is really interesting
and very useful.
But this gets vanished once the job is done.
Is there a way to get job details post processing?
Looking for Spark UI data, not standard input,output and error info.
Thanks,
Harsha
Re: SPARK UI - Details post job processiong
Posted by "Chester @work" <ch...@alpinenow.com>.
I am working on a PR that allows one to send the same spark listener event message back to the application in yarn cluster mode.
So far I have put this function in our application, our UI will receive and display the same spark job event message such as progress, job start, completed etc
Essentially, it establish a communication channel , you can send over progress, messages and detailed exceptions from spark job inside yarn to your application, on you application side , you can display , or log, make use it in other ways.
You can send send message to the running spark job via the channel.
I will cleanup the code and send PR soon
Chester
Alpine Data Lab
Sent from my iPhone
> On Sep 26, 2014, at 7:38 AM, Matt Narrell <ma...@gmail.com> wrote:
>
> Yes, I’m running Hadoop’s Timeline server that does this for the YARN/Hadoop logs (and works very nicely btw). Are you saying I can do the same for the SparkUI as well? Also, where do I set these Spark configurations since this will be executed inside a YARN container? On the “client” machine via spark-env.sh? Do I pass these as command line arguments to spark-submit? Do I set them explicitly on my SparkConf?
>
> Thanks in advance.
>
> mn
>
>> On Sep 25, 2014, at 9:13 PM, Andrew Ash <an...@andrewash.com> wrote:
>>
>> Matt you should be able to set an HDFS path so you'll get logs written to a unified place instead of to local disk on a random box on the cluster.
>>
>>> On Thu, Sep 25, 2014 at 1:38 PM, Matt Narrell <ma...@gmail.com> wrote:
>>> How does this work with a cluster manager like YARN?
>>>
>>> mn
>>>
>>>> On Sep 25, 2014, at 2:23 PM, Andrew Or <an...@databricks.com> wrote:
>>>>
>>>> Hi Harsha,
>>>>
>>>> You can turn on `spark.eventLog.enabled` as documented here: http://spark.apache.org/docs/latest/monitoring.html. Then, if you are running standalone mode, you can access the finished SparkUI through the Master UI. Otherwise, you can start a HistoryServer to display finished UIs.
>>>>
>>>> -Andrew
>>>>
>>>> 2014-09-25 12:55 GMT-07:00 Harsha HN <99...@gmail.com>:
>>>>> Hi,
>>>>>
>>>>> Details laid out in Spark UI for the job in progress is really interesting and very useful.
>>>>> But this gets vanished once the job is done.
>>>>> Is there a way to get job details post processing?
>>>>>
>>>>> Looking for Spark UI data, not standard input,output and error info.
>>>>>
>>>>> Thanks,
>>>>> Harsha
>
Re: SPARK UI - Details post job processiong
Posted by Matt Narrell <ma...@gmail.com>.
Yes, I’m running Hadoop’s Timeline server that does this for the YARN/Hadoop logs (and works very nicely btw). Are you saying I can do the same for the SparkUI as well? Also, where do I set these Spark configurations since this will be executed inside a YARN container? On the “client” machine via spark-env.sh? Do I pass these as command line arguments to spark-submit? Do I set them explicitly on my SparkConf?
Thanks in advance.
mn
On Sep 25, 2014, at 9:13 PM, Andrew Ash <an...@andrewash.com> wrote:
> Matt you should be able to set an HDFS path so you'll get logs written to a unified place instead of to local disk on a random box on the cluster.
>
> On Thu, Sep 25, 2014 at 1:38 PM, Matt Narrell <ma...@gmail.com> wrote:
> How does this work with a cluster manager like YARN?
>
> mn
>
> On Sep 25, 2014, at 2:23 PM, Andrew Or <an...@databricks.com> wrote:
>
>> Hi Harsha,
>>
>> You can turn on `spark.eventLog.enabled` as documented here: http://spark.apache.org/docs/latest/monitoring.html. Then, if you are running standalone mode, you can access the finished SparkUI through the Master UI. Otherwise, you can start a HistoryServer to display finished UIs.
>>
>> -Andrew
>>
>> 2014-09-25 12:55 GMT-07:00 Harsha HN <99...@gmail.com>:
>> Hi,
>>
>> Details laid out in Spark UI for the job in progress is really interesting and very useful.
>> But this gets vanished once the job is done.
>> Is there a way to get job details post processing?
>>
>> Looking for Spark UI data, not standard input,output and error info.
>>
>> Thanks,
>> Harsha
>>
>
>
Re: SPARK UI - Details post job processiong
Posted by Andrew Ash <an...@andrewash.com>.
Matt you should be able to set an HDFS path so you'll get logs written to a
unified place instead of to local disk on a random box on the cluster.
On Thu, Sep 25, 2014 at 1:38 PM, Matt Narrell <ma...@gmail.com>
wrote:
> How does this work with a cluster manager like YARN?
>
> mn
>
> On Sep 25, 2014, at 2:23 PM, Andrew Or <an...@databricks.com> wrote:
>
> Hi Harsha,
>
> You can turn on `spark.eventLog.enabled` as documented here:
> http://spark.apache.org/docs/latest/monitoring.html. Then, if you are
> running standalone mode, you can access the finished SparkUI through the
> Master UI. Otherwise, you can start a HistoryServer to display finished UIs.
>
> -Andrew
>
> 2014-09-25 12:55 GMT-07:00 Harsha HN <99...@gmail.com>:
>
>> Hi,
>>
>> Details laid out in Spark UI for the job in progress is really
>> interesting and very useful.
>> But this gets vanished once the job is done.
>> Is there a way to get job details post processing?
>>
>> Looking for Spark UI data, not standard input,output and error info.
>>
>> Thanks,
>> Harsha
>>
>
>
>
Re: SPARK UI - Details post job processiong
Posted by Matt Narrell <ma...@gmail.com>.
How does this work with a cluster manager like YARN?
mn
On Sep 25, 2014, at 2:23 PM, Andrew Or <an...@databricks.com> wrote:
> Hi Harsha,
>
> You can turn on `spark.eventLog.enabled` as documented here: http://spark.apache.org/docs/latest/monitoring.html. Then, if you are running standalone mode, you can access the finished SparkUI through the Master UI. Otherwise, you can start a HistoryServer to display finished UIs.
>
> -Andrew
>
> 2014-09-25 12:55 GMT-07:00 Harsha HN <99...@gmail.com>:
> Hi,
>
> Details laid out in Spark UI for the job in progress is really interesting and very useful.
> But this gets vanished once the job is done.
> Is there a way to get job details post processing?
>
> Looking for Spark UI data, not standard input,output and error info.
>
> Thanks,
> Harsha
>
Re: SPARK UI - Details post job processiong
Posted by Andrew Or <an...@databricks.com>.
Hi Harsha,
You can turn on `spark.eventLog.enabled` as documented here:
http://spark.apache.org/docs/latest/monitoring.html. Then, if you are
running standalone mode, you can access the finished SparkUI through the
Master UI. Otherwise, you can start a HistoryServer to display finished UIs.
-Andrew
2014-09-25 12:55 GMT-07:00 Harsha HN <99...@gmail.com>:
> Hi,
>
> Details laid out in Spark UI for the job in progress is really interesting
> and very useful.
> But this gets vanished once the job is done.
> Is there a way to get job details post processing?
>
> Looking for Spark UI data, not standard input,output and error info.
>
> Thanks,
> Harsha
>