You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sa...@wellsfargo.com on 2015/08/07 21:26:44 UTC
Spark master driver UI: How to keep it after process finished?
Hi,
A silly question here. The Driver Web UI dies when the spark-submit program finish. I would like some time to analyze after the program ends, as the page does not refresh it self, when I hit F5 I lose all the info.
Thanks,
Saif
Re: Spark master driver UI: How to keep it after process finished?
Posted by Andrew Or <an...@databricks.com>.
Hi Saif,
You need to run your application with `spark.eventLog.enabled` set to true.
Then if you are using standalone mode, you can view the Master UI at port
8080. Otherwise, you may start a history server through
`sbin/start-history-server.sh`, which by default starts the history UI at
port 18080.
For more information on how to set this up, visit:
http://spark.apache.org/docs/latest/monitoring.html
-Andrew
2015-08-07 13:16 GMT-07:00 François Pelletier <
newsletters@francoispelletier.org>:
>
> look at
> spark.history.ui.port, if you use standalone
> spark.yarn.historyServer.address, if you use YARN
>
> in your Spark config file
>
> Mine is located at
> /etc/spark/conf/spark-defaults.conf
>
> If you use Apache Ambari you can find this settings in the Spark / Configs
> / Advanced spark-defaults tab
>
> François
>
>
> Le 2015-08-07 15:58, Saif.A.Ellafi@wellsfargo.com a écrit :
>
> Hello, thank you, but that port is unreachable for me. Can you please
> share where can I find that port equivalent in my environment?
>
>
>
> Thank you
>
> Saif
>
>
>
> *From:* François Pelletier [mailto:newsletters@francoispelletier.org
> <ne...@francoispelletier.org>]
> *Sent:* Friday, August 07, 2015 4:38 PM
> *To:* user@spark.apache.org
> *Subject:* Re: Spark master driver UI: How to keep it after process
> finished?
>
>
>
> Hi, all spark processes are saved in the Spark History Server
>
> look at your host on port 18080 instead of 4040
>
> François
>
> Le 2015-08-07 15:26, Saif.A.Ellafi@wellsfargo.com a écrit :
>
> Hi,
>
>
>
> A silly question here. The Driver Web UI dies when the spark-submit
> program finish. I would like some time to analyze after the program ends,
> as the page does not refresh it self, when I hit F5 I lose all the info.
>
>
>
> Thanks,
>
> Saif
>
>
>
>
>
>
>
Re: Spark master driver UI: How to keep it after process finished?
Posted by François Pelletier <ne...@francoispelletier.org>.
look at
spark.history.ui.port, if you use standalone
spark.yarn.historyServer.address, if you use YARN
in your Spark config file
Mine is located at
/etc/spark/conf/spark-defaults.conf
If you use Apache Ambari you can find this settings in the Spark /
Configs / Advanced spark-defaults tab
François
Le 2015-08-07 15:58, Saif.A.Ellafi@wellsfargo.com a écrit :
>
> Hello, thank you, but that port is unreachable for me. Can you please
> share where can I find that port equivalent in my environment?
>
>
>
> Thank you
>
> Saif
>
>
>
> *From:*François Pelletier [mailto:newsletters@francoispelletier.org]
> *Sent:* Friday, August 07, 2015 4:38 PM
> *To:* user@spark.apache.org
> *Subject:* Re: Spark master driver UI: How to keep it after process
> finished?
>
>
>
> Hi, all spark processes are saved in the Spark History Server
>
> look at your host on port 18080 instead of 4040
>
> François
>
> Le 2015-08-07 15:26, Saif.A.Ellafi@wellsfargo.com
> <ma...@wellsfargo.com> a écrit :
>
> Hi,
>
>
>
> A silly question here. The Driver Web UI dies when the
> spark-submit program finish. I would like some time to analyze
> after the program ends, as the page does not refresh it self, when
> I hit F5 I lose all the info.
>
>
>
> Thanks,
>
> Saif
>
>
>
>
>
RE: Spark master driver UI: How to keep it after process finished?
Posted by Koen Vantomme <ko...@gmail.com>.
Verzonden vanaf mijn Sony Xperia™-smartphone
---- Saif.A.Ellafi@wellsfargo.com schreef ----
><!-- /* Font Definitions */ @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} @font-face {font-family:Tahoma; panose-1:2 11 6 4 3 5 4 4 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:12.0pt; font-family:"Times New Roman","serif"; color:black;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:purple; text-decoration:underline;} p.emailquote, li.emailquote, div.emailquote {mso-style-name:emailquote; mso-margin-top-alt:auto; margin-right:0in; mso-margin-bottom-alt:auto; margin-left:1.0pt; border:none; padding:0in; font-size:12.0pt; font-family:"Times New Roman","serif"; color:black;} span.EmailStyle18 {mso-style-type:personal-reply; font-family:"Calibri","sans-serif"; color:#1F497D;} .MsoChpDefault {mso-style-type:export-only; font-size:10.0pt;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} -->
>
>Hello, thank you, but that port is unreachable for me. Can you please share where can I find that port equivalent in my environment?
>
>
>
>Thank you
>
>Saif
>
>
>
>From: François Pelletier [mailto:newsletters@francoispelletier.org]
>Sent: Friday, August 07, 2015 4:38 PM
>To: user@spark.apache.org
>Subject: Re: Spark master driver UI: How to keep it after process finished?
>
>
>
>Hi, all spark processes are saved in the Spark History Server
>
>look at your host on port 18080 instead of 4040
>
>François
>
>Le 2015-08-07 15:26, Saif.A.Ellafi@wellsfargo.com a écrit :
>
>Hi,
>
>
>
>A silly question here. The Driver Web UI dies when the spark-submit program finish. I would like some time to analyze after the program ends, as the page does not refresh it self, when I hit F5 I lose all the info.
>
>
>
>Thanks,
>
>Saif
>
>
>
>
>
RE: Spark master driver UI: How to keep it after process finished?
Posted by Sa...@wellsfargo.com.
Hello, thank you, but that port is unreachable for me. Can you please share where can I find that port equivalent in my environment?
Thank you
Saif
From: François Pelletier [mailto:newsletters@francoispelletier.org]
Sent: Friday, August 07, 2015 4:38 PM
To: user@spark.apache.org
Subject: Re: Spark master driver UI: How to keep it after process finished?
Hi, all spark processes are saved in the Spark History Server
look at your host on port 18080 instead of 4040
François
Le 2015-08-07 15:26, Saif.A.Ellafi@wellsfargo.com<ma...@wellsfargo.com> a écrit :
Hi,
A silly question here. The Driver Web UI dies when the spark-submit program finish. I would like some time to analyze after the program ends, as the page does not refresh it self, when I hit F5 I lose all the info.
Thanks,
Saif
Re: Spark master driver UI: How to keep it after process finished?
Posted by François Pelletier <ne...@francoispelletier.org>.
Hi, all spark processes are saved in the Spark History Server
look at your host on port 18080 instead of 4040
François
Le 2015-08-07 15:26, Saif.A.Ellafi@wellsfargo.com a écrit :
> Hi,
>
> A silly question here. The Driver Web UI dies when the spark-submit
> program finish. I would like some time to analyze after the program
> ends, as the page does not refresh it self, when I hit F5 I lose all
> the info.
>
> Thanks,
> Saif
>