You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "Ruebenacker, Oliver A" <Ol...@altisource.com> on 2014/09/03 21:12:54 UTC
Web UI
Hello,
What is included in the Spark web UI? What are the available URLs? Can the information be obtained in a machine-readable way (e.g. JSON, XML, etc)?
Thanks!
Best,
Oliver
Oliver Ruebenacker | Solutions Architect
Altisource(tm)
290 Congress St, 7th Floor | Boston, Massachusetts 02210
P: (617) 728-5582 | ext: 275585
Oliver.Ruebenacker@Altisource.com<ma...@Altisource.com> | www.Altisource.com
***********************************************************************************************************************
This email message and any attachments are intended solely for the use of the addressee. If you are not the intended recipient, you are prohibited from reading, disclosing, reproducing, distributing, disseminating or otherwise using this transmission. If you have received this message in error, please promptly notify the sender by reply email and immediately delete this message from your system. This message and any attachments may contain information that is confidential, privileged or exempt from disclosure. Delivery of this message to any person other than the intended recipient is not intended to waive any right or privilege. Message transmission is not guaranteed to be secure or free of software viruses.
***********************************************************************************************************************
Re: Web UI
Posted by Andrew Or <an...@databricks.com>.
Sure, you can request it by filing an issue here:
https://issues.apache.org/jira/browse/SPARK
2014-09-05 6:50 GMT-07:00 Ruebenacker, Oliver A <
Oliver.Ruebenacker@altisource.com>:
>
>
> Hello,
>
>
>
> Thanks for the explanation. So events are stored internally as JSON, but
> there is no official support for having Spark serve that JSON via HTTP? So
> if I wanted to write an app that monitors Spark, I would either have to
> scrape the web UI in HTML or rely on unofficial JSON features? That is
> quite surprising, because I would expect dumping out the JSON would be
> easier for Spark developers to implement than converting it to HTML.
>
>
>
> Do I get that right? Should I make a feature request? Thanks!
>
>
>
> Best, Oliver
>
>
>
> *From:* Andrew Or [mailto:andrew@databricks.com]
> *Sent:* Thursday, September 04, 2014 2:11 PM
> *To:* Ruebenacker, Oliver A
> *Cc:* Akhil Das; Wonha Ryu; user@spark.apache.org
> *Subject:* Re: Web UI
>
>
>
> Hi all,
>
>
>
> The JSON version of the web UI is not officially supported; I don't
> believe this is documented anywhere.
>
>
>
> The alternative is to set `spark.eventLog.enabled` to true before running
> your application. This will create JSON SparkListenerEvents with details
> about each task and stage as a log file. Then you can easily reconstruct
> the web UI after the application has exited. This is what the standalone
> Master and the History Server does, actually. For local mode, you can use
> the latter to generate your UI after the fact. (This is documented here:
> http://spark.apache.org/docs/latest/monitoring.html).
>
>
>
> -Andrew
>
>
>
> 2014-09-04 5:28 GMT-07:00 Ruebenacker, Oliver A <
> Oliver.Ruebenacker@altisource.com>:
>
>
>
> Hello,
>
>
>
> Thanks for the link – this is for standalone, though, and most URLs
> don’t work for local.
>
> I will look into deploying as standalone on a single node for testing
> and development.
>
>
>
> Best, Oliver
>
>
>
> *From:* Akhil Das [mailto:akhil@sigmoidanalytics.com]
> *Sent:* Thursday, September 04, 2014 3:09 AM
> *To:* Ruebenacker, Oliver A
> *Cc:* Wonha Ryu; user@spark.apache.org
> *Subject:* Re: Web UI
>
>
>
> Hi
>
>
>
> You can see this doc
> <https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security>
> for all the available webUI ports.
>
>
>
> Yes there are ways to get the data metrics in Json format, One of them is
> below:
>
>
>
> **
>
> *http://webUI:8080/json/ <http://webUI:8080/json/>* Or simply
>
> **
>
> *curl webUI:8080/json/*
>
>
>
> There are some PRs about it you can read it over here
> https://github.com/apache/spark/pull/1682
>
>
> Thanks
>
> Best Regards
>
>
>
> On Thu, Sep 4, 2014 at 2:24 AM, Ruebenacker, Oliver A <
> Oliver.Ruebenacker@altisource.com> wrote:
>
>
>
> Hello,
>
>
>
> Interestingly, http://localhost:4040/metrics/json/ gives some numbers,
> but only a few which never seem to change during the application’s lifetime.
>
>
>
> Either the web UI has some very strange limitations, or there are some
> URLs yet to be discovered that do something interesting.
>
>
>
> Best,
>
> Oliver
>
>
>
>
>
> *From:* Wonha Ryu [mailto:wonha.ryu@gmail.com]
> *Sent:* Wednesday, September 03, 2014 4:27 PM
>
>
> *To:* Ruebenacker, Oliver A
> *Cc:* user@spark.apache.org
> *Subject:* Re: Web UI
>
>
>
> Hey Oliver,
>
>
>
> IIRC there's no JSON endpoint for application web UI. They only exist for
> cluster master and worker.
>
>
>
> - Wonha
>
>
>
>
>
> On Wed, Sep 3, 2014 at 12:58 PM, Ruebenacker, Oliver A <
> Oliver.Ruebenacker@altisource.com> wrote:
>
>
>
> Hello,
>
>
>
> Thanks for the help! But I tried starting with “–master local[4]” and
> when I load http://localhost:4040/json I just get forwarded to
> http://localhost:4040/stages/, and it’s all human-readable HTML, no JSON.
>
>
>
> Best,
>
> Oliver
>
>
>
>
>
> *From:* Wonha Ryu [mailto:wonha.ryu@gmail.com]
> *Sent:* Wednesday, September 03, 2014 3:36 PM
> *To:* Ruebenacker, Oliver A
> *Cc:* user@spark.apache.org
> *Subject:* Re: Web UI
>
>
>
> Hi Oliver,
>
>
>
> Spark standalone master and worker support '/json' endpoint in web UI,
> which returns some of the information in JSON format.
>
> I wasn't able to find relevant documentation, though.
>
>
>
> - Wonha
>
>
>
> On Wed, Sep 3, 2014 at 12:12 PM, Ruebenacker, Oliver A <
> Oliver.Ruebenacker@altisource.com> wrote:
>
>
>
> Hello,
>
>
>
> What is included in the Spark web UI? What are the available URLs? Can
> the information be obtained in a machine-readable way (e.g. JSON, XML, etc)?
>
>
>
> Thanks!
>
>
>
> Best,
>
> Oliver
>
>
>
> Oliver Ruebenacker | Solutions Architect
>
>
>
> Altisource™
>
> 290 Congress St, 7th Floor | Boston, Massachusetts 02210
>
> P: (617) 728-5582 | ext: 275585
>
> Oliver.Ruebenacker@Altisource.com | www.Altisource.com
>
>
>
>
> ***********************************************************************************************************************
>
> This email message and any attachments are intended solely for the use of
> the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system.
>
> This message and any attachments may contain information that is
> confidential, privileged or exempt from disclosure. Delivery of this
> message to any person other than the intended recipient is not intended to
> waive any right or privilege. Message transmission is not guaranteed to be
> secure or free of software viruses.
>
> ***********************************************************************************************************************
>
>
>
>
> ***********************************************************************************************************************
>
> This email message and any attachments are intended solely for the use of
> the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system.
>
> This message and any attachments may contain information that is
> confidential, privileged or exempt from disclosure. Delivery of this
> message to any person other than the intended recipient is not intended to
> waive any right or privilege. Message transmission is not guaranteed to be
> secure or free of software viruses.
>
> ***********************************************************************************************************************
>
>
>
>
> ***********************************************************************************************************************
>
> This email message and any attachments are intended solely for the use of
> the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system.
>
> This message and any attachments may contain information that is
> confidential, privileged or exempt from disclosure. Delivery of this
> message to any person other than the intended recipient is not intended to
> waive any right or privilege. Message transmission is not guaranteed to be
> secure or free of software viruses.
>
> ***********************************************************************************************************************
>
>
>
>
> ***********************************************************************************************************************
>
> This email message and any attachments are intended solely for the use of
> the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system.
>
> This message and any attachments may contain information that is
> confidential, privileged or exempt from disclosure. Delivery of this
> message to any person other than the intended recipient is not intended to
> waive any right or privilege. Message transmission is not guaranteed to be
> secure or free of software viruses.
>
> ***********************************************************************************************************************
>
>
>
>
> ***********************************************************************************************************************
>
> This email message and any attachments are intended solely for the use of
> the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system.
> This message and any attachments may contain information that is
> confidential, privileged or exempt from disclosure. Delivery of this
> message to any person other than the intended recipient is not intended to
> waive any right or privilege. Message transmission is not guaranteed to be
> secure or free of software viruses.
>
> ***********************************************************************************************************************
>
RE: Web UI
Posted by "Ruebenacker, Oliver A" <Ol...@altisource.com>.
Hello,
Thanks for the explanation. So events are stored internally as JSON, but there is no official support for having Spark serve that JSON via HTTP? So if I wanted to write an app that monitors Spark, I would either have to scrape the web UI in HTML or rely on unofficial JSON features? That is quite surprising, because I would expect dumping out the JSON would be easier for Spark developers to implement than converting it to HTML.
Do I get that right? Should I make a feature request? Thanks!
Best, Oliver
From: Andrew Or [mailto:andrew@databricks.com]
Sent: Thursday, September 04, 2014 2:11 PM
To: Ruebenacker, Oliver A
Cc: Akhil Das; Wonha Ryu; user@spark.apache.org
Subject: Re: Web UI
Hi all,
The JSON version of the web UI is not officially supported; I don't believe this is documented anywhere.
The alternative is to set `spark.eventLog.enabled` to true before running your application. This will create JSON SparkListenerEvents with details about each task and stage as a log file. Then you can easily reconstruct the web UI after the application has exited. This is what the standalone Master and the History Server does, actually. For local mode, you can use the latter to generate your UI after the fact. (This is documented here: http://spark.apache.org/docs/latest/monitoring.html).
-Andrew
2014-09-04 5:28 GMT-07:00 Ruebenacker, Oliver A <Ol...@altisource.com>>:
Hello,
Thanks for the link – this is for standalone, though, and most URLs don’t work for local.
I will look into deploying as standalone on a single node for testing and development.
Best, Oliver
From: Akhil Das [mailto:akhil@sigmoidanalytics.com<ma...@sigmoidanalytics.com>]
Sent: Thursday, September 04, 2014 3:09 AM
To: Ruebenacker, Oliver A
Cc: Wonha Ryu; user@spark.apache.org<ma...@spark.apache.org>
Subject: Re: Web UI
Hi
You can see this doc<https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security> for all the available webUI ports.
Yes there are ways to get the data metrics in Json format, One of them is below:
http://webUI:8080/json/ Or simply
curl webUI:8080/json/
There are some PRs about it you can read it over here https://github.com/apache/spark/pull/1682
Thanks
Best Regards
On Thu, Sep 4, 2014 at 2:24 AM, Ruebenacker, Oliver A <Ol...@altisource.com>> wrote:
Hello,
Interestingly, http://localhost:4040/metrics/json/ gives some numbers, but only a few which never seem to change during the application’s lifetime.
Either the web UI has some very strange limitations, or there are some URLs yet to be discovered that do something interesting.
Best,
Oliver
From: Wonha Ryu [mailto:wonha.ryu@gmail.com<ma...@gmail.com>]
Sent: Wednesday, September 03, 2014 4:27 PM
To: Ruebenacker, Oliver A
Cc: user@spark.apache.org<ma...@spark.apache.org>
Subject: Re: Web UI
Hey Oliver,
IIRC there's no JSON endpoint for application web UI. They only exist for cluster master and worker.
- Wonha
On Wed, Sep 3, 2014 at 12:58 PM, Ruebenacker, Oliver A <Ol...@altisource.com>> wrote:
Hello,
Thanks for the help! But I tried starting with “–master local[4]” and when I load http://localhost:4040/json I just get forwarded to http://localhost:4040/stages/, and it’s all human-readable HTML, no JSON.
Best,
Oliver
From: Wonha Ryu [mailto:wonha.ryu@gmail.com<ma...@gmail.com>]
Sent: Wednesday, September 03, 2014 3:36 PM
To: Ruebenacker, Oliver A
Cc: user@spark.apache.org<ma...@spark.apache.org>
Subject: Re: Web UI
Hi Oliver,
Spark standalone master and worker support '/json' endpoint in web UI, which returns some of the information in JSON format.
I wasn't able to find relevant documentation, though.
- Wonha
On Wed, Sep 3, 2014 at 12:12 PM, Ruebenacker, Oliver A <Ol...@altisource.com>> wrote:
Hello,
What is included in the Spark web UI? What are the available URLs? Can the information be obtained in a machine-readable way (e.g. JSON, XML, etc)?
Thanks!
Best,
Oliver
Oliver Ruebenacker | Solutions Architect
Altisource™
290 Congress St, 7th Floor | Boston, Massachusetts 02210
P: (617) 728-5582<tel:%28617%29%20728-5582> | ext: 275585
Oliver.Ruebenacker@Altisource.com<ma...@Altisource.com> | www.Altisource.com<http://www.Altisource.com>
***********************************************************************************************************************
This email message and any attachments are intended solely for the use of the addressee. If you are not the intended recipient, you are prohibited from reading, disclosing, reproducing, distributing, disseminating or otherwise using this transmission. If you have received this message in error, please promptly notify the sender by reply email and immediately delete this message from your system.
This message and any attachments may contain information that is confidential, privileged or exempt from disclosure. Delivery of this message to any person other than the intended recipient is not intended to waive any right or privilege. Message transmission is not guaranteed to be secure or free of software viruses.
***********************************************************************************************************************
***********************************************************************************************************************
This email message and any attachments are intended solely for the use of the addressee. If you are not the intended recipient, you are prohibited from reading, disclosing, reproducing, distributing, disseminating or otherwise using this transmission. If you have received this message in error, please promptly notify the sender by reply email and immediately delete this message from your system.
This message and any attachments may contain information that is confidential, privileged or exempt from disclosure. Delivery of this message to any person other than the intended recipient is not intended to waive any right or privilege. Message transmission is not guaranteed to be secure or free of software viruses.
***********************************************************************************************************************
***********************************************************************************************************************
This email message and any attachments are intended solely for the use of the addressee. If you are not the intended recipient, you are prohibited from reading, disclosing, reproducing, distributing, disseminating or otherwise using this transmission. If you have received this message in error, please promptly notify the sender by reply email and immediately delete this message from your system.
This message and any attachments may contain information that is confidential, privileged or exempt from disclosure. Delivery of this message to any person other than the intended recipient is not intended to waive any right or privilege. Message transmission is not guaranteed to be secure or free of software viruses.
***********************************************************************************************************************
***********************************************************************************************************************
This email message and any attachments are intended solely for the use of the addressee. If you are not the intended recipient, you are prohibited from reading, disclosing, reproducing, distributing, disseminating or otherwise using this transmission. If you have received this message in error, please promptly notify the sender by reply email and immediately delete this message from your system.
This message and any attachments may contain information that is confidential, privileged or exempt from disclosure. Delivery of this message to any person other than the intended recipient is not intended to waive any right or privilege. Message transmission is not guaranteed to be secure or free of software viruses.
***********************************************************************************************************************
***********************************************************************************************************************
This email message and any attachments are intended solely for the use of the addressee. If you are not the intended recipient, you are prohibited from reading, disclosing, reproducing, distributing, disseminating or otherwise using this transmission. If you have received this message in error, please promptly notify the sender by reply email and immediately delete this message from your system. This message and any attachments may contain information that is confidential, privileged or exempt from disclosure. Delivery of this message to any person other than the intended recipient is not intended to waive any right or privilege. Message transmission is not guaranteed to be secure or free of software viruses.
***********************************************************************************************************************
Re: Web UI
Posted by Andrew Or <an...@databricks.com>.
Hi all,
The JSON version of the web UI is not officially supported; I don't believe
this is documented anywhere.
The alternative is to set `spark.eventLog.enabled` to true before running
your application. This will create JSON SparkListenerEvents with details
about each task and stage as a log file. Then you can easily reconstruct
the web UI after the application has exited. This is what the standalone
Master and the History Server does, actually. For local mode, you can use
the latter to generate your UI after the fact. (This is documented here:
http://spark.apache.org/docs/latest/monitoring.html).
-Andrew
2014-09-04 5:28 GMT-07:00 Ruebenacker, Oliver A <
Oliver.Ruebenacker@altisource.com>:
>
>
> Hello,
>
>
>
> Thanks for the link – this is for standalone, though, and most URLs
> don’t work for local.
>
> I will look into deploying as standalone on a single node for testing
> and development.
>
>
>
> Best, Oliver
>
>
>
> *From:* Akhil Das [mailto:akhil@sigmoidanalytics.com]
> *Sent:* Thursday, September 04, 2014 3:09 AM
> *To:* Ruebenacker, Oliver A
> *Cc:* Wonha Ryu; user@spark.apache.org
> *Subject:* Re: Web UI
>
>
>
> Hi
>
>
>
> You can see this doc
> <https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security>
> for all the available webUI ports.
>
>
>
> Yes there are ways to get the data metrics in Json format, One of them is
> below:
>
>
>
> **
>
> *http://webUI:8080/json/ <http://webUI:8080/json/>* Or simply
>
> **
>
> *curl webUI:8080/json/*
>
>
>
> There are some PRs about it you can read it over here
> https://github.com/apache/spark/pull/1682
>
>
> Thanks
>
> Best Regards
>
>
>
> On Thu, Sep 4, 2014 at 2:24 AM, Ruebenacker, Oliver A <
> Oliver.Ruebenacker@altisource.com> wrote:
>
>
>
> Hello,
>
>
>
> Interestingly, http://localhost:4040/metrics/json/ gives some numbers,
> but only a few which never seem to change during the application’s lifetime.
>
>
>
> Either the web UI has some very strange limitations, or there are some
> URLs yet to be discovered that do something interesting.
>
>
>
> Best,
>
> Oliver
>
>
>
>
>
> *From:* Wonha Ryu [mailto:wonha.ryu@gmail.com]
> *Sent:* Wednesday, September 03, 2014 4:27 PM
>
>
> *To:* Ruebenacker, Oliver A
> *Cc:* user@spark.apache.org
> *Subject:* Re: Web UI
>
>
>
> Hey Oliver,
>
>
>
> IIRC there's no JSON endpoint for application web UI. They only exist for
> cluster master and worker.
>
>
>
> - Wonha
>
>
>
>
>
> On Wed, Sep 3, 2014 at 12:58 PM, Ruebenacker, Oliver A <
> Oliver.Ruebenacker@altisource.com> wrote:
>
>
>
> Hello,
>
>
>
> Thanks for the help! But I tried starting with “–master local[4]” and
> when I load http://localhost:4040/json I just get forwarded to
> http://localhost:4040/stages/, and it’s all human-readable HTML, no JSON.
>
>
>
> Best,
>
> Oliver
>
>
>
>
>
> *From:* Wonha Ryu [mailto:wonha.ryu@gmail.com]
> *Sent:* Wednesday, September 03, 2014 3:36 PM
> *To:* Ruebenacker, Oliver A
> *Cc:* user@spark.apache.org
> *Subject:* Re: Web UI
>
>
>
> Hi Oliver,
>
>
>
> Spark standalone master and worker support '/json' endpoint in web UI,
> which returns some of the information in JSON format.
>
> I wasn't able to find relevant documentation, though.
>
>
>
> - Wonha
>
>
>
> On Wed, Sep 3, 2014 at 12:12 PM, Ruebenacker, Oliver A <
> Oliver.Ruebenacker@altisource.com> wrote:
>
>
>
> Hello,
>
>
>
> What is included in the Spark web UI? What are the available URLs? Can
> the information be obtained in a machine-readable way (e.g. JSON, XML, etc)?
>
>
>
> Thanks!
>
>
>
> Best,
>
> Oliver
>
>
>
> Oliver Ruebenacker | Solutions Architect
>
>
>
> Altisource™
>
> 290 Congress St, 7th Floor | Boston, Massachusetts 02210
>
> P: (617) 728-5582 | ext: 275585
>
> Oliver.Ruebenacker@Altisource.com | www.Altisource.com
>
>
>
>
> ***********************************************************************************************************************
>
> This email message and any attachments are intended solely for the use of
> the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system.
>
> This message and any attachments may contain information that is
> confidential, privileged or exempt from disclosure. Delivery of this
> message to any person other than the intended recipient is not intended to
> waive any right or privilege. Message transmission is not guaranteed to be
> secure or free of software viruses.
>
> ***********************************************************************************************************************
>
>
>
>
> ***********************************************************************************************************************
>
> This email message and any attachments are intended solely for the use of
> the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system.
>
> This message and any attachments may contain information that is
> confidential, privileged or exempt from disclosure. Delivery of this
> message to any person other than the intended recipient is not intended to
> waive any right or privilege. Message transmission is not guaranteed to be
> secure or free of software viruses.
>
> ***********************************************************************************************************************
>
>
>
>
> ***********************************************************************************************************************
>
> This email message and any attachments are intended solely for the use of
> the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system.
>
> This message and any attachments may contain information that is
> confidential, privileged or exempt from disclosure. Delivery of this
> message to any person other than the intended recipient is not intended to
> waive any right or privilege. Message transmission is not guaranteed to be
> secure or free of software viruses.
>
> ***********************************************************************************************************************
>
>
>
>
> ***********************************************************************************************************************
>
> This email message and any attachments are intended solely for the use of
> the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system.
> This message and any attachments may contain information that is
> confidential, privileged or exempt from disclosure. Delivery of this
> message to any person other than the intended recipient is not intended to
> waive any right or privilege. Message transmission is not guaranteed to be
> secure or free of software viruses.
>
> ***********************************************************************************************************************
>
RE: Web UI
Posted by "Ruebenacker, Oliver A" <Ol...@altisource.com>.
Hello,
Thanks for the link – this is for standalone, though, and most URLs don’t work for local.
I will look into deploying as standalone on a single node for testing and development.
Best, Oliver
From: Akhil Das [mailto:akhil@sigmoidanalytics.com]
Sent: Thursday, September 04, 2014 3:09 AM
To: Ruebenacker, Oliver A
Cc: Wonha Ryu; user@spark.apache.org
Subject: Re: Web UI
Hi
You can see this doc<https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security> for all the available webUI ports.
Yes there are ways to get the data metrics in Json format, One of them is below:
http://webUI:8080/json/ Or simply
curl webUI:8080/json/
There are some PRs about it you can read it over here https://github.com/apache/spark/pull/1682
Thanks
Best Regards
On Thu, Sep 4, 2014 at 2:24 AM, Ruebenacker, Oliver A <Ol...@altisource.com>> wrote:
Hello,
Interestingly, http://localhost:4040/metrics/json/ gives some numbers, but only a few which never seem to change during the application’s lifetime.
Either the web UI has some very strange limitations, or there are some URLs yet to be discovered that do something interesting.
Best,
Oliver
From: Wonha Ryu [mailto:wonha.ryu@gmail.com<ma...@gmail.com>]
Sent: Wednesday, September 03, 2014 4:27 PM
To: Ruebenacker, Oliver A
Cc: user@spark.apache.org<ma...@spark.apache.org>
Subject: Re: Web UI
Hey Oliver,
IIRC there's no JSON endpoint for application web UI. They only exist for cluster master and worker.
- Wonha
On Wed, Sep 3, 2014 at 12:58 PM, Ruebenacker, Oliver A <Ol...@altisource.com>> wrote:
Hello,
Thanks for the help! But I tried starting with “–master local[4]” and when I load http://localhost:4040/json I just get forwarded to http://localhost:4040/stages/, and it’s all human-readable HTML, no JSON.
Best,
Oliver
From: Wonha Ryu [mailto:wonha.ryu@gmail.com<ma...@gmail.com>]
Sent: Wednesday, September 03, 2014 3:36 PM
To: Ruebenacker, Oliver A
Cc: user@spark.apache.org<ma...@spark.apache.org>
Subject: Re: Web UI
Hi Oliver,
Spark standalone master and worker support '/json' endpoint in web UI, which returns some of the information in JSON format.
I wasn't able to find relevant documentation, though.
- Wonha
On Wed, Sep 3, 2014 at 12:12 PM, Ruebenacker, Oliver A <Ol...@altisource.com>> wrote:
Hello,
What is included in the Spark web UI? What are the available URLs? Can the information be obtained in a machine-readable way (e.g. JSON, XML, etc)?
Thanks!
Best,
Oliver
Oliver Ruebenacker | Solutions Architect
Altisource™
290 Congress St, 7th Floor | Boston, Massachusetts 02210
P: (617) 728-5582<tel:%28617%29%20728-5582> | ext: 275585
Oliver.Ruebenacker@Altisource.com<ma...@Altisource.com> | www.Altisource.com<http://www.Altisource.com>
***********************************************************************************************************************
This email message and any attachments are intended solely for the use of the addressee. If you are not the intended recipient, you are prohibited from reading, disclosing, reproducing, distributing, disseminating or otherwise using this transmission. If you have received this message in error, please promptly notify the sender by reply email and immediately delete this message from your system.
This message and any attachments may contain information that is confidential, privileged or exempt from disclosure. Delivery of this message to any person other than the intended recipient is not intended to waive any right or privilege. Message transmission is not guaranteed to be secure or free of software viruses.
***********************************************************************************************************************
***********************************************************************************************************************
This email message and any attachments are intended solely for the use of the addressee. If you are not the intended recipient, you are prohibited from reading, disclosing, reproducing, distributing, disseminating or otherwise using this transmission. If you have received this message in error, please promptly notify the sender by reply email and immediately delete this message from your system.
This message and any attachments may contain information that is confidential, privileged or exempt from disclosure. Delivery of this message to any person other than the intended recipient is not intended to waive any right or privilege. Message transmission is not guaranteed to be secure or free of software viruses.
***********************************************************************************************************************
***********************************************************************************************************************
This email message and any attachments are intended solely for the use of the addressee. If you are not the intended recipient, you are prohibited from reading, disclosing, reproducing, distributing, disseminating or otherwise using this transmission. If you have received this message in error, please promptly notify the sender by reply email and immediately delete this message from your system.
This message and any attachments may contain information that is confidential, privileged or exempt from disclosure. Delivery of this message to any person other than the intended recipient is not intended to waive any right or privilege. Message transmission is not guaranteed to be secure or free of software viruses.
***********************************************************************************************************************
***********************************************************************************************************************
This email message and any attachments are intended solely for the use of the addressee. If you are not the intended recipient, you are prohibited from reading, disclosing, reproducing, distributing, disseminating or otherwise using this transmission. If you have received this message in error, please promptly notify the sender by reply email and immediately delete this message from your system. This message and any attachments may contain information that is confidential, privileged or exempt from disclosure. Delivery of this message to any person other than the intended recipient is not intended to waive any right or privilege. Message transmission is not guaranteed to be secure or free of software viruses.
***********************************************************************************************************************
Re: Web UI
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Hi
You can see this doc
<https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security>
for all the available webUI ports.
Yes there are ways to get the data metrics in Json format, One of them is
below:
*http://webUI:8080/json/ <http://webUI:8080/json/>* Or simply
> *curl webUI:8080/json/*
There are some PRs about it you can read it over here
https://github.com/apache/spark/pull/1682
Thanks
Best Regards
On Thu, Sep 4, 2014 at 2:24 AM, Ruebenacker, Oliver A <
Oliver.Ruebenacker@altisource.com> wrote:
>
>
> Hello,
>
>
>
> Interestingly, http://localhost:4040/metrics/json/ gives some numbers,
> but only a few which never seem to change during the application’s lifetime.
>
>
>
> Either the web UI has some very strange limitations, or there are some
> URLs yet to be discovered that do something interesting.
>
>
>
> Best,
>
> Oliver
>
>
>
>
>
> *From:* Wonha Ryu [mailto:wonha.ryu@gmail.com]
> *Sent:* Wednesday, September 03, 2014 4:27 PM
>
> *To:* Ruebenacker, Oliver A
> *Cc:* user@spark.apache.org
> *Subject:* Re: Web UI
>
>
>
> Hey Oliver,
>
>
>
> IIRC there's no JSON endpoint for application web UI. They only exist for
> cluster master and worker.
>
>
>
> - Wonha
>
>
>
>
>
> On Wed, Sep 3, 2014 at 12:58 PM, Ruebenacker, Oliver A <
> Oliver.Ruebenacker@altisource.com> wrote:
>
>
>
> Hello,
>
>
>
> Thanks for the help! But I tried starting with “–master local[4]” and
> when I load http://localhost:4040/json I just get forwarded to
> http://localhost:4040/stages/, and it’s all human-readable HTML, no JSON.
>
>
>
> Best,
>
> Oliver
>
>
>
>
>
> *From:* Wonha Ryu [mailto:wonha.ryu@gmail.com]
> *Sent:* Wednesday, September 03, 2014 3:36 PM
> *To:* Ruebenacker, Oliver A
> *Cc:* user@spark.apache.org
> *Subject:* Re: Web UI
>
>
>
> Hi Oliver,
>
>
>
> Spark standalone master and worker support '/json' endpoint in web UI,
> which returns some of the information in JSON format.
>
> I wasn't able to find relevant documentation, though.
>
>
>
> - Wonha
>
>
>
> On Wed, Sep 3, 2014 at 12:12 PM, Ruebenacker, Oliver A <
> Oliver.Ruebenacker@altisource.com> wrote:
>
>
>
> Hello,
>
>
>
> What is included in the Spark web UI? What are the available URLs? Can
> the information be obtained in a machine-readable way (e.g. JSON, XML, etc)?
>
>
>
> Thanks!
>
>
>
> Best,
>
> Oliver
>
>
>
> Oliver Ruebenacker | Solutions Architect
>
>
>
> Altisource™
>
> 290 Congress St, 7th Floor | Boston, Massachusetts 02210
>
> P: (617) 728-5582 | ext: 275585
>
> Oliver.Ruebenacker@Altisource.com | www.Altisource.com
>
>
>
>
> ***********************************************************************************************************************
>
> This email message and any attachments are intended solely for the use of
> the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system.
>
> This message and any attachments may contain information that is
> confidential, privileged or exempt from disclosure. Delivery of this
> message to any person other than the intended recipient is not intended to
> waive any right or privilege. Message transmission is not guaranteed to be
> secure or free of software viruses.
>
> ***********************************************************************************************************************
>
>
>
>
> ***********************************************************************************************************************
>
> This email message and any attachments are intended solely for the use of
> the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system.
>
> This message and any attachments may contain information that is
> confidential, privileged or exempt from disclosure. Delivery of this
> message to any person other than the intended recipient is not intended to
> waive any right or privilege. Message transmission is not guaranteed to be
> secure or free of software viruses.
>
> ***********************************************************************************************************************
>
>
>
>
> ***********************************************************************************************************************
>
> This email message and any attachments are intended solely for the use of
> the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system.
> This message and any attachments may contain information that is
> confidential, privileged or exempt from disclosure. Delivery of this
> message to any person other than the intended recipient is not intended to
> waive any right or privilege. Message transmission is not guaranteed to be
> secure or free of software viruses.
>
> ***********************************************************************************************************************
>
RE: Web UI
Posted by "Ruebenacker, Oliver A" <Ol...@altisource.com>.
Hello,
Interestingly, http://localhost:4040/metrics/json/ gives some numbers, but only a few which never seem to change during the application’s lifetime.
Either the web UI has some very strange limitations, or there are some URLs yet to be discovered that do something interesting.
Best,
Oliver
From: Wonha Ryu [mailto:wonha.ryu@gmail.com]
Sent: Wednesday, September 03, 2014 4:27 PM
To: Ruebenacker, Oliver A
Cc: user@spark.apache.org
Subject: Re: Web UI
Hey Oliver,
IIRC there's no JSON endpoint for application web UI. They only exist for cluster master and worker.
- Wonha
On Wed, Sep 3, 2014 at 12:58 PM, Ruebenacker, Oliver A <Ol...@altisource.com>> wrote:
Hello,
Thanks for the help! But I tried starting with “–master local[4]” and when I load http://localhost:4040/json I just get forwarded to http://localhost:4040/stages/, and it’s all human-readable HTML, no JSON.
Best,
Oliver
From: Wonha Ryu [mailto:wonha.ryu@gmail.com<ma...@gmail.com>]
Sent: Wednesday, September 03, 2014 3:36 PM
To: Ruebenacker, Oliver A
Cc: user@spark.apache.org<ma...@spark.apache.org>
Subject: Re: Web UI
Hi Oliver,
Spark standalone master and worker support '/json' endpoint in web UI, which returns some of the information in JSON format.
I wasn't able to find relevant documentation, though.
- Wonha
On Wed, Sep 3, 2014 at 12:12 PM, Ruebenacker, Oliver A <Ol...@altisource.com>> wrote:
Hello,
What is included in the Spark web UI? What are the available URLs? Can the information be obtained in a machine-readable way (e.g. JSON, XML, etc)?
Thanks!
Best,
Oliver
Oliver Ruebenacker | Solutions Architect
Altisource™
290 Congress St, 7th Floor | Boston, Massachusetts 02210
P: (617) 728-5582<tel:%28617%29%20728-5582> | ext: 275585
Oliver.Ruebenacker@Altisource.com<ma...@Altisource.com> | www.Altisource.com<http://www.Altisource.com>
***********************************************************************************************************************
This email message and any attachments are intended solely for the use of the addressee. If you are not the intended recipient, you are prohibited from reading, disclosing, reproducing, distributing, disseminating or otherwise using this transmission. If you have received this message in error, please promptly notify the sender by reply email and immediately delete this message from your system.
This message and any attachments may contain information that is confidential, privileged or exempt from disclosure. Delivery of this message to any person other than the intended recipient is not intended to waive any right or privilege. Message transmission is not guaranteed to be secure or free of software viruses.
***********************************************************************************************************************
***********************************************************************************************************************
This email message and any attachments are intended solely for the use of the addressee. If you are not the intended recipient, you are prohibited from reading, disclosing, reproducing, distributing, disseminating or otherwise using this transmission. If you have received this message in error, please promptly notify the sender by reply email and immediately delete this message from your system.
This message and any attachments may contain information that is confidential, privileged or exempt from disclosure. Delivery of this message to any person other than the intended recipient is not intended to waive any right or privilege. Message transmission is not guaranteed to be secure or free of software viruses.
***********************************************************************************************************************
***********************************************************************************************************************
This email message and any attachments are intended solely for the use of the addressee. If you are not the intended recipient, you are prohibited from reading, disclosing, reproducing, distributing, disseminating or otherwise using this transmission. If you have received this message in error, please promptly notify the sender by reply email and immediately delete this message from your system. This message and any attachments may contain information that is confidential, privileged or exempt from disclosure. Delivery of this message to any person other than the intended recipient is not intended to waive any right or privilege. Message transmission is not guaranteed to be secure or free of software viruses.
***********************************************************************************************************************
Re: Web UI
Posted by Wonha Ryu <wo...@gmail.com>.
Hey Oliver,
IIRC there's no JSON endpoint for application web UI. They only exist for
cluster master and worker.
- Wonha
On Wed, Sep 3, 2014 at 12:58 PM, Ruebenacker, Oliver A <
Oliver.Ruebenacker@altisource.com> wrote:
>
>
> Hello,
>
>
>
> Thanks for the help! But I tried starting with “–master local[4]” and
> when I load http://localhost:4040/json I just get forwarded to
> http://localhost:4040/stages/, and it’s all human-readable HTML, no JSON.
>
>
>
> Best,
>
> Oliver
>
>
>
>
>
> *From:* Wonha Ryu [mailto:wonha.ryu@gmail.com]
> *Sent:* Wednesday, September 03, 2014 3:36 PM
> *To:* Ruebenacker, Oliver A
> *Cc:* user@spark.apache.org
> *Subject:* Re: Web UI
>
>
>
> Hi Oliver,
>
>
>
> Spark standalone master and worker support '/json' endpoint in web UI,
> which returns some of the information in JSON format.
>
> I wasn't able to find relevant documentation, though.
>
>
>
> - Wonha
>
>
>
> On Wed, Sep 3, 2014 at 12:12 PM, Ruebenacker, Oliver A <
> Oliver.Ruebenacker@altisource.com> wrote:
>
>
>
> Hello,
>
>
>
> What is included in the Spark web UI? What are the available URLs? Can
> the information be obtained in a machine-readable way (e.g. JSON, XML, etc)?
>
>
>
> Thanks!
>
>
>
> Best,
>
> Oliver
>
>
>
> Oliver Ruebenacker | Solutions Architect
>
>
>
> Altisource™
>
> 290 Congress St, 7th Floor | Boston, Massachusetts 02210
>
> P: (617) 728-5582 | ext: 275585
>
> Oliver.Ruebenacker@Altisource.com | www.Altisource.com
>
>
>
>
> ***********************************************************************************************************************
>
> This email message and any attachments are intended solely for the use of
> the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system.
>
> This message and any attachments may contain information that is
> confidential, privileged or exempt from disclosure. Delivery of this
> message to any person other than the intended recipient is not intended to
> waive any right or privilege. Message transmission is not guaranteed to be
> secure or free of software viruses.
>
> ***********************************************************************************************************************
>
>
>
>
> ***********************************************************************************************************************
>
> This email message and any attachments are intended solely for the use of
> the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system.
> This message and any attachments may contain information that is
> confidential, privileged or exempt from disclosure. Delivery of this
> message to any person other than the intended recipient is not intended to
> waive any right or privilege. Message transmission is not guaranteed to be
> secure or free of software viruses.
>
> ***********************************************************************************************************************
>
RE: Web UI
Posted by "Ruebenacker, Oliver A" <Ol...@altisource.com>.
Hello,
Thanks for the help! But I tried starting with “–master local[4]” and when I load http://localhost:4040/json I just get forwarded to http://localhost:4040/stages/, and it’s all human-readable HTML, no JSON.
Best,
Oliver
From: Wonha Ryu [mailto:wonha.ryu@gmail.com]
Sent: Wednesday, September 03, 2014 3:36 PM
To: Ruebenacker, Oliver A
Cc: user@spark.apache.org
Subject: Re: Web UI
Hi Oliver,
Spark standalone master and worker support '/json' endpoint in web UI, which returns some of the information in JSON format.
I wasn't able to find relevant documentation, though.
- Wonha
On Wed, Sep 3, 2014 at 12:12 PM, Ruebenacker, Oliver A <Ol...@altisource.com>> wrote:
Hello,
What is included in the Spark web UI? What are the available URLs? Can the information be obtained in a machine-readable way (e.g. JSON, XML, etc)?
Thanks!
Best,
Oliver
Oliver Ruebenacker | Solutions Architect
Altisource™
290 Congress St, 7th Floor | Boston, Massachusetts 02210
P: (617) 728-5582<tel:%28617%29%20728-5582> | ext: 275585
Oliver.Ruebenacker@Altisource.com<ma...@Altisource.com> | www.Altisource.com<http://www.Altisource.com>
***********************************************************************************************************************
This email message and any attachments are intended solely for the use of the addressee. If you are not the intended recipient, you are prohibited from reading, disclosing, reproducing, distributing, disseminating or otherwise using this transmission. If you have received this message in error, please promptly notify the sender by reply email and immediately delete this message from your system.
This message and any attachments may contain information that is confidential, privileged or exempt from disclosure. Delivery of this message to any person other than the intended recipient is not intended to waive any right or privilege. Message transmission is not guaranteed to be secure or free of software viruses.
***********************************************************************************************************************
***********************************************************************************************************************
This email message and any attachments are intended solely for the use of the addressee. If you are not the intended recipient, you are prohibited from reading, disclosing, reproducing, distributing, disseminating or otherwise using this transmission. If you have received this message in error, please promptly notify the sender by reply email and immediately delete this message from your system. This message and any attachments may contain information that is confidential, privileged or exempt from disclosure. Delivery of this message to any person other than the intended recipient is not intended to waive any right or privilege. Message transmission is not guaranteed to be secure or free of software viruses.
***********************************************************************************************************************
Re: Web UI
Posted by Wonha Ryu <wo...@gmail.com>.
Hi Oliver,
Spark standalone master and worker support '/json' endpoint in web UI,
which returns some of the information in JSON format.
I wasn't able to find relevant documentation, though.
- Wonha
On Wed, Sep 3, 2014 at 12:12 PM, Ruebenacker, Oliver A <
Oliver.Ruebenacker@altisource.com> wrote:
>
>
> Hello,
>
>
>
> What is included in the Spark web UI? What are the available URLs? Can
> the information be obtained in a machine-readable way (e.g. JSON, XML, etc)?
>
>
>
> Thanks!
>
>
>
> Best,
>
> Oliver
>
>
>
> Oliver Ruebenacker | Solutions Architect
>
>
>
> Altisource™
>
> 290 Congress St, 7th Floor | Boston, Massachusetts 02210
>
> P: (617) 728-5582 | ext: 275585
>
> Oliver.Ruebenacker@Altisource.com | www.Altisource.com
>
>
>
>
> ***********************************************************************************************************************
>
> This email message and any attachments are intended solely for the use of
> the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system.
> This message and any attachments may contain information that is
> confidential, privileged or exempt from disclosure. Delivery of this
> message to any person other than the intended recipient is not intended to
> waive any right or privilege. Message transmission is not guaranteed to be
> secure or free of software viruses.
>
> ***********************************************************************************************************************
>