You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Faisal (JIRA)" <ji...@apache.org> on 2016/06/16 23:04:05 UTC
[jira] [Comment Edited] (SPARK-15999) Wrong/Missing information for
Spark UI/REST port
[ https://issues.apache.org/jira/browse/SPARK-15999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334895#comment-15334895 ]
Faisal edited comment on SPARK-15999 at 6/16/16 11:03 PM:
----------------------------------------------------------
Appreciate your prompt response but seems like it was never tested that rest service is exposed on 4040 or 4041 in yarn cluster mode.
My application is running on server URL, and i can see the list of executors and driver running for this streaming job
http://server-url:8088/proxy/application_1463099418950_9290/executors/
There is nothing running on 4040 or 4041 in all driver and executors. Error is obvious that nothing is running on that port and browser returns with error webpage
*This webpage is not available"*
However if i run it in standalone mode on my local machine it works fine on port 4040. I suspect someone written this documentation, did this experiment on local and felt sufficient for all modes. Later no one ever checked that?
I just checked the latest documentation and it is same
https://spark.apache.org/docs/latest/monitoring.html
I think i reported an issue as per guideline. I faced an issue on 1.5.0 i had to report effected version 1.5.0.
Besides, if i understand correctly, you are saying documentation is correct but application not working as stated in doc?
- If this is the case then we can update the component
- If this issue is already reported then we can mark it duplicate with duplicate jira#
But unable to understand the sense of closing the issue??
was (Author: faisal.siddiqui):
Appreciate your prompt response but seems like it was never tested that rest service is exposed on 4040 or 4041 in yarn cluster mode.
My application is running on server URL, and i can see the list of executors and driver running for this streaming job
http://server-url:8088/proxy/application_1463099418950_9290/executors/
There is nothing running on 4040 or 4041 in all driver and executors. Error is obvious that nothing is running on that port and browser returns with error webpage
*This webpage is not available"*
However if i run it in standalone mode on my local machine it works fine on port 4040. I suspect someone written this documentation did this experiment on local and felt sufficient for all modes and later no one ever checked that?
I just checked the latest documentation and it is same
https://spark.apache.org/docs/latest/monitoring.html
I think i reported an issue as per guideline. I faced an issue on 1.5.0 i had to report effected version 1.5.0.
Besides, if i understand correctly documentation is right but application not working as stated in doc? then we can update the component but unable to understand the sense of of closing the issue??
> Wrong/Missing information for Spark UI/REST port
> ------------------------------------------------
>
> Key: SPARK-15999
> URL: https://issues.apache.org/jira/browse/SPARK-15999
> Project: Spark
> Issue Type: Bug
> Components: Documentation, Streaming
> Affects Versions: 1.5.0
> Environment: CDH5.5.2, Spark 1.5.0
> Reporter: Faisal
> Priority: Minor
>
> *Spark Monitoring documentation*
> https://spark.apache.org/docs/1.5.0/monitoring.html
> {quote}
> You can access this interface by simply opening http://<driver-node>:4040 in a web browser. If multiple SparkContexts are running on the same host, they will bind to successive ports beginning with 4040 (4041, 4042, etc).
> {quote}
> This statement is very confusing and doesn't apply at all in spark streaming jobs(unless i am missing something)
> Same is the case with REST API calls.
> {quote}
> REST API
> In addition to viewing the metrics in the UI, they are also available as JSON. This gives developers an easy way to create new visualizations and monitoring tools for Spark. The JSON is available for both running applications, and in the history server. The endpoints are mounted at /api/v1. Eg., for the history server, they would typically be accessible at http://<server-url>:18080/api/v1, and for a running application, at http://localhost:4040/api/v1.
> {quote}
> I am running spark streaming job in CDH-5.5.2 Spark version 1.5.0
> and nowhere on driver node, executor node for running/live application i am able to call rest service.
> My spark streaming jobs running in yarn cluster mode
> --master yarn-cluster
> However for historyServer
> i am able to call REST service and can pull up json messages
> using the URL
> http://historyServer:18088/api/v1/applications
> {code}
> [ {
> "id" : "application_1463099418950_11465",
> "name" : "PySparkShell",
> "attempts" : [ {
> "startTime" : "2016-06-15T15:28:32.460GMT",
> "endTime" : "2016-06-15T19:01:39.100GMT",
> "sparkUser" : "abc",
> "completed" : true
> } ]
> }, {
> "id" : "application_1463099418950_11635",
> "name" : "DataProcessor-ETL.ETIME",
> "attempts" : [ {
> "attemptId" : "1",
> "startTime" : "2016-06-15T18:56:04.413GMT",
> "endTime" : "2016-06-15T18:58:00.022GMT",
> "sparkUser" : "abc",
> "completed" : true
> } ]
> },
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org