You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Andrew Davidson (JIRA)" <ji...@apache.org> on 2016/07/01 16:26:11 UTC
[jira] [Commented] (SPARK-15829) spark master webpage links to
application UI broke when running in cluster mode
[ https://issues.apache.org/jira/browse/SPARK-15829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359215#comment-15359215 ]
Andrew Davidson commented on SPARK-15829:
-----------------------------------------
Hi Sean
I am not sure how to check the value of 'SPARK_MASTER_HOST'. I looked at the documentation pags http://spark.apache.org/docs/latest/configuration.html and http://spark.apache.org/docs/latest/ec2-scripts.html. They do not mention SPARK_MASTER_HOST
when I submit my jobs I use
MASTER_URL=spark://ec2-54-215-230-73.us-west-1.compute.amazonaws.com:6066
I use the stand alone cluster manager
I think the problem may be that the web UI assume the driver is always running on the master machine. I assume the cluster manager decides which worker the driver will run on. Is there a way for the web UI to discover where the driver is running?
On my master
[ec2-user@ip-172-31-22-140 conf]$ pwd
/root/spark/conf
[ec2-user@ip-172-31-22-140 conf]$ cat slaves
ec2-54-193-94-207.us-west-1.compute.amazonaws.com
ec2-54-67-13-246.us-west-1.compute.amazonaws.com
ec2-54-67-48-49.us-west-1.compute.amazonaws.com
ec2-54-193-104-169.us-west-1.compute.amazonaws.com
[ec2-user@ip-172-31-22-140 conf]$
[ec2-user@ip-172-31-22-140 conf]$ grep SPARK_MASTER_HOST *
[ec2-user@ip-172-31-22-140 conf]$ pwd
/root/spark/conf
[ec2-user@ip-172-31-22-140 conf]$ grep SPARK_MASTER_HOST *
[ec2-user@ip-172-31-22-140 conf]$
ec2-user@ip-172-31-22-140 sbin]$ pwd
/root/spark/sbin
[ec2-user@ip-172-31-22-140 sbin]$ grep SPARK_MASTER_HOST *
[ec2-user@ip-172-31-22-140 bin]$ !grep
grep SPARK_MASTER_HOST *
[ec2-user@ip-172-31-22-140 bin]$
Thanks for looking into this
Andy
> spark master webpage links to application UI broke when running in cluster mode
> -------------------------------------------------------------------------------
>
> Key: SPARK-15829
> URL: https://issues.apache.org/jira/browse/SPARK-15829
> Project: Spark
> Issue Type: Bug
> Components: EC2
> Affects Versions: 1.6.1
> Environment: AWS ec2 cluster
> Reporter: Andrew Davidson
> Priority: Critical
>
> Hi
> I created a cluster using the spark-1.6.1-bin-hadoop2.6/ec2/spark-ec2
> I use the stand alone cluster manager. I have a streaming app running in cluster mode. I notice the master webpage links to the application UI page are incorrect
> It does not look like jira will let my upload images. I'll try and describe the web pages and the bug
> My master is running on
> http://ec2-54-215-230-73.us-west-1.compute.amazonaws.com:8080/
> It has a section marked "applications". If I click on one of the running application ids I am taken to a page showing "Executor Summary". This page has a link to teh 'application detail UI' the url is http://ec2-54-215-230-73.us-west-1.compute.amazonaws.com:4041/
> Notice it things the application UI is running on the cluster master.
> It is actually running on the same machine as the driver on port 4041. I was able to reverse engine the url by noticing the private ip address is part of the worker id . For example worker-20160322041632-172.31.23.201-34909
> next I went on the aws ec2 console to find the public DNS name for this machine
> http://ec2-54-193-104-169.us-west-1.compute.amazonaws.com:4041/streaming/
> Kind regards
> Andy
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org