You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jestin Ma <je...@gmail.com> on 2016/07/26 01:21:29 UTC

Spark Web UI port 4040 not working

Hello, when running spark jobs, I can access the master UI (port 8080 one)
no problem. However, I'm confused as to how to access the web UI to see
jobs/tasks/stages/etc.

I can access the master UI at http://<master-node>:8080. But port 4040
gives me a -connection cannot be reached-.

Is the web UI http://<master-node> with a port of 4040?

I'm running my Spark job on a cluster machine and submitting it to a master
node part of the cluster. I heard of ssh tunneling; is that relevant here?

Thank you!

Re: Spark Web UI port 4040 not working

Posted by Jacek Laskowski <ja...@japila.pl>.
Hi,

Go to 8080 and under Running Applications click the Application ID.
You're on the page with Application Detail UI just before Executor
Summary table. Use it to access the web UI.

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Tue, Jul 26, 2016 at 3:21 AM, Jestin Ma <je...@gmail.com> wrote:
> Hello, when running spark jobs, I can access the master UI (port 8080 one)
> no problem. However, I'm confused as to how to access the web UI to see
> jobs/tasks/stages/etc.
>
> I can access the master UI at http://<master-node>:8080. But port 4040 gives
> me a -connection cannot be reached-.
>
> Is the web UI http://<master-node> with a port of 4040?
>
> I'm running my Spark job on a cluster machine and submitting it to a master
> node part of the cluster. I heard of ssh tunneling; is that relevant here?
>
> Thank you!

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Re: Spark Web UI port 4040 not working

Posted by Jestin Ma <je...@gmail.com>.
Also, sorry for the repeated updates, I checked my master UI and it says no
drivers are running but 1 application is running.

On Tue, Jul 26, 2016 at 6:49 AM, Jestin Ma <je...@gmail.com>
wrote:

> I did netstat -apn | grep 4040 on machine 6, and I see
>
> tcp        0      0 :::4040                     :::*
>  LISTEN      30597/java
>
> What does this mean?
>
> On Tue, Jul 26, 2016 at 6:47 AM, Jestin Ma <je...@gmail.com>
> wrote:
>
>> I do not deploy using cluster mode and I don't use EC2.
>>
>> I just read that launching as client mode: "the driver is launched
>> directly within the spark-submit process which acts as a *client* to the
>> cluster."
>>
>> My current setup is that I have cluster machines 1, 2, 3, 4, 5, with 1
>> being the master.
>> I submit from another cluster machine 6 in client mode. So I'm taking
>> that the driver is launched in my machine 6.
>>
>> On Tue, Jul 26, 2016 at 6:38 AM, Jacek Laskowski <ja...@japila.pl> wrote:
>>
>>> Hi,
>>>
>>> Do you perhaps deploy using cluster mode? Is this EC2? You'd need to
>>> figure out where the driver runs and use the machine's IP.
>>>
>>> Pozdrawiam,
>>> Jacek Laskowski
>>> ----
>>> https://medium.com/@jaceklaskowski/
>>> Mastering Apache Spark http://bit.ly/mastering-apache-spark
>>> Follow me at https://twitter.com/jaceklaskowski
>>>
>>>
>>> On Tue, Jul 26, 2016 at 3:36 PM, Jestin Ma <je...@gmail.com>
>>> wrote:
>>> > I tried doing that on my master node.
>>> > I got nothing.
>>> > However, I grep'd port 8080 and I got the standalone UI.
>>> >
>>> > On Tue, Jul 26, 2016 at 12:39 AM, Chanh Le <gi...@gmail.com>
>>> wrote:
>>> >>
>>> >> You’re running in StandAlone Mode?
>>> >> Usually inside active task it will show the address of current job.
>>> >> or you can check in master node by using netstat -apn | grep 4040
>>> >>
>>> >>
>>> >>
>>> >> > On Jul 26, 2016, at 8:21 AM, Jestin Ma <je...@gmail.com>
>>> >> > wrote:
>>> >> >
>>> >> > Hello, when running spark jobs, I can access the master UI (port
>>> 8080
>>> >> > one) no problem. However, I'm confused as to how to access the web
>>> UI to see
>>> >> > jobs/tasks/stages/etc.
>>> >> >
>>> >> > I can access the master UI at http://<master-node>:8080. But port
>>> 4040
>>> >> > gives me a -connection cannot be reached-.
>>> >> >
>>> >> > Is the web UI http://<master-node> with a port of 4040?
>>> >> >
>>> >> > I'm running my Spark job on a cluster machine and submitting it to a
>>> >> > master node part of the cluster. I heard of ssh tunneling; is that
>>> relevant
>>> >> > here?
>>> >> >
>>> >> > Thank you!
>>> >>
>>> >
>>>
>>
>>
>

Re: Spark Web UI port 4040 not working

Posted by Marius Soutier <mp...@gmail.com>.
That's to be expected - the application UI is not started by the master, but by the driver. So the UI will run on the machine that submits the job.


> On 26.07.2016, at 15:49, Jestin Ma <je...@gmail.com> wrote:
> 
> I did netstat -apn | grep 4040 on machine 6, and I see
> 
> tcp        0      0 :::4040                     :::*                        LISTEN      30597/java
> 
> What does this mean?
> 
> On Tue, Jul 26, 2016 at 6:47 AM, Jestin Ma <jestinwith.an.e@gmail.com <ma...@gmail.com>> wrote:
> I do not deploy using cluster mode and I don't use EC2.
> 
> I just read that launching as client mode: "the driver is launched directly within the spark-submit process which acts as a client to the cluster."
> 
> My current setup is that I have cluster machines 1, 2, 3, 4, 5, with 1 being the master. 
> I submit from another cluster machine 6 in client mode. So I'm taking that the driver is launched in my machine 6.
> 
> On Tue, Jul 26, 2016 at 6:38 AM, Jacek Laskowski <jacek@japila.pl <ma...@japila.pl>> wrote:
> Hi,
> 
> Do you perhaps deploy using cluster mode? Is this EC2? You'd need to
> figure out where the driver runs and use the machine's IP.
> 
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/ <https://medium.com/@jaceklaskowski/>
> Mastering Apache Spark http://bit.ly/mastering-apache-spark <http://bit.ly/mastering-apache-spark>
> Follow me at https://twitter.com/jaceklaskowski <https://twitter.com/jaceklaskowski>
> 
> 
> On Tue, Jul 26, 2016 at 3:36 PM, Jestin Ma <jestinwith.an.e@gmail.com <ma...@gmail.com>> wrote:
> > I tried doing that on my master node.
> > I got nothing.
> > However, I grep'd port 8080 and I got the standalone UI.
> >
> > On Tue, Jul 26, 2016 at 12:39 AM, Chanh Le <giaosudau@gmail.com <ma...@gmail.com>> wrote:
> >>
> >> You’re running in StandAlone Mode?
> >> Usually inside active task it will show the address of current job.
> >> or you can check in master node by using netstat -apn | grep 4040
> >>
> >>
> >>
> >> > On Jul 26, 2016, at 8:21 AM, Jestin Ma <jestinwith.an.e@gmail.com <ma...@gmail.com>>
> >> > wrote:
> >> >
> >> > Hello, when running spark jobs, I can access the master UI (port 8080
> >> > one) no problem. However, I'm confused as to how to access the web UI to see
> >> > jobs/tasks/stages/etc.
> >> >
> >> > I can access the master UI at http://<master-node>:8080. But port 4040
> >> > gives me a -connection cannot be reached-.
> >> >
> >> > Is the web UI http://<master-node> with a port of 4040?
> >> >
> >> > I'm running my Spark job on a cluster machine and submitting it to a
> >> > master node part of the cluster. I heard of ssh tunneling; is that relevant
> >> > here?
> >> >
> >> > Thank you!
> >>
> >
> 
> 


Re: Spark Web UI port 4040 not working

Posted by Jestin Ma <je...@gmail.com>.
I did netstat -apn | grep 4040 on machine 6, and I see

tcp        0      0 :::4040                     :::*
 LISTEN      30597/java

What does this mean?

On Tue, Jul 26, 2016 at 6:47 AM, Jestin Ma <je...@gmail.com>
wrote:

> I do not deploy using cluster mode and I don't use EC2.
>
> I just read that launching as client mode: "the driver is launched
> directly within the spark-submit process which acts as a *client* to the
> cluster."
>
> My current setup is that I have cluster machines 1, 2, 3, 4, 5, with 1
> being the master.
> I submit from another cluster machine 6 in client mode. So I'm taking that
> the driver is launched in my machine 6.
>
> On Tue, Jul 26, 2016 at 6:38 AM, Jacek Laskowski <ja...@japila.pl> wrote:
>
>> Hi,
>>
>> Do you perhaps deploy using cluster mode? Is this EC2? You'd need to
>> figure out where the driver runs and use the machine's IP.
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> ----
>> https://medium.com/@jaceklaskowski/
>> Mastering Apache Spark http://bit.ly/mastering-apache-spark
>> Follow me at https://twitter.com/jaceklaskowski
>>
>>
>> On Tue, Jul 26, 2016 at 3:36 PM, Jestin Ma <je...@gmail.com>
>> wrote:
>> > I tried doing that on my master node.
>> > I got nothing.
>> > However, I grep'd port 8080 and I got the standalone UI.
>> >
>> > On Tue, Jul 26, 2016 at 12:39 AM, Chanh Le <gi...@gmail.com> wrote:
>> >>
>> >> You’re running in StandAlone Mode?
>> >> Usually inside active task it will show the address of current job.
>> >> or you can check in master node by using netstat -apn | grep 4040
>> >>
>> >>
>> >>
>> >> > On Jul 26, 2016, at 8:21 AM, Jestin Ma <je...@gmail.com>
>> >> > wrote:
>> >> >
>> >> > Hello, when running spark jobs, I can access the master UI (port 8080
>> >> > one) no problem. However, I'm confused as to how to access the web
>> UI to see
>> >> > jobs/tasks/stages/etc.
>> >> >
>> >> > I can access the master UI at http://<master-node>:8080. But port
>> 4040
>> >> > gives me a -connection cannot be reached-.
>> >> >
>> >> > Is the web UI http://<master-node> with a port of 4040?
>> >> >
>> >> > I'm running my Spark job on a cluster machine and submitting it to a
>> >> > master node part of the cluster. I heard of ssh tunneling; is that
>> relevant
>> >> > here?
>> >> >
>> >> > Thank you!
>> >>
>> >
>>
>
>

Re: Spark Web UI port 4040 not working

Posted by Jestin Ma <je...@gmail.com>.
I do not deploy using cluster mode and I don't use EC2.

I just read that launching as client mode: "the driver is launched directly
within the spark-submit process which acts as a *client* to the cluster."

My current setup is that I have cluster machines 1, 2, 3, 4, 5, with 1
being the master.
I submit from another cluster machine 6 in client mode. So I'm taking that
the driver is launched in my machine 6.

On Tue, Jul 26, 2016 at 6:38 AM, Jacek Laskowski <ja...@japila.pl> wrote:

> Hi,
>
> Do you perhaps deploy using cluster mode? Is this EC2? You'd need to
> figure out where the driver runs and use the machine's IP.
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
>
> On Tue, Jul 26, 2016 at 3:36 PM, Jestin Ma <je...@gmail.com>
> wrote:
> > I tried doing that on my master node.
> > I got nothing.
> > However, I grep'd port 8080 and I got the standalone UI.
> >
> > On Tue, Jul 26, 2016 at 12:39 AM, Chanh Le <gi...@gmail.com> wrote:
> >>
> >> You’re running in StandAlone Mode?
> >> Usually inside active task it will show the address of current job.
> >> or you can check in master node by using netstat -apn | grep 4040
> >>
> >>
> >>
> >> > On Jul 26, 2016, at 8:21 AM, Jestin Ma <je...@gmail.com>
> >> > wrote:
> >> >
> >> > Hello, when running spark jobs, I can access the master UI (port 8080
> >> > one) no problem. However, I'm confused as to how to access the web UI
> to see
> >> > jobs/tasks/stages/etc.
> >> >
> >> > I can access the master UI at http://<master-node>:8080. But port
> 4040
> >> > gives me a -connection cannot be reached-.
> >> >
> >> > Is the web UI http://<master-node> with a port of 4040?
> >> >
> >> > I'm running my Spark job on a cluster machine and submitting it to a
> >> > master node part of the cluster. I heard of ssh tunneling; is that
> relevant
> >> > here?
> >> >
> >> > Thank you!
> >>
> >
>

Re: Spark Web UI port 4040 not working

Posted by Andy Davidson <An...@SantaCruzIntegration.com>.
Yup in cluster mode you need to figure out what machine the driver is
running on. That is the machine the UI will run on

https://issues.apache.org/jira/browse/SPARK-15829

You may also have fire wall issues

Here are some notes I made about how to figure out what machine the driver
is running on when using cluster mode

Application UI Bug work around
If our application is running in cluster mode the Master Console UI will
incorrectly assume the Application UI¹s are running on the master machine.
The UI is actually running on the same machine as the driver
To find the URL to the apps running in cluster mode
1. find the worker id for the app you are interested in.
* on the master console UI go to the Running App¹s section
* you should see a column ŒWorker¹
* a worker is something like worker-20160322041632-172.31.23.201-34909
2. got to the AWS ec2 console
3. from the private ip find the public DNS name
* the private ip in our example is 172.31.23.201

From:  Jacek Laskowski <ja...@japila.pl>
Date:  Tuesday, July 26, 2016 at 6:38 AM
To:  Jestin Ma <je...@gmail.com>
Cc:  Chanh Le <gi...@gmail.com>, "user @spark" <us...@spark.apache.org>
Subject:  Re: Spark Web UI port 4040 not working

> Hi,
> 
> Do you perhaps deploy using cluster mode? Is this EC2? You'd need to
> figure out where the driver runs and use the machine's IP.
> 
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
> 
> 
> On Tue, Jul 26, 2016 at 3:36 PM, Jestin Ma <je...@gmail.com> wrote:
>>  I tried doing that on my master node.
>>  I got nothing.
>>  However, I grep'd port 8080 and I got the standalone UI.
>> 
>>  On Tue, Jul 26, 2016 at 12:39 AM, Chanh Le <gi...@gmail.com> wrote:
>>> 
>>>  You¹re running in StandAlone Mode?
>>>  Usually inside active task it will show the address of current job.
>>>  or you can check in master node by using netstat -apn | grep 4040
>>> 
>>> 
>>> 
>>>>  > On Jul 26, 2016, at 8:21 AM, Jestin Ma <je...@gmail.com>
>>>>  > wrote:
>>>>  >
>>>>  > Hello, when running spark jobs, I can access the master UI (port 8080
>>>>  > one) no problem. However, I'm confused as to how to access the web UI to
>>>> see
>>>>  > jobs/tasks/stages/etc.
>>>>  >
>>>>  > I can access the master UI at http://<master-node>:8080. But port 4040
>>>>  > gives me a -connection cannot be reached-.
>>>>  >
>>>>  > Is the web UI http://<master-node> with a port of 4040?
>>>>  >
>>>>  > I'm running my Spark job on a cluster machine and submitting it to a
>>>>  > master node part of the cluster. I heard of ssh tunneling; is that
>>>> relevant
>>>>  > here?
>>>>  >
>>>>  > Thank you!
>>> 
>> 
> 
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> 
> 



Re: Spark Web UI port 4040 not working

Posted by Jacek Laskowski <ja...@japila.pl>.
Hi,

Do you perhaps deploy using cluster mode? Is this EC2? You'd need to
figure out where the driver runs and use the machine's IP.

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Tue, Jul 26, 2016 at 3:36 PM, Jestin Ma <je...@gmail.com> wrote:
> I tried doing that on my master node.
> I got nothing.
> However, I grep'd port 8080 and I got the standalone UI.
>
> On Tue, Jul 26, 2016 at 12:39 AM, Chanh Le <gi...@gmail.com> wrote:
>>
>> You’re running in StandAlone Mode?
>> Usually inside active task it will show the address of current job.
>> or you can check in master node by using netstat -apn | grep 4040
>>
>>
>>
>> > On Jul 26, 2016, at 8:21 AM, Jestin Ma <je...@gmail.com>
>> > wrote:
>> >
>> > Hello, when running spark jobs, I can access the master UI (port 8080
>> > one) no problem. However, I'm confused as to how to access the web UI to see
>> > jobs/tasks/stages/etc.
>> >
>> > I can access the master UI at http://<master-node>:8080. But port 4040
>> > gives me a -connection cannot be reached-.
>> >
>> > Is the web UI http://<master-node> with a port of 4040?
>> >
>> > I'm running my Spark job on a cluster machine and submitting it to a
>> > master node part of the cluster. I heard of ssh tunneling; is that relevant
>> > here?
>> >
>> > Thank you!
>>
>

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Re: Spark Web UI port 4040 not working

Posted by Jestin Ma <je...@gmail.com>.
I tried doing that on my master node.
I got nothing.
However, I grep'd port 8080 and I got the standalone UI.

On Tue, Jul 26, 2016 at 12:39 AM, Chanh Le <gi...@gmail.com> wrote:

> You’re running in StandAlone Mode?
> Usually inside active task it will show the address of current job.
> or you can check in master node by using netstat -apn | grep 4040
>
>
>
> > On Jul 26, 2016, at 8:21 AM, Jestin Ma <je...@gmail.com>
> wrote:
> >
> > Hello, when running spark jobs, I can access the master UI (port 8080
> one) no problem. However, I'm confused as to how to access the web UI to
> see jobs/tasks/stages/etc.
> >
> > I can access the master UI at http://<master-node>:8080. But port 4040
> gives me a -connection cannot be reached-.
> >
> > Is the web UI http://<master-node> with a port of 4040?
> >
> > I'm running my Spark job on a cluster machine and submitting it to a
> master node part of the cluster. I heard of ssh tunneling; is that relevant
> here?
> >
> > Thank you!
>
>

Re: Spark Web UI port 4040 not working

Posted by Chanh Le <gi...@gmail.com>.
You’re running in StandAlone Mode?
Usually inside active task it will show the address of current job.
or you can check in master node by using netstat -apn | grep 4040



> On Jul 26, 2016, at 8:21 AM, Jestin Ma <je...@gmail.com> wrote:
> 
> Hello, when running spark jobs, I can access the master UI (port 8080 one) no problem. However, I'm confused as to how to access the web UI to see jobs/tasks/stages/etc.
> 
> I can access the master UI at http://<master-node>:8080. But port 4040 gives me a -connection cannot be reached-.
> 
> Is the web UI http://<master-node> with a port of 4040?
> 
> I'm running my Spark job on a cluster machine and submitting it to a master node part of the cluster. I heard of ssh tunneling; is that relevant here?
> 
> Thank you!


---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org