You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Philip Lee <ph...@gmail.com> on 2016/01/25 17:36:56 UTC
a question about web ui log
Hello, a questino about web UI log.
I could see web interface log after forwarding the port on my cluster to
my local and click completed application, but when I clicked "application
detail UI"
[image: Inline image 1]
It happened to me. I do not know why. I also checked the specific log
folder. It has a log file in it. Actually, that's why I could click the
completed application link, right?
So is it okay for me to copy the log file in my cluster to my local machine.
And after turning on spark Job Manger on my local by myself, I could see
application deatils UI in my local machine?
Best,
Phil
RE: a question about web ui log
Posted by Mohammed Guller <mo...@glassbeam.com>.
If the application history is turned on, it should work, even through ssh tunnel. Can you elaborate on what you mean by “it does not work?”
Also, are you able to see the application web UI while an application is executing a job?
Mohammed
Author: Big Data Analytics with Spark<http://www.amazon.com/Big-Data-Analytics-Spark-Practitioners/dp/1484209656/>
From: Philip Lee [mailto:philjjoon@gmail.com]
Sent: Tuesday, January 26, 2016 5:12 AM
To: Mohammed Guller
Cc: user@spark.apache.org
Subject: Re: a question about web ui log
Yes, I tried it, but it simply does not work.
so, my concern is to use "ssh tunnel" to forward a port of cluster to localhost port.
But in Spark UI, there are two ports which I should forward using "ssh tunnel".
Considering a default port, 8080 is web-ui port to come into web-ui, and 4040 is web-monitoring port to see time execution like DAG in application details UI, as you probably know.
But after finishing a job, I can see the list of a job on the web-ui on 8080, but when I click "application details UI" on port 4040 to see time excution, it does not work.
Any suggestion? I really need to see execution of DAG.
Best,
Phil
On Tue, Jan 26, 2016 at 12:04 AM, Mohammed Guller <mo...@glassbeam.com>> wrote:
I am not sure whether you can copy the log files from Spark workers to your local machine and view it from the Web UI. In fact, if you are able to copy the log files locally, you can just view them directly in any text editor.
I suspect what you really want to see is the application history. Here is the relevant information from Spark’s monitoring page (http://spark.apache.org/docs/latest/monitoring.html)
To view the web UI after the fact, set spark.eventLog.enabled to true before starting the application. This configures Spark to log Spark events that encode the information displayed in the UI to persisted storage.
Mohammed
Author: Big Data Analytics with Spark<http://www.amazon.com/Big-Data-Analytics-Spark-Practitioners/dp/1484209656/>
From: Philip Lee [mailto:philjjoon@gmail.com<ma...@gmail.com>]
Sent: Monday, January 25, 2016 9:51 AM
To: user@spark.apache.org<ma...@spark.apache.org>
Subject: Re: a question about web ui log
As I mentioned before, I am tryint to see the spark log on a cluster via ssh-tunnel
1) The error on application details UI is probably from monitoring porting 4044. Web UI port is 8088, right? so how could I see job web ui view and application details UI view in the web ui on my local machine?
2) still wondering how to see the log after copyting log file to my local.
The error was metioned in previous mail.
Thanks,
Phil
On Mon, Jan 25, 2016 at 5:36 PM, Philip Lee <ph...@gmail.com>> wrote:
Hello, a questino about web UI log.
I could see web interface log after forwarding the port on my cluster to my local and click completed application, but when I clicked "application detail UI"
[Inline image 1]
It happened to me. I do not know why. I also checked the specific log folder. It has a log file in it. Actually, that's why I could click the completed application link, right?
So is it okay for me to copy the log file in my cluster to my local machine.
And after turning on spark Job Manger on my local by myself, I could see application deatils UI in my local machine?
Best,
Phil
Re: a question about web ui log
Posted by Philip Lee <ph...@gmail.com>.
Yes, I tried it, but it simply does not work.
so, my concern is *to use "ssh tunnel" to forward a port of cluster to
localhost port. *
But in Spark UI, there are two ports which I should forward using "*ssh
tunnel*".
Considering a default port, 8080 is web-ui port to come into web-ui, and
4040 is web-monitoring port to see time execution like DAG in application
details UI, as you probably know.
But after finishing a job, I can see the list of a job on the web-ui on
8080, but when I click "application details UI" on port 4040 to see time
excution, it does not work.
Any suggestion? I really need to see execution of DAG.
Best,
Phil
On Tue, Jan 26, 2016 at 12:04 AM, Mohammed Guller <mo...@glassbeam.com>
wrote:
> I am not sure whether you can copy the log files from Spark workers to
> your local machine and view it from the Web UI. In fact, if you are able to
> copy the log files locally, you can just view them directly in any text
> editor.
>
>
>
> I suspect what you really want to see is the application history. Here is
> the relevant information from Spark’s monitoring page (
> http://spark.apache.org/docs/latest/monitoring.html)
>
>
>
> To view the web UI after the fact, set spark.eventLog.enabled to true
> before starting the application. This configures Spark to log Spark events
> that encode the information displayed in the UI to persisted storage.
>
>
>
> Mohammed
>
> Author: Big Data Analytics with Spark
> <http://www.amazon.com/Big-Data-Analytics-Spark-Practitioners/dp/1484209656/>
>
>
>
> *From:* Philip Lee [mailto:philjjoon@gmail.com]
> *Sent:* Monday, January 25, 2016 9:51 AM
> *To:* user@spark.apache.org
> *Subject:* Re: a question about web ui log
>
>
>
> As I mentioned before, I am tryint to see the spark log on a cluster via
> ssh-tunnel
>
>
>
> 1) The error on application details UI is probably from monitoring porting
> 4044. Web UI port is 8088, right? so how could I see job web ui view and
> application details UI view in the web ui on my local machine?
>
>
>
> 2) still wondering how to see the log after copyting log file to my local.
>
>
>
> The error was metioned in previous mail.
>
>
>
> Thanks,
>
> Phil
>
>
>
>
>
>
>
> On Mon, Jan 25, 2016 at 5:36 PM, Philip Lee <ph...@gmail.com> wrote:
>
> Hello, a questino about web UI log.
>
>
>
> I could see web interface log after forwarding the port on my cluster to
> my local and click completed application, but when I clicked "application
> detail UI"
>
>
>
> [image: Inline image 1]
>
>
>
> It happened to me. I do not know why. I also checked the specific log
> folder. It has a log file in it. Actually, that's why I could click the
> completed application link, right?
>
>
>
> So is it okay for me to copy the log file in my cluster to my local
> machine.
>
> And after turning on spark Job Manger on my local by myself, I could see
> application deatils UI in my local machine?
>
>
>
> Best,
>
> Phil
>
>
>
RE: a question about web ui log
Posted by Mohammed Guller <mo...@glassbeam.com>.
I am not sure whether you can copy the log files from Spark workers to your local machine and view it from the Web UI. In fact, if you are able to copy the log files locally, you can just view them directly in any text editor.
I suspect what you really want to see is the application history. Here is the relevant information from Spark’s monitoring page (http://spark.apache.org/docs/latest/monitoring.html)
To view the web UI after the fact, set spark.eventLog.enabled to true before starting the application. This configures Spark to log Spark events that encode the information displayed in the UI to persisted storage.
Mohammed
Author: Big Data Analytics with Spark<http://www.amazon.com/Big-Data-Analytics-Spark-Practitioners/dp/1484209656/>
From: Philip Lee [mailto:philjjoon@gmail.com]
Sent: Monday, January 25, 2016 9:51 AM
To: user@spark.apache.org
Subject: Re: a question about web ui log
As I mentioned before, I am tryint to see the spark log on a cluster via ssh-tunnel
1) The error on application details UI is probably from monitoring porting 4044. Web UI port is 8088, right? so how could I see job web ui view and application details UI view in the web ui on my local machine?
2) still wondering how to see the log after copyting log file to my local.
The error was metioned in previous mail.
Thanks,
Phil
On Mon, Jan 25, 2016 at 5:36 PM, Philip Lee <ph...@gmail.com>> wrote:
Hello, a questino about web UI log.
I could see web interface log after forwarding the port on my cluster to my local and click completed application, but when I clicked "application detail UI"
[Inline image 1]
It happened to me. I do not know why. I also checked the specific log folder. It has a log file in it. Actually, that's why I could click the completed application link, right?
So is it okay for me to copy the log file in my cluster to my local machine.
And after turning on spark Job Manger on my local by myself, I could see application deatils UI in my local machine?
Best,
Phil
Re: a question about web ui log
Posted by Philip Lee <ph...@gmail.com>.
As I mentioned before, I am tryint to see the spark log on a cluster via
ssh-tunnel
1) The error on application details UI is probably from monitoring porting
4044. Web UI port is 8088, right? so how could I see job web ui view and
application details UI view in the web ui on my local machine?
2) still wondering how to see the log after copyting log file to my local.
The error was metioned in previous mail.
Thanks,
Phil
On Mon, Jan 25, 2016 at 5:36 PM, Philip Lee <ph...@gmail.com> wrote:
> Hello, a questino about web UI log.
>
> I could see web interface log after forwarding the port on my cluster to
> my local and click completed application, but when I clicked "application
> detail UI"
>
> [image: Inline image 1]
>
> It happened to me. I do not know why. I also checked the specific log
> folder. It has a log file in it. Actually, that's why I could click the
> completed application link, right?
>
> So is it okay for me to copy the log file in my cluster to my local
> machine.
> And after turning on spark Job Manger on my local by myself, I could see
> application deatils UI in my local machine?
>
> Best,
> Phil
>