You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by satish saley <sa...@gmail.com> on 2016/06/07 18:11:38 UTC
Environment tab meaning
Hi,
In spark history server, we see environment tab. Is it show environment of
Driver or Executor or both?
- Jobs
<http://localhost:18080/history/application_1465230614593_0025/jobs/>
- Stages
<http://localhost:18080/history/application_1465230614593_0025/stages/>
- Storage
<http://localhost:18080/history/application_1465230614593_0025/storage/>
- Environment
<http://localhost:18080/history/application_1465230614593_0025/environment/>
- Executors
<http://localhost:18080/history/application_1465230614593_0025/executors/>
-
Re: Environment tab meaning
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi,
I'm not surprised to see Hadoop jars on the driver (yet I couldn't
explain exactly why they need to be there). I can't find a way now to
display the classpath for executors.
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
On Tue, Jun 7, 2016 at 10:05 PM, satish saley <sa...@gmail.com> wrote:
> Thank you Jacek.
> In case of YARN, I see that hadoop jars are present in system classpath for
> Driver. Will it be the same for all executors?
>
> On Tue, Jun 7, 2016 at 11:22 AM, Jacek Laskowski <ja...@japila.pl> wrote:
>>
>> Ouch, I made a mistake :( Sorry.
>>
>> You've asked about spark **history** server. It's pretty much the same.
>>
>> HistoryServer is a web interface for completed and running (aka
>> incomplete) Spark applications. It uses EventLoggingListener to
>> collect events as JSON using org.apache.spark.util.JsonProtocol
>> object.
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> ----
>> https://medium.com/@jaceklaskowski/
>> Mastering Apache Spark http://bit.ly/mastering-apache-spark
>> Follow me at https://twitter.com/jaceklaskowski
>>
>>
>> On Tue, Jun 7, 2016 at 8:18 PM, Jacek Laskowski <ja...@japila.pl> wrote:
>> > Hi,
>> >
>> > It is the driver - see the port. Is this 4040 or similar? It's started
>> > when SparkContext starts and is controlled by spark.ui.enabled.
>> >
>> > spark.ui.enabled (default: true) = controls whether the web UI is
>> > started or not.
>> >
>> > It's through JobProgressListener which is the SparkListener for web UI
>> > that the console knows what happens under the covers (and can
>> > calculate the stats).
>> >
>> > BTW, spark.ui.port (default: 4040) controls the port Web UI binds to.
>> >
>> > Pozdrawiam,
>> > Jacek Laskowski
>> > ----
>> > https://medium.com/@jaceklaskowski/
>> > Mastering Apache Spark http://bit.ly/mastering-apache-spark
>> > Follow me at https://twitter.com/jaceklaskowski
>> >
>> >
>> > On Tue, Jun 7, 2016 at 8:11 PM, satish saley <sa...@gmail.com>
>> > wrote:
>> >> Hi,
>> >> In spark history server, we see environment tab. Is it show environment
>> >> of
>> >> Driver or Executor or both?
>> >>
>> >> Jobs
>> >> Stages
>> >> Storage
>> >> Environment
>> >> Executors
>> >>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Environment tab meaning
Posted by satish saley <sa...@gmail.com>.
Thank you Jacek.
In case of YARN, I see that hadoop jars are present in system classpath for
Driver. Will it be the same for all executors?
On Tue, Jun 7, 2016 at 11:22 AM, Jacek Laskowski <ja...@japila.pl> wrote:
> Ouch, I made a mistake :( Sorry.
>
> You've asked about spark **history** server. It's pretty much the same.
>
> HistoryServer is a web interface for completed and running (aka
> incomplete) Spark applications. It uses EventLoggingListener to
> collect events as JSON using org.apache.spark.util.JsonProtocol
> object.
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
>
> On Tue, Jun 7, 2016 at 8:18 PM, Jacek Laskowski <ja...@japila.pl> wrote:
> > Hi,
> >
> > It is the driver - see the port. Is this 4040 or similar? It's started
> > when SparkContext starts and is controlled by spark.ui.enabled.
> >
> > spark.ui.enabled (default: true) = controls whether the web UI is
> > started or not.
> >
> > It's through JobProgressListener which is the SparkListener for web UI
> > that the console knows what happens under the covers (and can
> > calculate the stats).
> >
> > BTW, spark.ui.port (default: 4040) controls the port Web UI binds to.
> >
> > Pozdrawiam,
> > Jacek Laskowski
> > ----
> > https://medium.com/@jaceklaskowski/
> > Mastering Apache Spark http://bit.ly/mastering-apache-spark
> > Follow me at https://twitter.com/jaceklaskowski
> >
> >
> > On Tue, Jun 7, 2016 at 8:11 PM, satish saley <sa...@gmail.com>
> wrote:
> >> Hi,
> >> In spark history server, we see environment tab. Is it show environment
> of
> >> Driver or Executor or both?
> >>
> >> Jobs
> >> Stages
> >> Storage
> >> Environment
> >> Executors
> >>
>
Re: Environment tab meaning
Posted by Jacek Laskowski <ja...@japila.pl>.
Ouch, I made a mistake :( Sorry.
You've asked about spark **history** server. It's pretty much the same.
HistoryServer is a web interface for completed and running (aka
incomplete) Spark applications. It uses EventLoggingListener to
collect events as JSON using org.apache.spark.util.JsonProtocol
object.
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
On Tue, Jun 7, 2016 at 8:18 PM, Jacek Laskowski <ja...@japila.pl> wrote:
> Hi,
>
> It is the driver - see the port. Is this 4040 or similar? It's started
> when SparkContext starts and is controlled by spark.ui.enabled.
>
> spark.ui.enabled (default: true) = controls whether the web UI is
> started or not.
>
> It's through JobProgressListener which is the SparkListener for web UI
> that the console knows what happens under the covers (and can
> calculate the stats).
>
> BTW, spark.ui.port (default: 4040) controls the port Web UI binds to.
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
>
> On Tue, Jun 7, 2016 at 8:11 PM, satish saley <sa...@gmail.com> wrote:
>> Hi,
>> In spark history server, we see environment tab. Is it show environment of
>> Driver or Executor or both?
>>
>> Jobs
>> Stages
>> Storage
>> Environment
>> Executors
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Environment tab meaning
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi,
It is the driver - see the port. Is this 4040 or similar? It's started
when SparkContext starts and is controlled by spark.ui.enabled.
spark.ui.enabled (default: true) = controls whether the web UI is
started or not.
It's through JobProgressListener which is the SparkListener for web UI
that the console knows what happens under the covers (and can
calculate the stats).
BTW, spark.ui.port (default: 4040) controls the port Web UI binds to.
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
On Tue, Jun 7, 2016 at 8:11 PM, satish saley <sa...@gmail.com> wrote:
> Hi,
> In spark history server, we see environment tab. Is it show environment of
> Driver or Executor or both?
>
> Jobs
> Stages
> Storage
> Environment
> Executors
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org