You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Srini Karri <sk...@gmail.com> on 2015/03/03 18:47:11 UTC
Spark Monitoring UI for Hadoop Yarn Cluster
Hi All,
I am having trouble finding data related to my requirement. Here is the
context, I have tried Standalone Spark Installation on Windows, I am able
to submit the logs, able to see the history of events. My question is, is
it possible to achieve the same monitoring UI experience with Yarn Cluster
like Viewing workers, running/completed job stages in the Web UI.
Currently, if we go to our Yarn Resource manager UI, we are able to see the
Spark Jobs and it's logs. But it is not as rich as Spark Standalone master
UI. Is this limitation for hadoop yarn cluster or is there any way we can
hook this Spark Standalone master to Yarn Cluster?
Any help is highly appreciated.
Regards,
Srini.
Re: Spark Monitoring UI for Hadoop Yarn Cluster
Posted by Srini Karri <sk...@gmail.com>.
Hi Marcelo,
I found the problem from
http://mail-archives.apache.org/mod_mbox/spark-user/201409.mbox/%3cCAL+LEBfzzjugOoB2iFFdz_=9TQsH=DaiKY=cvydfYDG3Ac5NxA@mail.gmail.com%3e
this link. The problem is the application I am running, is not generating
"APPLICATION_COMPLETE" file. If I add this file manually it is showing
application in the UI. So the problem is with application which is not
calling Stop method on the spark context.
Thank you and Todd for helping. Hopefully I will be able to apply these on
the actual cluster.
Regards,
Srini.
On Wed, Mar 4, 2015 at 10:20 AM, Srini Karri <sk...@gmail.com> wrote:
> Yes. I do see files, actually I missed copying the other settings:
>
> spark.master spark://
> skarri-lt05.redmond.corp.microsoft.com:7077
> spark.eventLog.enabled true
> spark.rdd.compress true
> spark.storage.memoryFraction 1
> spark.core.connection.ack.wait.timeout 6000
> spark.akka.frameSize 50
> spark.executor.extraClassPath
> D:\\Apache\\spark-1.2.1-bin-hadoop2\\spark-1.2.1-bin-hadoop2.4\\bin\\classes
> spark.eventLog.dir
> D:/Apache/spark-1.2.1-bin-hadoop2/spark-1.2.1-bin-hadoop2.4/bin/tmp/spark-events
> spark.history.fs.logDirectory
> D:/Apache/spark-1.2.1-bin-hadoop2/spark-1.2.1-bin-hadoop2.4/bin/tmp/spark-events
>
>
> On Wed, Mar 4, 2015 at 10:15 AM, Marcelo Vanzin <va...@cloudera.com>
> wrote:
>
>> On Wed, Mar 4, 2015 at 10:08 AM, Srini Karri <sk...@gmail.com>
>> wrote:
>> > spark.executor.extraClassPath
>> >
>> D:\\Apache\\spark-1.2.1-bin-hadoop2\\spark-1.2.1-bin-hadoop2.4\\bin\\classes
>> > spark.eventLog.dir
>> >
>> D:/Apache/spark-1.2.1-bin-hadoop2/spark-1.2.1-bin-hadoop2.4/bin/tmp/spark-events
>> > spark.history.fs.logDirectory
>> >
>> D:/Apache/spark-1.2.1-bin-hadoop2/spark-1.2.1-bin-hadoop2.4/bin/tmp/spark-events
>>
>> Do you see any files in that directory?
>>
>> spark.eventLog.dir won't do anything unless you also have
>> spark.eventLog.enabled=true somewhere. And these are application
>> configs, so make sure they're set when running your application (not
>> when starting the history server).
>>
>> --
>> Marcelo
>>
>
>
Re: Spark Monitoring UI for Hadoop Yarn Cluster
Posted by Srini Karri <sk...@gmail.com>.
Yes. I do see files, actually I missed copying the other settings:
spark.master spark://
skarri-lt05.redmond.corp.microsoft.com:7077
spark.eventLog.enabled true
spark.rdd.compress true
spark.storage.memoryFraction 1
spark.core.connection.ack.wait.timeout 6000
spark.akka.frameSize 50
spark.executor.extraClassPath
D:\\Apache\\spark-1.2.1-bin-hadoop2\\spark-1.2.1-bin-hadoop2.4\\bin\\classes
spark.eventLog.dir
D:/Apache/spark-1.2.1-bin-hadoop2/spark-1.2.1-bin-hadoop2.4/bin/tmp/spark-events
spark.history.fs.logDirectory
D:/Apache/spark-1.2.1-bin-hadoop2/spark-1.2.1-bin-hadoop2.4/bin/tmp/spark-events
On Wed, Mar 4, 2015 at 10:15 AM, Marcelo Vanzin <va...@cloudera.com> wrote:
> On Wed, Mar 4, 2015 at 10:08 AM, Srini Karri <sk...@gmail.com> wrote:
> > spark.executor.extraClassPath
> >
> D:\\Apache\\spark-1.2.1-bin-hadoop2\\spark-1.2.1-bin-hadoop2.4\\bin\\classes
> > spark.eventLog.dir
> >
> D:/Apache/spark-1.2.1-bin-hadoop2/spark-1.2.1-bin-hadoop2.4/bin/tmp/spark-events
> > spark.history.fs.logDirectory
> >
> D:/Apache/spark-1.2.1-bin-hadoop2/spark-1.2.1-bin-hadoop2.4/bin/tmp/spark-events
>
> Do you see any files in that directory?
>
> spark.eventLog.dir won't do anything unless you also have
> spark.eventLog.enabled=true somewhere. And these are application
> configs, so make sure they're set when running your application (not
> when starting the history server).
>
> --
> Marcelo
>
Re: Spark Monitoring UI for Hadoop Yarn Cluster
Posted by Marcelo Vanzin <va...@cloudera.com>.
On Wed, Mar 4, 2015 at 10:08 AM, Srini Karri <sk...@gmail.com> wrote:
> spark.executor.extraClassPath
> D:\\Apache\\spark-1.2.1-bin-hadoop2\\spark-1.2.1-bin-hadoop2.4\\bin\\classes
> spark.eventLog.dir
> D:/Apache/spark-1.2.1-bin-hadoop2/spark-1.2.1-bin-hadoop2.4/bin/tmp/spark-events
> spark.history.fs.logDirectory
> D:/Apache/spark-1.2.1-bin-hadoop2/spark-1.2.1-bin-hadoop2.4/bin/tmp/spark-events
Do you see any files in that directory?
spark.eventLog.dir won't do anything unless you also have
spark.eventLog.enabled=true somewhere. And these are application
configs, so make sure they're set when running your application (not
when starting the history server).
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark Monitoring UI for Hadoop Yarn Cluster
Posted by Srini Karri <sk...@gmail.com>.
Hi Todd and Marcelo,
Thanks for helping me. I was to able to lunch the history server on windows
with out any issues. One problem I am running into right now. I always get
the message no completed applications found in history server UI. But I was
able to browse through these applications from Spark Master. Do you have
any thoughts what could be problem? Following are my settings in spark conf
file:
spark.executor.extraClassPath
D:\\Apache\\spark-1.2.1-bin-hadoop2\\spark-1.2.1-bin-hadoop2.4\\bin\\classes
spark.eventLog.dir
D:/Apache/spark-1.2.1-bin-hadoop2/spark-1.2.1-bin-hadoop2.4/bin/tmp/spark-events
spark.history.fs.logDirectory
D:/Apache/spark-1.2.1-bin-hadoop2/spark-1.2.1-bin-hadoop2.4/bin/tmp/spark-events
Also I have attached Spark Master and Spark History server UI screen shots
for convenience. And all the logs are available and I granted directory
permissions to "Everyone with full control". Following is the console
output from History server:
D:\Apache\spark-1.2.1-bin-hadoop2\spark-1.2.1-bin-hadoop2.4\bin>spark-class.cmd
org.apache.spark.deploy.history.HistoryServer
Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties
15/03/04 08:59:42 INFO SecurityManager: Changing view acls to: skarri
15/03/04 08:59:42 INFO SecurityManager: Changing modify acls to: skarri
15/03/04 08:59:42 INFO SecurityManager: SecurityManager: authentication
disabled
; ui acls disabled; users with view permissions: Set(skarri); users with
modify
permissions: Set(skarri)
15/03/04 08:59:49 WARN NativeCodeLoader: Unable to load native-hadoop
library fo
r your platform... using builtin-java classes where applicable
15/03/04 08:59:56 INFO Utils: Successfully started service on port 18080.
15/03/04 08:59:56 INFO HistoryServer: Started HistoryServer at
http://skarri-lt0
5.redmond.corp.microsoft.com:18080
Regards,
Srini.
On Tue, Mar 3, 2015 at 11:41 AM, Marcelo Vanzin <va...@cloudera.com> wrote:
> Spark applications shown in the RM's UI should have an "Application
> Master" link when they're running. That takes you to the Spark UI for
> that application where you can see all the information you're looking
> for.
>
> If you're running a history server and add
> "spark.yarn.historyServer.address" to your config, that link will
> become a "History" link after the application is finished, and will
> take you to the history server to view the app's UI.
>
>
>
> On Tue, Mar 3, 2015 at 9:47 AM, Srini Karri <sk...@gmail.com> wrote:
> > Hi All,
> >
> > I am having trouble finding data related to my requirement. Here is the
> > context, I have tried Standalone Spark Installation on Windows, I am
> able to
> > submit the logs, able to see the history of events. My question is, is it
> > possible to achieve the same monitoring UI experience with Yarn Cluster
> like
> > Viewing workers, running/completed job stages in the Web UI. Currently,
> if
> > we go to our Yarn Resource manager UI, we are able to see the Spark Jobs
> and
> > it's logs. But it is not as rich as Spark Standalone master UI. Is this
> > limitation for hadoop yarn cluster or is there any way we can hook this
> > Spark Standalone master to Yarn Cluster?
> >
> > Any help is highly appreciated.
> >
> > Regards,
> > Srini.
>
>
>
> --
> Marcelo
>
Re: Spark Monitoring UI for Hadoop Yarn Cluster
Posted by Marcelo Vanzin <va...@cloudera.com>.
Spark applications shown in the RM's UI should have an "Application
Master" link when they're running. That takes you to the Spark UI for
that application where you can see all the information you're looking
for.
If you're running a history server and add
"spark.yarn.historyServer.address" to your config, that link will
become a "History" link after the application is finished, and will
take you to the history server to view the app's UI.
On Tue, Mar 3, 2015 at 9:47 AM, Srini Karri <sk...@gmail.com> wrote:
> Hi All,
>
> I am having trouble finding data related to my requirement. Here is the
> context, I have tried Standalone Spark Installation on Windows, I am able to
> submit the logs, able to see the history of events. My question is, is it
> possible to achieve the same monitoring UI experience with Yarn Cluster like
> Viewing workers, running/completed job stages in the Web UI. Currently, if
> we go to our Yarn Resource manager UI, we are able to see the Spark Jobs and
> it's logs. But it is not as rich as Spark Standalone master UI. Is this
> limitation for hadoop yarn cluster or is there any way we can hook this
> Spark Standalone master to Yarn Cluster?
>
> Any help is highly appreciated.
>
> Regards,
> Srini.
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark Monitoring UI for Hadoop Yarn Cluster
Posted by Todd Nist <ts...@gmail.com>.
Hi Srini,
If you start the $SPARK_HOME/sbin/start-history-server, you should be able
to see the basic spark ui. You will not see the master, but you will be
able to see the rest as I recall. You also need to add an entry into the
spark-defaults.conf, something like this:
*## Make sure the host and port match the node where your YARN history
server is running*
spark.yarn.historyServer.address localhost:18080
HTH.
-Todd
On Tue, Mar 3, 2015 at 12:47 PM, Srini Karri <sk...@gmail.com> wrote:
> Hi All,
>
> I am having trouble finding data related to my requirement. Here is the
> context, I have tried Standalone Spark Installation on Windows, I am able
> to submit the logs, able to see the history of events. My question is, is
> it possible to achieve the same monitoring UI experience with Yarn Cluster
> like Viewing workers, running/completed job stages in the Web UI.
> Currently, if we go to our Yarn Resource manager UI, we are able to see the
> Spark Jobs and it's logs. But it is not as rich as Spark Standalone master
> UI. Is this limitation for hadoop yarn cluster or is there any way we can
> hook this Spark Standalone master to Yarn Cluster?
>
> Any help is highly appreciated.
>
> Regards,
> Srini.
>