You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Rutuja Kulkarni <ru...@gmail.com> on 2016/06/08 22:56:37 UTC
[ Standalone Spark Cluster ] - Track node status
Hello!
I'm trying to setup a standalone spark cluster and wondering how to track
status of all of it's nodes. I wonder if something like Yarn REST API or
HDFS CLI exists in Spark world that can provide status of nodes on such a
cluster. Any pointers would be greatly appreciated.
--
*Regards,*
*Rutuja Kulkarni*
Re: [ Standalone Spark Cluster ] - Track node status
Posted by Rutuja Kulkarni <ru...@gmail.com>.
I am wondering if I could get the worker statuses irrespective of the
applications running on them?
To check whether the worker is healthy even in its idle state?
Regards,
Rutuja
On Thu, Jun 9, 2016 at 2:48 PM, Todd Nist <ts...@gmail.com> wrote:
> Does the Spark Rest API defined here,
> http://spark.apache.org/docs/latest/monitoring.html#rest-api, meet you
> needs?
>
> You can query the driver to get a list of applications, then use the
> application id to get a list of executors, stages, ... and more.
>
> For example, if you query your driver at port 4040, default port, it will
> give you a list of applications:
> Query
>
> http://localhost:4040api/v1/applications
>
> Results:
>
> [ {
> "id" : "app-20160609163621-0000",
> "name" : "YourAapplicationNameHere",
> "attempts" : [ {
> "startTime" : "2016-06-09T20:36:20.007GMT",
> "endTime" : "1969-12-31T23:59:59.999GMT",
> "sparkUser" : "",
> "completed" : false
> } ]} ]
>
> If you then apply the application id to the below query it will give you a
> list of executors, something like this:
> Query
>
> http://localhost:4040/api/v1/applications/app-20160609163621-0000/executors
>
> Results
>
> [ {
> "id" : "2",
> "hostPort" : "radtech-mbp:59278",
> "rddBlocks" : 0,
> "memoryUsed" : 0,
> "diskUsed" : 0,
> "activeTasks" : 0,
> "failedTasks" : 0,
> "completedTasks" : 30374,
> "totalTasks" : 30374,
> "totalDuration" : 1086651,
> "totalInputBytes" : 71157816,
> "totalShuffleRead" : 64189233,
> "totalShuffleWrite" : 18476037,
> "maxMemory" : 535953408,
> "executorLogs" : {
> "stdout" : "http://192.168.1.109:8082/logPage/?appId=app-20160609163621-0000&executorId=2&logType=stdout",
> "stderr" : "http://192.168.1.109:8082/logPage/?appId=app-20160609163621-0000&executorId=2&logType=stderr"
> }}, {
> "id" : "driver",
> "hostPort" : "192.168.1.109:59247",
> "rddBlocks" : 0,
> "memoryUsed" : 0,
> "diskUsed" : 0,
> "activeTasks" : 0,
> "failedTasks" : 0,
> "completedTasks" : 0,
> "totalTasks" : 0,
> "totalDuration" : 0,
> "totalInputBytes" : 0,
> "totalShuffleRead" : 0,
> "totalShuffleWrite" : 0,
> "maxMemory" : 480116736,
> "executorLogs" : { }}, {
> "id" : "1",
> "hostPort" : "radtech-mbp:59275",
> "rddBlocks" : 0,
> "memoryUsed" : 0,
> "diskUsed" : 0,
> "activeTasks" : 0,
> "failedTasks" : 0,
> "completedTasks" : 33068,
> "totalTasks" : 33068,
> "totalDuration" : 1090921,
> "totalInputBytes" : 77805244,
> "totalShuffleRead" : 55007706,
> "totalShuffleWrite" : 24178333,
> "maxMemory" : 535953408,
> "executorLogs" : {
> "stdout" : "http://192.168.1.109:8083/logPage/?appId=app-20160609163621-0000&executorId=1&logType=stdout",
> "stderr" : "http://192.168.1.109:8083/logPage/?appId=app-20160609163621-0000&executorId=1&logType=stderr"
> }}, {
> "id" : "0",
> "hostPort" : "radtech-mbp:59277",
> "rddBlocks" : 0,
> "memoryUsed" : 0,
> "diskUsed" : 0,
> "activeTasks" : 0,
> "failedTasks" : 0,
> "completedTasks" : 32985,
> "totalTasks" : 32985,
> "totalDuration" : 1096802,
> "totalInputBytes" : 75209684,
> "totalShuffleRead" : 69982060,
> "totalShuffleWrite" : 17572462,
> "maxMemory" : 535953408,
> "executorLogs" : {
> "stdout" : "http://192.168.1.109:8081/logPage/?appId=app-20160609163621-0000&executorId=0&logType=stdout",
> "stderr" : "http://192.168.1.109:8081/logPage/?appId=app-20160609163621-0000&executorId=0&logType=stderr"
> }} ]
>
> Not sure if that's what your looking for or not.
>
> HTH.
>
> -Todd
>
> On Thu, Jun 9, 2016 at 3:20 PM, Mich Talebzadeh <mich.talebzadeh@gmail.com
> > wrote:
>
>> Hi Rutuja,
>>
>> I am not certain whether such tool exists or not, However, opening a JIRA
>> may be beneficial and would not do any harm.
>>
>> You may look for workaround. Now my understanding is that your need is
>> for monitoring the health of the cluster?
>>
>> HTH
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 9 June 2016 at 19:45, Rutuja Kulkarni <ru...@gmail.com>
>> wrote:
>>
>>>
>>>
>>> Thanks again Mich!
>>> If there does not exist any interface like REST API or CLI for this, I
>>> would like to open a JIRA on exposing such a REST interface in SPARK which
>>> would list all the worker nodes.
>>> Please let me know if this seems to be the right thing to do for the
>>> community.
>>>
>>>
>>> Regards,
>>> Rutuja Kulkarni
>>>
>>>
>>> On Wed, Jun 8, 2016 at 5:36 PM, Mich Talebzadeh <
>>> mich.talebzadeh@gmail.com> wrote:
>>>
>>>> The other way is to log in to the individual nodes and do
>>>>
>>>> jps
>>>>
>>>> 24819 Worker
>>>>
>>>> And you Processes identified as worker
>>>>
>>>> Also you can use jmonitor to see what they are doing resource wise
>>>>
>>>> You can of course write a small shell script to see if Worker(s) are up
>>>> and running in every node and alert if they are down?
>>>>
>>>> HTH
>>>>
>>>> Dr Mich Talebzadeh
>>>>
>>>>
>>>>
>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>
>>>>
>>>>
>>>> http://talebzadehmich.wordpress.com
>>>>
>>>>
>>>>
>>>> On 9 June 2016 at 01:27, Rutuja Kulkarni <ru...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thank you for the quick response.
>>>>> So the workers section would list all the running worker nodes in the
>>>>> standalone Spark cluster?
>>>>> I was also wondering if this is the only way to retrieve worker nodes
>>>>> or is there something like a Web API or CLI I could use?
>>>>> Thanks.
>>>>>
>>>>> Regards,
>>>>> Rutuja
>>>>>
>>>>> On Wed, Jun 8, 2016 at 4:02 PM, Mich Talebzadeh <
>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>
>>>>>> check port 8080 on the node that you started start-master.sh
>>>>>>
>>>>>>
>>>>>>
>>>>>> [image: Inline images 2]
>>>>>>
>>>>>> HTH
>>>>>>
>>>>>>
>>>>>> Dr Mich Talebzadeh
>>>>>>
>>>>>>
>>>>>>
>>>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>>>
>>>>>>
>>>>>>
>>>>>> http://talebzadehmich.wordpress.com
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 8 June 2016 at 23:56, Rutuja Kulkarni <rutuja.kulkarni24@gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Hello!
>>>>>>>
>>>>>>> I'm trying to setup a standalone spark cluster and wondering how to
>>>>>>> track status of all of it's nodes. I wonder if something like Yarn REST API
>>>>>>> or HDFS CLI exists in Spark world that can provide status of nodes on such
>>>>>>> a cluster. Any pointers would be greatly appreciated.
>>>>>>>
>>>>>>> --
>>>>>>> *Regards,*
>>>>>>> *Rutuja Kulkarni*
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Regards,*
>>>>> *Rutuja Kulkarni*
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> *Regards,*
>>> *Rutuja Kulkarni*
>>>
>>>
>>>
>>
>
--
*Regards,*
*Rutuja Kulkarni*
Re: [ Standalone Spark Cluster ] - Track node status
Posted by Todd Nist <ts...@gmail.com>.
Does the Spark Rest API defined here,
http://spark.apache.org/docs/latest/monitoring.html#rest-api, meet you
needs?
You can query the driver to get a list of applications, then use the
application id to get a list of executors, stages, ... and more.
For example, if you query your driver at port 4040, default port, it will
give you a list of applications:
Query
http://localhost:4040api/v1/applications
Results:
[ {
"id" : "app-20160609163621-0000",
"name" : "YourAapplicationNameHere",
"attempts" : [ {
"startTime" : "2016-06-09T20:36:20.007GMT",
"endTime" : "1969-12-31T23:59:59.999GMT",
"sparkUser" : "",
"completed" : false
} ]} ]
If you then apply the application id to the below query it will give you a
list of executors, something like this:
Query
http://localhost:4040/api/v1/applications/app-20160609163621-0000/executors
Results
[ {
"id" : "2",
"hostPort" : "radtech-mbp:59278",
"rddBlocks" : 0,
"memoryUsed" : 0,
"diskUsed" : 0,
"activeTasks" : 0,
"failedTasks" : 0,
"completedTasks" : 30374,
"totalTasks" : 30374,
"totalDuration" : 1086651,
"totalInputBytes" : 71157816,
"totalShuffleRead" : 64189233,
"totalShuffleWrite" : 18476037,
"maxMemory" : 535953408,
"executorLogs" : {
"stdout" : "http://192.168.1.109:8082/logPage/?appId=app-20160609163621-0000&executorId=2&logType=stdout",
"stderr" : "http://192.168.1.109:8082/logPage/?appId=app-20160609163621-0000&executorId=2&logType=stderr"
}}, {
"id" : "driver",
"hostPort" : "192.168.1.109:59247",
"rddBlocks" : 0,
"memoryUsed" : 0,
"diskUsed" : 0,
"activeTasks" : 0,
"failedTasks" : 0,
"completedTasks" : 0,
"totalTasks" : 0,
"totalDuration" : 0,
"totalInputBytes" : 0,
"totalShuffleRead" : 0,
"totalShuffleWrite" : 0,
"maxMemory" : 480116736,
"executorLogs" : { }}, {
"id" : "1",
"hostPort" : "radtech-mbp:59275",
"rddBlocks" : 0,
"memoryUsed" : 0,
"diskUsed" : 0,
"activeTasks" : 0,
"failedTasks" : 0,
"completedTasks" : 33068,
"totalTasks" : 33068,
"totalDuration" : 1090921,
"totalInputBytes" : 77805244,
"totalShuffleRead" : 55007706,
"totalShuffleWrite" : 24178333,
"maxMemory" : 535953408,
"executorLogs" : {
"stdout" : "http://192.168.1.109:8083/logPage/?appId=app-20160609163621-0000&executorId=1&logType=stdout",
"stderr" : "http://192.168.1.109:8083/logPage/?appId=app-20160609163621-0000&executorId=1&logType=stderr"
}}, {
"id" : "0",
"hostPort" : "radtech-mbp:59277",
"rddBlocks" : 0,
"memoryUsed" : 0,
"diskUsed" : 0,
"activeTasks" : 0,
"failedTasks" : 0,
"completedTasks" : 32985,
"totalTasks" : 32985,
"totalDuration" : 1096802,
"totalInputBytes" : 75209684,
"totalShuffleRead" : 69982060,
"totalShuffleWrite" : 17572462,
"maxMemory" : 535953408,
"executorLogs" : {
"stdout" : "http://192.168.1.109:8081/logPage/?appId=app-20160609163621-0000&executorId=0&logType=stdout",
"stderr" : "http://192.168.1.109:8081/logPage/?appId=app-20160609163621-0000&executorId=0&logType=stderr"
}} ]
Not sure if that's what your looking for or not.
HTH.
-Todd
On Thu, Jun 9, 2016 at 3:20 PM, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Hi Rutuja,
>
> I am not certain whether such tool exists or not, However, opening a JIRA
> may be beneficial and would not do any harm.
>
> You may look for workaround. Now my understanding is that your need is for
> monitoring the health of the cluster?
>
> HTH
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 9 June 2016 at 19:45, Rutuja Kulkarni <ru...@gmail.com>
> wrote:
>
>>
>>
>> Thanks again Mich!
>> If there does not exist any interface like REST API or CLI for this, I
>> would like to open a JIRA on exposing such a REST interface in SPARK which
>> would list all the worker nodes.
>> Please let me know if this seems to be the right thing to do for the
>> community.
>>
>>
>> Regards,
>> Rutuja Kulkarni
>>
>>
>> On Wed, Jun 8, 2016 at 5:36 PM, Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>>
>>> The other way is to log in to the individual nodes and do
>>>
>>> jps
>>>
>>> 24819 Worker
>>>
>>> And you Processes identified as worker
>>>
>>> Also you can use jmonitor to see what they are doing resource wise
>>>
>>> You can of course write a small shell script to see if Worker(s) are up
>>> and running in every node and alert if they are down?
>>>
>>> HTH
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>> On 9 June 2016 at 01:27, Rutuja Kulkarni <ru...@gmail.com>
>>> wrote:
>>>
>>>> Thank you for the quick response.
>>>> So the workers section would list all the running worker nodes in the
>>>> standalone Spark cluster?
>>>> I was also wondering if this is the only way to retrieve worker nodes
>>>> or is there something like a Web API or CLI I could use?
>>>> Thanks.
>>>>
>>>> Regards,
>>>> Rutuja
>>>>
>>>> On Wed, Jun 8, 2016 at 4:02 PM, Mich Talebzadeh <
>>>> mich.talebzadeh@gmail.com> wrote:
>>>>
>>>>> check port 8080 on the node that you started start-master.sh
>>>>>
>>>>>
>>>>>
>>>>> [image: Inline images 2]
>>>>>
>>>>> HTH
>>>>>
>>>>>
>>>>> Dr Mich Talebzadeh
>>>>>
>>>>>
>>>>>
>>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>>
>>>>>
>>>>>
>>>>> http://talebzadehmich.wordpress.com
>>>>>
>>>>>
>>>>>
>>>>> On 8 June 2016 at 23:56, Rutuja Kulkarni <ru...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hello!
>>>>>>
>>>>>> I'm trying to setup a standalone spark cluster and wondering how to
>>>>>> track status of all of it's nodes. I wonder if something like Yarn REST API
>>>>>> or HDFS CLI exists in Spark world that can provide status of nodes on such
>>>>>> a cluster. Any pointers would be greatly appreciated.
>>>>>>
>>>>>> --
>>>>>> *Regards,*
>>>>>> *Rutuja Kulkarni*
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> *Regards,*
>>>> *Rutuja Kulkarni*
>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> *Regards,*
>> *Rutuja Kulkarni*
>>
>>
>>
>
Re: [ Standalone Spark Cluster ] - Track node status
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi Rutuja,
I am not certain whether such tool exists or not, However, opening a JIRA
may be beneficial and would not do any harm.
You may look for workaround. Now my understanding is that your need is for
monitoring the health of the cluster?
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 9 June 2016 at 19:45, Rutuja Kulkarni <ru...@gmail.com>
wrote:
>
>
> Thanks again Mich!
> If there does not exist any interface like REST API or CLI for this, I
> would like to open a JIRA on exposing such a REST interface in SPARK which
> would list all the worker nodes.
> Please let me know if this seems to be the right thing to do for the
> community.
>
>
> Regards,
> Rutuja Kulkarni
>
>
> On Wed, Jun 8, 2016 at 5:36 PM, Mich Talebzadeh <mich.talebzadeh@gmail.com
> > wrote:
>
>> The other way is to log in to the individual nodes and do
>>
>> jps
>>
>> 24819 Worker
>>
>> And you Processes identified as worker
>>
>> Also you can use jmonitor to see what they are doing resource wise
>>
>> You can of course write a small shell script to see if Worker(s) are up
>> and running in every node and alert if they are down?
>>
>> HTH
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 9 June 2016 at 01:27, Rutuja Kulkarni <ru...@gmail.com>
>> wrote:
>>
>>> Thank you for the quick response.
>>> So the workers section would list all the running worker nodes in the
>>> standalone Spark cluster?
>>> I was also wondering if this is the only way to retrieve worker nodes or
>>> is there something like a Web API or CLI I could use?
>>> Thanks.
>>>
>>> Regards,
>>> Rutuja
>>>
>>> On Wed, Jun 8, 2016 at 4:02 PM, Mich Talebzadeh <
>>> mich.talebzadeh@gmail.com> wrote:
>>>
>>>> check port 8080 on the node that you started start-master.sh
>>>>
>>>>
>>>>
>>>> [image: Inline images 2]
>>>>
>>>> HTH
>>>>
>>>>
>>>> Dr Mich Talebzadeh
>>>>
>>>>
>>>>
>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>
>>>>
>>>>
>>>> http://talebzadehmich.wordpress.com
>>>>
>>>>
>>>>
>>>> On 8 June 2016 at 23:56, Rutuja Kulkarni <ru...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hello!
>>>>>
>>>>> I'm trying to setup a standalone spark cluster and wondering how to
>>>>> track status of all of it's nodes. I wonder if something like Yarn REST API
>>>>> or HDFS CLI exists in Spark world that can provide status of nodes on such
>>>>> a cluster. Any pointers would be greatly appreciated.
>>>>>
>>>>> --
>>>>> *Regards,*
>>>>> *Rutuja Kulkarni*
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> *Regards,*
>>> *Rutuja Kulkarni*
>>>
>>>
>>>
>>
>
>
> --
> *Regards,*
> *Rutuja Kulkarni*
>
>
>
Re: [ Standalone Spark Cluster ] - Track node status
Posted by Rutuja Kulkarni <ru...@gmail.com>.
Thanks again Mich!
If there does not exist any interface like REST API or CLI for this, I
would like to open a JIRA on exposing such a REST interface in SPARK which
would list all the worker nodes.
Please let me know if this seems to be the right thing to do for the
community.
Regards,
Rutuja Kulkarni
On Wed, Jun 8, 2016 at 5:36 PM, Mich Talebzadeh <mi...@gmail.com>
wrote:
> The other way is to log in to the individual nodes and do
>
> jps
>
> 24819 Worker
>
> And you Processes identified as worker
>
> Also you can use jmonitor to see what they are doing resource wise
>
> You can of course write a small shell script to see if Worker(s) are up
> and running in every node and alert if they are down?
>
> HTH
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 9 June 2016 at 01:27, Rutuja Kulkarni <ru...@gmail.com>
> wrote:
>
>> Thank you for the quick response.
>> So the workers section would list all the running worker nodes in the
>> standalone Spark cluster?
>> I was also wondering if this is the only way to retrieve worker nodes or
>> is there something like a Web API or CLI I could use?
>> Thanks.
>>
>> Regards,
>> Rutuja
>>
>> On Wed, Jun 8, 2016 at 4:02 PM, Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>>
>>> check port 8080 on the node that you started start-master.sh
>>>
>>>
>>>
>>> [image: Inline images 2]
>>>
>>> HTH
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>> On 8 June 2016 at 23:56, Rutuja Kulkarni <ru...@gmail.com>
>>> wrote:
>>>
>>>> Hello!
>>>>
>>>> I'm trying to setup a standalone spark cluster and wondering how to
>>>> track status of all of it's nodes. I wonder if something like Yarn REST API
>>>> or HDFS CLI exists in Spark world that can provide status of nodes on such
>>>> a cluster. Any pointers would be greatly appreciated.
>>>>
>>>> --
>>>> *Regards,*
>>>> *Rutuja Kulkarni*
>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> *Regards,*
>> *Rutuja Kulkarni*
>>
>>
>>
>
--
*Regards,*
*Rutuja Kulkarni*
Re: [ Standalone Spark Cluster ] - Track node status
Posted by Mich Talebzadeh <mi...@gmail.com>.
The other way is to log in to the individual nodes and do
jps
24819 Worker
And you Processes identified as worker
Also you can use jmonitor to see what they are doing resource wise
You can of course write a small shell script to see if Worker(s) are up and
running in every node and alert if they are down?
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 9 June 2016 at 01:27, Rutuja Kulkarni <ru...@gmail.com>
wrote:
> Thank you for the quick response.
> So the workers section would list all the running worker nodes in the
> standalone Spark cluster?
> I was also wondering if this is the only way to retrieve worker nodes or
> is there something like a Web API or CLI I could use?
> Thanks.
>
> Regards,
> Rutuja
>
> On Wed, Jun 8, 2016 at 4:02 PM, Mich Talebzadeh <mich.talebzadeh@gmail.com
> > wrote:
>
>> check port 8080 on the node that you started start-master.sh
>>
>>
>>
>> [image: Inline images 2]
>>
>> HTH
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 8 June 2016 at 23:56, Rutuja Kulkarni <ru...@gmail.com>
>> wrote:
>>
>>> Hello!
>>>
>>> I'm trying to setup a standalone spark cluster and wondering how to
>>> track status of all of it's nodes. I wonder if something like Yarn REST API
>>> or HDFS CLI exists in Spark world that can provide status of nodes on such
>>> a cluster. Any pointers would be greatly appreciated.
>>>
>>> --
>>> *Regards,*
>>> *Rutuja Kulkarni*
>>>
>>>
>>>
>>
>
>
> --
> *Regards,*
> *Rutuja Kulkarni*
>
>
>
Re: [ Standalone Spark Cluster ] - Track node status
Posted by Rutuja Kulkarni <ru...@gmail.com>.
Thank you for the quick response.
So the workers section would list all the running worker nodes in the
standalone Spark cluster?
I was also wondering if this is the only way to retrieve worker nodes or is
there something like a Web API or CLI I could use?
Thanks.
Regards,
Rutuja
On Wed, Jun 8, 2016 at 4:02 PM, Mich Talebzadeh <mi...@gmail.com>
wrote:
> check port 8080 on the node that you started start-master.sh
>
>
>
> [image: Inline images 2]
>
> HTH
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 8 June 2016 at 23:56, Rutuja Kulkarni <ru...@gmail.com>
> wrote:
>
>> Hello!
>>
>> I'm trying to setup a standalone spark cluster and wondering how to track
>> status of all of it's nodes. I wonder if something like Yarn REST API or
>> HDFS CLI exists in Spark world that can provide status of nodes on such a
>> cluster. Any pointers would be greatly appreciated.
>>
>> --
>> *Regards,*
>> *Rutuja Kulkarni*
>>
>>
>>
>
--
*Regards,*
*Rutuja Kulkarni*
Re: [ Standalone Spark Cluster ] - Track node status
Posted by Mich Talebzadeh <mi...@gmail.com>.
check port 8080 on the node that you started start-master.sh
[image: Inline images 2]
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 8 June 2016 at 23:56, Rutuja Kulkarni <ru...@gmail.com>
wrote:
> Hello!
>
> I'm trying to setup a standalone spark cluster and wondering how to track
> status of all of it's nodes. I wonder if something like Yarn REST API or
> HDFS CLI exists in Spark world that can provide status of nodes on such a
> cluster. Any pointers would be greatly appreciated.
>
> --
> *Regards,*
> *Rutuja Kulkarni*
>
>
>