You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Kenichi Maehashi <we...@kenichimaehashi.com> on 2014/11/18 09:27:13 UTC
Slave Node Management in Standalone Cluster
Hi,
I'm operating Spark in standalone cluster configuration (3 slaves) and
have some question.
1. How can I stop a slave on the specific node?
Under `sbin/` directory, there are
`start-{all,master,slave,slaves}` and `stop-{all,master,slaves}`, but
no `stop-slave`. Are there any way to stop the specific (e.g., the
2nd) slave via command line?
2. How can I check cluster status from command line?
Are there any way to confirm that all Master / Workers are up and
working without using Web UI?
Thanks in advance!
--
Kenichi Maehashi
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Slave Node Management in Standalone Cluster
Posted by Kenichi Maehashi <we...@kenichimaehashi.com>.
Hi Akhil and Kousuke,
Thank you for your quick response.
Monitoring through JSON API seems straightforward and cool.
Thanks again!
2014-11-18 19:06 GMT+09:00 Kousuke Saruta <sa...@oss.nttdata.co.jp>:
> Hi Kenichi
>
>> 1. How can I stop a slave on the specific node?
>> Under `sbin/` directory, there are
>> `start-{all,master,slave,slaves}` and `stop-{all,master,slaves}`, but
>> no `stop-slave`. Are there any way to stop the specific (e.g., the
>> 2nd) slave via command line?
>
>
> You can use sbin/spark-daemon.sh on the machine where the worker you'd like
> to stop runs.
> First, you find PID of the worker you'd like to stop and second, you find
> PID file of the worker.
> The PID file is on /tmp/ by default and the file name is like as follows.
>
> xxx.org.apache.spark.deploy.worker.Worker-<WorkerID>.pid
>
> After you find the PID file, you run the following command.
>
> sbin/spark-daemon.sh stop org.apache.spark.worker.Worker <WorkerID>
>
>> 2. How can I check cluster status from command line?
>> Are there any way to confirm that all Master / Workers are up and
>> working without using Web UI?
>
>
> AFAIK, there are no command line tools for checking statuses of standalone
> cluster.
> Instead of that, you can use special URL like as follows.
>
> http://<master or worker's hostname>:<webui-port>/json
>
> You can get Master and Worker status as JSON format data.
>
> - Kousuke
>
>
> (2014/11/18 0:27), Kenichi Maehashi wrote:
>>
>> Hi,
>>
>> I'm operating Spark in standalone cluster configuration (3 slaves) and
>> have some question.
>>
>> 1. How can I stop a slave on the specific node?
>> Under `sbin/` directory, there are
>> `start-{all,master,slave,slaves}` and `stop-{all,master,slaves}`, but
>> no `stop-slave`. Are there any way to stop the specific (e.g., the
>> 2nd) slave via command line?
>>
>> 2. How can I check cluster status from command line?
>> Are there any way to confirm that all Master / Workers are up and
>> working without using Web UI?
>>
>> Thanks in advance!
>>
>
--
Kenichi Maehashi
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Slave Node Management in Standalone Cluster
Posted by Kousuke Saruta <sa...@oss.nttdata.co.jp>.
Hi Kenichi
> 1. How can I stop a slave on the specific node?
> Under `sbin/` directory, there are
> `start-{all,master,slave,slaves}` and `stop-{all,master,slaves}`, but
> no `stop-slave`. Are there any way to stop the specific (e.g., the
> 2nd) slave via command line?
You can use sbin/spark-daemon.sh on the machine where the worker you'd
like to stop runs.
First, you find PID of the worker you'd like to stop and second, you
find PID file of the worker.
The PID file is on /tmp/ by default and the file name is like as follows.
xxx.org.apache.spark.deploy.worker.Worker-<WorkerID>.pid
After you find the PID file, you run the following command.
sbin/spark-daemon.sh stop org.apache.spark.worker.Worker <WorkerID>
> 2. How can I check cluster status from command line?
> Are there any way to confirm that all Master / Workers are up and
> working without using Web UI?
AFAIK, there are no command line tools for checking statuses of
standalone cluster.
Instead of that, you can use special URL like as follows.
http://<master or worker's hostname>:<webui-port>/json
You can get Master and Worker status as JSON format data.
- Kousuke
(2014/11/18 0:27), Kenichi Maehashi wrote:
> Hi,
>
> I'm operating Spark in standalone cluster configuration (3 slaves) and
> have some question.
>
> 1. How can I stop a slave on the specific node?
> Under `sbin/` directory, there are
> `start-{all,master,slave,slaves}` and `stop-{all,master,slaves}`, but
> no `stop-slave`. Are there any way to stop the specific (e.g., the
> 2nd) slave via command line?
>
> 2. How can I check cluster status from command line?
> Are there any way to confirm that all Master / Workers are up and
> working without using Web UI?
>
> Thanks in advance!
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Slave Node Management in Standalone Cluster
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
1. You can comment the rest of the workers from the conf/slaves file and do
a stop-slaves.sh from that machine to stop the specific worker.
2. There is no direct command for it, but you can do something like the
following:
$ curl localhost:8080 | grep "Applications" -C 10 | head -n20
Where localhost is your master machine and 8080 is the web ui port.
You can also look at the metrics
<http://spark.apache.org/docs/latest/monitoring.html#metrics> options for
more sophisticated version.
Thanks
Best Regards
On Tue, Nov 18, 2014 at 1:57 PM, Kenichi Maehashi <
webmaster@kenichimaehashi.com> wrote:
> Hi,
>
> I'm operating Spark in standalone cluster configuration (3 slaves) and
> have some question.
>
> 1. How can I stop a slave on the specific node?
> Under `sbin/` directory, there are
> `start-{all,master,slave,slaves}` and `stop-{all,master,slaves}`, but
> no `stop-slave`. Are there any way to stop the specific (e.g., the
> 2nd) slave via command line?
>
> 2. How can I check cluster status from command line?
> Are there any way to confirm that all Master / Workers are up and
> working without using Web UI?
>
> Thanks in advance!
>
> --
> Kenichi Maehashi
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>