You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ambari.apache.org by Lian Jiang <ji...@gmail.com> on 2018/02/18 03:29:49 UTC
How to run scripts on ambari client nodes after hadoop setup is done
Hi,
I am using terraform and ambari blueprint to setup hadoop cluster in cloud.
I want to run scripts on the ambari client hosts to install something (e.g.
install ExtJS for ooize, add topology_script.py into hdfs conf folder...)
after the hadoop is setup automatically.
What's the recommended way to do this? Should I make the bootstrap scripts
running on the ambari client nodes check the cluster creation status?
http://{ambariServerIP}:8080/api/v1/clusters/${CLUSTER_NAME}/requests/1
Thanks for any clue.
Re: How to run scripts on ambari client nodes after hadoop setup is
done
Posted by Sean Roberts <sr...@hortonworks.com>.
Lian – I assumed you were asking how to wait for all agents to register.
How is Ansible bring triggered? If it’s running from a single playbook from one of your hosts than there is nothing that the client hosts need to wait for as the control host knows the cluster creation status.
If running automation independently on each host then you’ll need something to wait on from each host. That could simply by what you do on the ambari-server:
* Poll the cluster creation status.
Or, from Terraform use “remote-exec” provisioner which runs inside a “null_resource”:
https://www.terraform.io/docs/provisioners/null_resource.html
p.s. I agree that ambari would benefit from “pre/post-commands”. If there is not already a request in JIRA, make one.
--
Sean Roberts
@seano
From: Lian Jiang <ji...@gmail.com>
Reply-To: "user@ambari.apache.org" <us...@ambari.apache.org>
Date: Sunday, February 18, 2018 at 11:06 PM
To: "user@ambari.apache.org" <us...@ambari.apache.org>
Subject: Re: How to run scripts on ambari client nodes after hadoop setup is done
Sorry. Please help me understand.
Polling /api/v1/hosts is assuming the clients will wait the server to be up. As mentioned above, the clients cannot do this since it can cause deadlock. The client scripts need to complete without waiting for anything so that all clients can be ready. Then the server waiting for the clients to be ready can continue. Plus, clients waiting for hosts available seems not to be helpful. They need to know for sure the hadoop cluster setup is done so that they can for example copy files to hdfs. Thanks.
On Sun, Feb 18, 2018 at 1:12 PM, Sean Roberts <sr...@hortonworks.com>> wrote:
Lian – Poll /api/v1/hosts until the appropriate number of hosts are available or be more strict by checking for the correct host names.
Here is an ugly version of doing which checks the number of hosts required have registered (written in 2015 but still works): https://github.com/seanorama/ambari-bootstrap/blob/master/deploy/deploy-recommended-cluster.bash
Or an ansible version: https://github.com/hortonworks/ansible-hortonworks/blob/master/playbooks/roles/ambari-config/tasks/main.yml#L97-L111
--
Sean Roberts
@seano
From: Lian Jiang <ji...@gmail.com>>
Reply-To: "user@ambari.apache.org<ma...@ambari.apache.org>" <us...@ambari.apache.org>>
Date: Sunday, February 18, 2018 at 8:35 PM
To: "user@ambari.apache.org<ma...@ambari.apache.org>" <us...@ambari.apache.org>>
Subject: Re: How to run scripts on ambari client nodes after hadoop setup is done
Thanks Sean.
In the 100% automated process of machine preparation and hadoop cluster setup, it is easy for ambari server to start the server and monitor. However the ambari clients cannot wait ambari server to start otherwise the server and client lock each other (ambari server waits for all clients to be ready while all clients wait for server to be up). How do you solve this issue? Thanks.
On Sun, Feb 18, 2018 at 8:49 AM, Sean Roberts <sr...@hortonworks.com>> wrote:
Yes. Best method is to monitor the cluster creation status and then trigger your post-actions.
--
Sean Roberts
... blame the phone for typos
> On 18 Feb 2018, at 03:30, Lian Jiang <ji...@gmail.com>> wrote:
>
> Hi,
>
> I am using terraform and ambari blueprint to setup hadoop cluster in cloud. I want to run scripts on the ambari client hosts to install something (e.g. install ExtJS for ooize, add topology_script.py into hdfs conf folder...) after the hadoop is setup automatically.
>
> What's the recommended way to do this? Should I make the bootstrap scripts running on the ambari client nodes check the cluster creation status?
>
> http://{ambariServerIP}:8080/api/v1/clusters/${CLUSTER_NAME}/requests/1
>
> Thanks for any clue.
>
>
Re: How to run scripts on ambari client nodes after hadoop setup is done
Posted by Lian Jiang <ji...@gmail.com>.
Sorry. Please help me understand.
Polling /api/v1/hosts is assuming the clients will wait the server to be
up. As mentioned above, the clients cannot do this since it can cause
deadlock. The client scripts need to complete without waiting for anything
so that all clients can be ready. Then the server waiting for the clients
to be ready can continue. Plus, clients waiting for hosts available seems
not to be helpful. They need to know for sure the hadoop cluster setup is
done so that they can for example copy files to hdfs. Thanks.
On Sun, Feb 18, 2018 at 1:12 PM, Sean Roberts <sr...@hortonworks.com>
wrote:
> Lian – Poll /api/v1/hosts until the appropriate number of hosts are
> available or be more strict by checking for the correct host names.
>
>
>
> Here is an ugly version of doing which checks the number of hosts required
> have registered (written in 2015 but still works):
> https://github.com/seanorama/ambari-bootstrap/blob/master/
> deploy/deploy-recommended-cluster.bash
>
>
>
> Or an ansible version: https://github.com/hortonworks/ansible-
> hortonworks/blob/master/playbooks/roles/ambari-config/
> tasks/main.yml#L97-L111
>
>
>
>
>
> --
>
> Sean Roberts
>
> @seano
>
>
>
> *From: *Lian Jiang <ji...@gmail.com>
> *Reply-To: *"user@ambari.apache.org" <us...@ambari.apache.org>
> *Date: *Sunday, February 18, 2018 at 8:35 PM
> *To: *"user@ambari.apache.org" <us...@ambari.apache.org>
> *Subject: *Re: How to run scripts on ambari client nodes after hadoop
> setup is done
>
>
>
> Thanks Sean.
>
>
>
> In the 100% automated process of machine preparation and hadoop cluster
> setup, it is easy for ambari server to start the server and monitor.
> However the ambari clients cannot wait ambari server to start otherwise the
> server and client lock each other (ambari server waits for all clients to
> be ready while all clients wait for server to be up). How do you solve this
> issue? Thanks.
>
>
>
> On Sun, Feb 18, 2018 at 8:49 AM, Sean Roberts <sr...@hortonworks.com>
> wrote:
>
> Yes. Best method is to monitor the cluster creation status and then
> trigger your post-actions.
>
> --
> Sean Roberts
>
> ... blame the phone for typos
>
>
> > On 18 Feb 2018, at 03:30, Lian Jiang <ji...@gmail.com> wrote:
> >
> > Hi,
> >
> > I am using terraform and ambari blueprint to setup hadoop cluster in
> cloud. I want to run scripts on the ambari client hosts to install
> something (e.g. install ExtJS for ooize, add topology_script.py into hdfs
> conf folder...) after the hadoop is setup automatically.
> >
> > What's the recommended way to do this? Should I make the bootstrap
> scripts running on the ambari client nodes check the cluster creation
> status?
> >
> > http://{ambariServerIP}:8080/api/v1/clusters/${CLUSTER_NAME}/requests/1
> >
> > Thanks for any clue.
> >
> >
>
>
>
Re: How to run scripts on ambari client nodes after hadoop setup is
done
Posted by Sean Roberts <sr...@hortonworks.com>.
Lian – Poll /api/v1/hosts until the appropriate number of hosts are available or be more strict by checking for the correct host names.
Here is an ugly version of doing which checks the number of hosts required have registered (written in 2015 but still works): https://github.com/seanorama/ambari-bootstrap/blob/master/deploy/deploy-recommended-cluster.bash
Or an ansible version: https://github.com/hortonworks/ansible-hortonworks/blob/master/playbooks/roles/ambari-config/tasks/main.yml#L97-L111
--
Sean Roberts
@seano
From: Lian Jiang <ji...@gmail.com>
Reply-To: "user@ambari.apache.org" <us...@ambari.apache.org>
Date: Sunday, February 18, 2018 at 8:35 PM
To: "user@ambari.apache.org" <us...@ambari.apache.org>
Subject: Re: How to run scripts on ambari client nodes after hadoop setup is done
Thanks Sean.
In the 100% automated process of machine preparation and hadoop cluster setup, it is easy for ambari server to start the server and monitor. However the ambari clients cannot wait ambari server to start otherwise the server and client lock each other (ambari server waits for all clients to be ready while all clients wait for server to be up). How do you solve this issue? Thanks.
On Sun, Feb 18, 2018 at 8:49 AM, Sean Roberts <sr...@hortonworks.com>> wrote:
Yes. Best method is to monitor the cluster creation status and then trigger your post-actions.
--
Sean Roberts
... blame the phone for typos
> On 18 Feb 2018, at 03:30, Lian Jiang <ji...@gmail.com>> wrote:
>
> Hi,
>
> I am using terraform and ambari blueprint to setup hadoop cluster in cloud. I want to run scripts on the ambari client hosts to install something (e.g. install ExtJS for ooize, add topology_script.py into hdfs conf folder...) after the hadoop is setup automatically.
>
> What's the recommended way to do this? Should I make the bootstrap scripts running on the ambari client nodes check the cluster creation status?
>
> http://{ambariServerIP}:8080/api/v1/clusters/${CLUSTER_NAME}/requests/1
>
> Thanks for any clue.
>
>
Re: How to run scripts on ambari client nodes after hadoop setup is done
Posted by Lian Jiang <ji...@gmail.com>.
Thanks Sean.
In the 100% automated process of machine preparation and hadoop cluster
setup, it is easy for ambari server to start the server and monitor.
However the ambari clients cannot wait ambari server to start otherwise the
server and client lock each other (ambari server waits for all clients to
be ready while all clients wait for server to be up). How do you solve this
issue? Thanks.
On Sun, Feb 18, 2018 at 8:49 AM, Sean Roberts <sr...@hortonworks.com>
wrote:
> Yes. Best method is to monitor the cluster creation status and then
> trigger your post-actions.
>
> --
> Sean Roberts
>
> ... blame the phone for typos
>
> > On 18 Feb 2018, at 03:30, Lian Jiang <ji...@gmail.com> wrote:
> >
> > Hi,
> >
> > I am using terraform and ambari blueprint to setup hadoop cluster in
> cloud. I want to run scripts on the ambari client hosts to install
> something (e.g. install ExtJS for ooize, add topology_script.py into hdfs
> conf folder...) after the hadoop is setup automatically.
> >
> > What's the recommended way to do this? Should I make the bootstrap
> scripts running on the ambari client nodes check the cluster creation
> status?
> >
> > http://{ambariServerIP}:8080/api/v1/clusters/${CLUSTER_NAME}/requests/1
> >
> > Thanks for any clue.
> >
> >
>
>
Re: How to run scripts on ambari client nodes after hadoop setup is
done
Posted by Sean Roberts <sr...@hortonworks.com>.
Yes. Best method is to monitor the cluster creation status and then trigger your post-actions.
--
Sean Roberts
... blame the phone for typos
> On 18 Feb 2018, at 03:30, Lian Jiang <ji...@gmail.com> wrote:
>
> Hi,
>
> I am using terraform and ambari blueprint to setup hadoop cluster in cloud. I want to run scripts on the ambari client hosts to install something (e.g. install ExtJS for ooize, add topology_script.py into hdfs conf folder...) after the hadoop is setup automatically.
>
> What's the recommended way to do this? Should I make the bootstrap scripts running on the ambari client nodes check the cluster creation status?
>
> http://{ambariServerIP}:8080/api/v1/clusters/${CLUSTER_NAME}/requests/1
>
> Thanks for any clue.
>
>