You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Ed Kohlwey <ek...@gmail.com> on 2015/02/13 22:31:21 UTC
Ambari + Ansible management
Hi,
I'm interested in using Ambari + Ansible to manage a Kerberized cluster.
The current contemplation is to use an Ansible module to make rest calls to
the Ambari server in order to configure the nodes.
We would also like to be able to automatically set up HA and secure mode.
We have scripts currently for setting up a primary and secondary KDC and
issuing keytabs, so that is not a big deal, but I recall that the API
currently has no official support for setting up secure mode or for
deploying HA.
Can we use the undocumented API's to do this successfully? How future proof
is that approach? I noticed that there has been a lot of work on setting up
a cluster KDC in version 2.
Can any of the devs working on this suggest a good path forward to minimize
rework after the next release?
Re: Ambari + Ansible management
Posted by Sean Roberts <sr...@hortonworks.com>.
Regarding Ansible:
- I’ve used it to deploy blueprints. That doesn’t get you the HA sections, but Ansible can easily handle the blueprint upload and cluster creation (even template it with ‘jinja2’ to fill in the hosts).
--
Hortonworks - We do Hadoop
Sean Roberts
Partner Solutions Engineer - EMEA
@seano
From: Robert Nettleton <rn...@hortonworks.com>
Reply: dev@ambari.apache.org <de...@ambari.apache.org>
Date: February 19, 2015 at 22:04:15
To: dev@ambari.apache.org <de...@ambari.apache.org>
Subject: Re: Ambari + Ansible management
Hi Ed,
I’m not sure when 2.0 will be released.
For HA setups, using the low-level REST APIs directly probably won’t work, since we also have stack definition script changes added in 2.0 that support setting up things like HDFS NameNode HA . I’m not familiar with Ansible, so I’m not sure if you could use this to automate the process.
Aside from HA, the Blueprints feature was introduced in Ambari 1.6.1 as a feature (tech preview in 1.6), so the general feature has been in Ambari for at least two releases.
In general, I’d recommend using Blueprints wherever a declarative model is needed for setting up an Ambari cluster. Using the REST APIs directly will likely be a difficult process, and will be harder to maintain over time. The Blueprints processor handles the details of the low-level REST APIs, and may be a better fit long-term, depending upon your requirements.
Thanks,
Bob
On Feb 19, 2015, at 1:01 PM, Ed Kohlwey <ek...@gmail.com> wrote:
> Hi Bob,
> Do you know what the release schedule is for 2.0? We would like to use this
> in production within the next quarter-ish, so working on a non-certified
> release would not be ideal for us.
>
> Would it make sense (or even be possible) to perform the configuration via
> the REST API? Ie. installing the ZKFC, adding the namenodes, and updating
> their configuration files to have everything configured correctly? Ansible
> would be able to also perform the appropriate shell commands.
>
> On Wed, Feb 18, 2015 at 4:15 PM, Robert Nettleton <
> rnettleton@hortonworks.com> wrote:
>
>> Hi Ed,
>>
>> In Ambari 2.0, there will be support for deploying HA clusters using the
>> Ambari Blueprints feature.
>>
>> If you are interested in Blueprints generally, the following link provides
>> a good overview:
>>
>> https://cwiki.apache.org/confluence/display/AMBARI/Blueprints
>>
>> With the new Blueprints HA functionality in 2.0, it is possible to define
>> certain types of HA clusters from scratch, rather than having to use the HA
>> Wizard after a cluster startup to enable HA.
>>
>> In the initial release of this feature, it will be possible to deploy
>> clusters based on Blueprints that describe:
>>
>> 1. HDFS NameNode HA clusters
>> 2. Yarn ResourceManager HA clusters
>> 3. HBase RegoinServer HA clusters
>>
>> Since this is a new feature, we currently don’t have proper documentation
>> for the HA support in Blueprints yet, but I hope to be adding that soon to
>> the wiki.
>>
>> Hope this helps,
>> Bob
>>
>>
>>
>>
>> On Feb 13, 2015, at 4:31 PM, Ed Kohlwey <ek...@gmail.com> wrote:
>>
>>> Hi,
>>> I'm interested in using Ambari + Ansible to manage a Kerberized cluster.
>>> The current contemplation is to use an Ansible module to make rest calls
>> to
>>> the Ambari server in order to configure the nodes.
>>>
>>> We would also like to be able to automatically set up HA and secure mode.
>>>
>>> We have scripts currently for setting up a primary and secondary KDC and
>>> issuing keytabs, so that is not a big deal, but I recall that the API
>>> currently has no official support for setting up secure mode or for
>>> deploying HA.
>>>
>>> Can we use the undocumented API's to do this successfully? How future
>> proof
>>> is that approach? I noticed that there has been a lot of work on setting
>> up
>>> a cluster KDC in version 2.
>>>
>>> Can any of the devs working on this suggest a good path forward to
>> minimize
>>> rework after the next release?
>>
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Ambari + Ansible management
Posted by Robert Nettleton <rn...@hortonworks.com>.
Hi Ed,
I’m not sure when 2.0 will be released.
For HA setups, using the low-level REST APIs directly probably won’t work, since we also have stack definition script changes added in 2.0 that support setting up things like HDFS NameNode HA . I’m not familiar with Ansible, so I’m not sure if you could use this to automate the process.
Aside from HA, the Blueprints feature was introduced in Ambari 1.6.1 as a feature (tech preview in 1.6), so the general feature has been in Ambari for at least two releases.
In general, I’d recommend using Blueprints wherever a declarative model is needed for setting up an Ambari cluster. Using the REST APIs directly will likely be a difficult process, and will be harder to maintain over time. The Blueprints processor handles the details of the low-level REST APIs, and may be a better fit long-term, depending upon your requirements.
Thanks,
Bob
On Feb 19, 2015, at 1:01 PM, Ed Kohlwey <ek...@gmail.com> wrote:
> Hi Bob,
> Do you know what the release schedule is for 2.0? We would like to use this
> in production within the next quarter-ish, so working on a non-certified
> release would not be ideal for us.
>
> Would it make sense (or even be possible) to perform the configuration via
> the REST API? Ie. installing the ZKFC, adding the namenodes, and updating
> their configuration files to have everything configured correctly? Ansible
> would be able to also perform the appropriate shell commands.
>
> On Wed, Feb 18, 2015 at 4:15 PM, Robert Nettleton <
> rnettleton@hortonworks.com> wrote:
>
>> Hi Ed,
>>
>> In Ambari 2.0, there will be support for deploying HA clusters using the
>> Ambari Blueprints feature.
>>
>> If you are interested in Blueprints generally, the following link provides
>> a good overview:
>>
>> https://cwiki.apache.org/confluence/display/AMBARI/Blueprints
>>
>> With the new Blueprints HA functionality in 2.0, it is possible to define
>> certain types of HA clusters from scratch, rather than having to use the HA
>> Wizard after a cluster startup to enable HA.
>>
>> In the initial release of this feature, it will be possible to deploy
>> clusters based on Blueprints that describe:
>>
>> 1. HDFS NameNode HA clusters
>> 2. Yarn ResourceManager HA clusters
>> 3. HBase RegoinServer HA clusters
>>
>> Since this is a new feature, we currently don’t have proper documentation
>> for the HA support in Blueprints yet, but I hope to be adding that soon to
>> the wiki.
>>
>> Hope this helps,
>> Bob
>>
>>
>>
>>
>> On Feb 13, 2015, at 4:31 PM, Ed Kohlwey <ek...@gmail.com> wrote:
>>
>>> Hi,
>>> I'm interested in using Ambari + Ansible to manage a Kerberized cluster.
>>> The current contemplation is to use an Ansible module to make rest calls
>> to
>>> the Ambari server in order to configure the nodes.
>>>
>>> We would also like to be able to automatically set up HA and secure mode.
>>>
>>> We have scripts currently for setting up a primary and secondary KDC and
>>> issuing keytabs, so that is not a big deal, but I recall that the API
>>> currently has no official support for setting up secure mode or for
>>> deploying HA.
>>>
>>> Can we use the undocumented API's to do this successfully? How future
>> proof
>>> is that approach? I noticed that there has been a lot of work on setting
>> up
>>> a cluster KDC in version 2.
>>>
>>> Can any of the devs working on this suggest a good path forward to
>> minimize
>>> rework after the next release?
>>
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Ambari + Ansible management
Posted by Ed Kohlwey <ek...@gmail.com>.
Hi Bob,
Do you know what the release schedule is for 2.0? We would like to use this
in production within the next quarter-ish, so working on a non-certified
release would not be ideal for us.
Would it make sense (or even be possible) to perform the configuration via
the REST API? Ie. installing the ZKFC, adding the namenodes, and updating
their configuration files to have everything configured correctly? Ansible
would be able to also perform the appropriate shell commands.
On Wed, Feb 18, 2015 at 4:15 PM, Robert Nettleton <
rnettleton@hortonworks.com> wrote:
> Hi Ed,
>
> In Ambari 2.0, there will be support for deploying HA clusters using the
> Ambari Blueprints feature.
>
> If you are interested in Blueprints generally, the following link provides
> a good overview:
>
> https://cwiki.apache.org/confluence/display/AMBARI/Blueprints
>
> With the new Blueprints HA functionality in 2.0, it is possible to define
> certain types of HA clusters from scratch, rather than having to use the HA
> Wizard after a cluster startup to enable HA.
>
> In the initial release of this feature, it will be possible to deploy
> clusters based on Blueprints that describe:
>
> 1. HDFS NameNode HA clusters
> 2. Yarn ResourceManager HA clusters
> 3. HBase RegoinServer HA clusters
>
> Since this is a new feature, we currently don’t have proper documentation
> for the HA support in Blueprints yet, but I hope to be adding that soon to
> the wiki.
>
> Hope this helps,
> Bob
>
>
>
>
> On Feb 13, 2015, at 4:31 PM, Ed Kohlwey <ek...@gmail.com> wrote:
>
> > Hi,
> > I'm interested in using Ambari + Ansible to manage a Kerberized cluster.
> > The current contemplation is to use an Ansible module to make rest calls
> to
> > the Ambari server in order to configure the nodes.
> >
> > We would also like to be able to automatically set up HA and secure mode.
> >
> > We have scripts currently for setting up a primary and secondary KDC and
> > issuing keytabs, so that is not a big deal, but I recall that the API
> > currently has no official support for setting up secure mode or for
> > deploying HA.
> >
> > Can we use the undocumented API's to do this successfully? How future
> proof
> > is that approach? I noticed that there has been a lot of work on setting
> up
> > a cluster KDC in version 2.
> >
> > Can any of the devs working on this suggest a good path forward to
> minimize
> > rework after the next release?
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Ambari + Ansible management
Posted by Robert Nettleton <rn...@hortonworks.com>.
Hi Ed,
In Ambari 2.0, there will be support for deploying HA clusters using the Ambari Blueprints feature.
If you are interested in Blueprints generally, the following link provides a good overview:
https://cwiki.apache.org/confluence/display/AMBARI/Blueprints
With the new Blueprints HA functionality in 2.0, it is possible to define certain types of HA clusters from scratch, rather than having to use the HA Wizard after a cluster startup to enable HA.
In the initial release of this feature, it will be possible to deploy clusters based on Blueprints that describe:
1. HDFS NameNode HA clusters
2. Yarn ResourceManager HA clusters
3. HBase RegoinServer HA clusters
Since this is a new feature, we currently don’t have proper documentation for the HA support in Blueprints yet, but I hope to be adding that soon to the wiki.
Hope this helps,
Bob
On Feb 13, 2015, at 4:31 PM, Ed Kohlwey <ek...@gmail.com> wrote:
> Hi,
> I'm interested in using Ambari + Ansible to manage a Kerberized cluster.
> The current contemplation is to use an Ansible module to make rest calls to
> the Ambari server in order to configure the nodes.
>
> We would also like to be able to automatically set up HA and secure mode.
>
> We have scripts currently for setting up a primary and secondary KDC and
> issuing keytabs, so that is not a big deal, but I recall that the API
> currently has no official support for setting up secure mode or for
> deploying HA.
>
> Can we use the undocumented API's to do this successfully? How future proof
> is that approach? I noticed that there has been a lot of work on setting up
> a cluster KDC in version 2.
>
> Can any of the devs working on this suggest a good path forward to minimize
> rework after the next release?
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Ambari + Ansible management
Posted by Robert Levas <rl...@hortonworks.com>.
Hi EdŠ
I have been working on the automated Kerberos implementation, so I can
give you some insight on configuring that, however I don¹t have any
experience with HA or Ansible.
In Ambari 2.0, you should be able to enable and disable Kerberos via the
REST API. I am not sure if any of this is documented. If not, it will be
soon.
The steps involved in enabling Kerberos are as follows:
1) Configure the KDC and Kerberos environment
2) Install the Kerberos service
3) Create/update the Kerberos Descriptors for the services installed on
your cluster
4) Set the KDC administrative credentials (in the HTTP session)
5) Enable Kerberos
Once Kerberos is enabled, adding new services and hosts should be
automatically handled, however you may want to update the default Kerberos
descriptors for any new services you are adding. Also, in order to
perform any Kerberos related tasks, you will need to provide the KDC
administrative credentials. These are not stored for longer than your
HTTP session. On that note, your connection to Ambari needs to be able to
handle a session.
I plan to send you more details on the steps listed above, I will try to
get that out to you shortly.
Sorry for the delay in getting this information out to you.
Rob
On 2/13/15, 4:31 PM, "Ed Kohlwey" <ek...@gmail.com> wrote:
>Hi,
>I'm interested in using Ambari + Ansible to manage a Kerberized cluster.
>The current contemplation is to use an Ansible module to make rest calls
>to
>the Ambari server in order to configure the nodes.
>
>We would also like to be able to automatically set up HA and secure mode.
>
>We have scripts currently for setting up a primary and secondary KDC and
>issuing keytabs, so that is not a big deal, but I recall that the API
>currently has no official support for setting up secure mode or for
>deploying HA.
>
>Can we use the undocumented API's to do this successfully? How future
>proof
>is that approach? I noticed that there has been a lot of work on setting
>up
>a cluster KDC in version 2.
>
>Can any of the devs working on this suggest a good path forward to
>minimize
>rework after the next release?