You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Jon Maron <jm...@hortonworks.com> on 2013/08/14 21:57:09 UTC
HDP Cluster scaling
Hi,
I'm working on the Savanna HDP plugin, currently working on the capability to scale a cluster (add hosts, remove hosts). The current approach to adding a host is to:
1) Install the ambari agent on the host
2) start the ambari agent
3) register the host
4) add the required host components (based on the node group to which it is being added)
5) install components (transition state from INIT to INSTALLED)
6) start the components (transition state from INSTALLED to STARTED)
I have a couple of questions regarding this approach:
1) Is it the ideal approach? Is there a more direct way of interacting with ambari to provision a new host?
2) I am having an issue with the final step. During the invocation I get the following error:
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 Invalid desired state for a client component</title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /api/v1/clusters/my-cluster/hosts/my-cluster-slave-003.novalocal/host_components. Reason:
<pre> Invalid desired state for a client component</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
The mix of components that are in the installed state is:
{
"href" : "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-003.novalocal/host_components?HostRoles/state=INSTALLED",
"items" : [
{
"href" : "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-003.novalocal/host_components/DATANODE",
"HostRoles" : {
"cluster_name" : "my-cluster",
"component_name" : "DATANODE",
"state" : "INSTALLED",
"host_name" : "my-cluster-slave-003.novalocal"
},
"host" : {
"href" : "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-003.novalocal"
}
},
{
"href" : "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-003.novalocal/host_components/MAPREDUCE_CLIENT",
"HostRoles" : {
"cluster_name" : "my-cluster",
"component_name" : "MAPREDUCE_CLIENT",
"state" : "INSTALLED",
"host_name" : "my-cluster-slave-003.novalocal"
},
"host" : {
"href" : "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-003.novalocal"
}
},
{
"href" : "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-003.novalocal/host_components/HDFS_CLIENT",
"HostRoles" : {
"cluster_name" : "my-cluster",
"component_name" : "HDFS_CLIENT",
"state" : "INSTALLED",
"host_name" : "my-cluster-slave-003.novalocal"
},
"host" : {
"href" : "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-003.novalocal"
}
},
{
"href" : "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-003.novalocal/host_components/GANGLIA_MONITOR",
"HostRoles" : {
"cluster_name" : "my-cluster",
"component_name" : "GANGLIA_MONITOR",
"state" : "INSTALLED",
"host_name" : "my-cluster-slave-003.novalocal"
},
"host" : {
"href" : "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-003.novalocal"
}
},
{
"href" : "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-003.novalocal/host_components/TASKTRACKER",
"HostRoles" : {
"cluster_name" : "my-cluster",
"component_name" : "TASKTRACKER",
"state" : "INSTALLED",
"host_name" : "my-cluster-slave-003.novalocal"
},
"host" : {
"href" : "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-003.novalocal"
}
}
]
}
So it appears that "CLIENT" components are not transitioned into the STARTED state? Do I need to iterate thru these and avoid starting components with CLIENT in their name?
-- Jon
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: HDP Cluster scaling
Posted by Sumit Mohanty <sm...@hortonworks.com>.
Host delete is not supported yet. The feature is available in trunk for a
subsequent release.
So there is no way to disassociate a host from a cluster using the APIs.
On 8/14/13 4:59 PM, "Jon Maron" <jm...@hortonworks.com> wrote:
>
>On Aug 14, 2013, at 6:56 PM, Sumit Mohanty <sm...@hortonworks.com>
>wrote:
>
>> Unfortunately, host delete feature is not yet available. Likely,
>> availability is after 1.2.5.
>>
>> The expectation is that host can be deleted after the mapped components
>> are deleted.
>
>I experimented by leveraging the REST API to transition the host
>components on a given host to INSTALLED and subsequently MAINTENANCE
>state. I was then able to actually delete those host components (except
>for client components - they could not be transitioned to MAINTENANCE or
>deleted). However, I can't seem to DELETE the associated host resource.
>Is there a way to remove a host currently? Failing that, is there a way
>to dissociate the host from the cluster?
>
>>
>> -Sumit
>>
>> On 8/14/13 3:50 PM, "Jon Maron" <jm...@hortonworks.com> wrote:
>>
>>> Thanks!
>>>
>>> Follow up question:
>>>
>>> To scale down, do I simply have to leverage the REST API to DELETE the
>>> given host?
>>>
>>> -- Jon
>>>
>>> On Aug 14, 2013, at 5:30 PM, Sumit Mohanty <sm...@hortonworks.com>
>>> wrote:
>>>
>>>> 1) Is it the ideal approach? Is there a more direct way of
>>>>interacting
>>>> with ambari to provision a new host?
>>>>
>>>> The steps you have is the ideal approach. Alternatively, there is
>>>> bootstrap API but it will also require Ambari to have SSH access. Even
>>>> then the number of steps will not be smaller.
>>>>
>>>> 2) I am having an issue with the final step. During the invocation I
>>>> get the following error:
>>>>
>>>>
>>>> You are right. The client components do not transition to STARTED. So
>>>> you
>>>> need to only issue INSTALLED-->STARTED for the non client components.
>>>>
>>>> -Sumit
>>>>
>>>> On 8/14/13 12:57 PM, "Jon Maron" <jm...@hortonworks.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I'm working on the Savanna HDP plugin, currently working on the
>>>>> capability to scale a cluster (add hosts, remove hosts). The current
>>>>> approach to adding a host is to:
>>>>>
>>>>> 1) Install the ambari agent on the host
>>>>> 2) start the ambari agent
>>>>> 3) register the host
>>>>> 4) add the required host components (based on the node group to
>>>>>which
>>>>> it is being added)
>>>>> 5) install components (transition state from INIT to INSTALLED)
>>>>> 6) start the components (transition state from INSTALLED to STARTED)
>>>>>
>>>>> I have a couple of questions regarding this approach:
>>>>>
>>>>> 1) Is it the ideal approach? Is there a more direct way of
>>>>> interacting with ambari to provision a new host?
>>>>>
>>>>> 2) I am having an issue with the final step. During the invocation
>>>>>I
>>>>> get the following error:
>>>>>
>>>>> <html>
>>>>> <head>
>>>>> <meta http-equiv="Content-Type"
>>>>> content="text/html;charset=ISO-8859-1"/>
>>>>> <title>Error 500 Invalid desired state for a client component</title>
>>>>> </head>
>>>>> <body>
>>>>> <h2>HTTP ERROR: 500</h2>
>>>>> <p>Problem accessing
>>>>>
>>>>>
>>>>>/api/v1/clusters/my-cluster/hosts/my-cluster-slave-003.novalocal/host_
>>>>>co
>>>>> mp
>>>>> onents. Reason:
>>>>> <pre> Invalid desired state for a client component</pre></p>
>>>>> <hr /><i><small>Powered by Jetty://</small></i>
>>>>>
>>>>> The mix of components that are in the installed state is:
>>>>>
>>>>> {
>>>>> "href" :
>>>>>
>>>>>
>>>>>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-sl
>>>>>av
>>>>> e-
>>>>> 003.novalocal/host_components?HostRoles/state=INSTALLED",
>>>>> "items" : [
>>>>> {
>>>>> "href" :
>>>>>
>>>>>
>>>>>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-sl
>>>>>av
>>>>> e-
>>>>> 003.novalocal/host_components/DATANODE",
>>>>> "HostRoles" : {
>>>>> "cluster_name" : "my-cluster",
>>>>> "component_name" : "DATANODE",
>>>>> "state" : "INSTALLED",
>>>>> "host_name" : "my-cluster-slave-003.novalocal"
>>>>> },
>>>>> "host" : {
>>>>> "href" :
>>>>>
>>>>>
>>>>>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-sl
>>>>>av
>>>>> e-
>>>>> 003.novalocal"
>>>>> }
>>>>> },
>>>>> {
>>>>> "href" :
>>>>>
>>>>>
>>>>>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-sl
>>>>>av
>>>>> e-
>>>>> 003.novalocal/host_components/MAPREDUCE_CLIENT",
>>>>> "HostRoles" : {
>>>>> "cluster_name" : "my-cluster",
>>>>> "component_name" : "MAPREDUCE_CLIENT",
>>>>> "state" : "INSTALLED",
>>>>> "host_name" : "my-cluster-slave-003.novalocal"
>>>>> },
>>>>> "host" : {
>>>>> "href" :
>>>>>
>>>>>
>>>>>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-sl
>>>>>av
>>>>> e-
>>>>> 003.novalocal"
>>>>> }
>>>>> },
>>>>> {
>>>>> "href" :
>>>>>
>>>>>
>>>>>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-sl
>>>>>av
>>>>> e-
>>>>> 003.novalocal/host_components/HDFS_CLIENT",
>>>>> "HostRoles" : {
>>>>> "cluster_name" : "my-cluster",
>>>>> "component_name" : "HDFS_CLIENT",
>>>>> "state" : "INSTALLED",
>>>>> "host_name" : "my-cluster-slave-003.novalocal"
>>>>> },
>>>>> "host" : {
>>>>> "href" :
>>>>>
>>>>>
>>>>>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-sl
>>>>>av
>>>>> e-
>>>>> 003.novalocal"
>>>>> }
>>>>> },
>>>>> {
>>>>> "href" :
>>>>>
>>>>>
>>>>>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-sl
>>>>>av
>>>>> e-
>>>>> 003.novalocal/host_components/GANGLIA_MONITOR",
>>>>> "HostRoles" : {
>>>>> "cluster_name" : "my-cluster",
>>>>> "component_name" : "GANGLIA_MONITOR",
>>>>> "state" : "INSTALLED",
>>>>> "host_name" : "my-cluster-slave-003.novalocal"
>>>>> },
>>>>> "host" : {
>>>>> "href" :
>>>>>
>>>>>
>>>>>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-sl
>>>>>av
>>>>> e-
>>>>> 003.novalocal"
>>>>> }
>>>>> },
>>>>> {
>>>>> "href" :
>>>>>
>>>>>
>>>>>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-sl
>>>>>av
>>>>> e-
>>>>> 003.novalocal/host_components/TASKTRACKER",
>>>>> "HostRoles" : {
>>>>> "cluster_name" : "my-cluster",
>>>>> "component_name" : "TASKTRACKER",
>>>>> "state" : "INSTALLED",
>>>>> "host_name" : "my-cluster-slave-003.novalocal"
>>>>> },
>>>>> "host" : {
>>>>> "href" :
>>>>>
>>>>>
>>>>>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-sl
>>>>>av
>>>>> e-
>>>>> 003.novalocal"
>>>>> }
>>>>> }
>>>>> ]
>>>>> }
>>>>>
>>>>> So it appears that "CLIENT" components are not transitioned into the
>>>>> STARTED state? Do I need to iterate thru these and avoid starting
>>>>> components with CLIENT in their name?
>>>>>
>>>>> -- Jon
>>>>> --
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>>>> entity
>>>>> to
>>>>> which it is addressed and may contain information that is
>>>>>confidential,
>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>> reader
>>>>> of this message is not the intended recipient, you are hereby
>>>>>notified
>>>>> that
>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>> received this communication in error, please contact the sender
>>>>> immediately
>>>>> and delete it from your system. Thank You.
>>>>
>>>>
>>>>
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to
>>>> which it is addressed and may contain information that is
>>>>confidential,
>>>> privileged and exempt from disclosure under applicable law. If the
>>>> reader
>>>> of this message is not the intended recipient, you are hereby notified
>>>> that
>>>> any printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you have
>>>> received this communication in error, please contact the sender
>>>> immediately
>>>> and delete it from your system. Thank You.
>>>
>>>
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or
>>>entity
>>> to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the
>>>reader
>>> of this message is not the intended recipient, you are hereby notified
>>> that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender
>>> immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or
>>entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the
>>reader
>> of this message is not the intended recipient, you are hereby notified
>>that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender
>>immediately
>> and delete it from your system. Thank You.
>
>
>--
>CONFIDENTIALITY NOTICE
>NOTICE: This message is intended for the use of the individual or entity
>to
>which it is addressed and may contain information that is confidential,
>privileged and exempt from disclosure under applicable law. If the reader
>of this message is not the intended recipient, you are hereby notified
>that
>any printing, copying, dissemination, distribution, disclosure or
>forwarding of this communication is strictly prohibited. If you have
>received this communication in error, please contact the sender
>immediately
>and delete it from your system. Thank You.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: HDP Cluster scaling
Posted by Jon Maron <jm...@hortonworks.com>.
On Aug 14, 2013, at 6:56 PM, Sumit Mohanty <sm...@hortonworks.com> wrote:
> Unfortunately, host delete feature is not yet available. Likely,
> availability is after 1.2.5.
>
> The expectation is that host can be deleted after the mapped components
> are deleted.
I experimented by leveraging the REST API to transition the host components on a given host to INSTALLED and subsequently MAINTENANCE state. I was then able to actually delete those host components (except for client components - they could not be transitioned to MAINTENANCE or deleted). However, I can't seem to DELETE the associated host resource. Is there a way to remove a host currently? Failing that, is there a way to dissociate the host from the cluster?
>
> -Sumit
>
> On 8/14/13 3:50 PM, "Jon Maron" <jm...@hortonworks.com> wrote:
>
>> Thanks!
>>
>> Follow up question:
>>
>> To scale down, do I simply have to leverage the REST API to DELETE the
>> given host?
>>
>> -- Jon
>>
>> On Aug 14, 2013, at 5:30 PM, Sumit Mohanty <sm...@hortonworks.com>
>> wrote:
>>
>>> 1) Is it the ideal approach? Is there a more direct way of interacting
>>> with ambari to provision a new host?
>>>
>>> The steps you have is the ideal approach. Alternatively, there is
>>> bootstrap API but it will also require Ambari to have SSH access. Even
>>> then the number of steps will not be smaller.
>>>
>>> 2) I am having an issue with the final step. During the invocation I
>>> get the following error:
>>>
>>>
>>> You are right. The client components do not transition to STARTED. So
>>> you
>>> need to only issue INSTALLED-->STARTED for the non client components.
>>>
>>> -Sumit
>>>
>>> On 8/14/13 12:57 PM, "Jon Maron" <jm...@hortonworks.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm working on the Savanna HDP plugin, currently working on the
>>>> capability to scale a cluster (add hosts, remove hosts). The current
>>>> approach to adding a host is to:
>>>>
>>>> 1) Install the ambari agent on the host
>>>> 2) start the ambari agent
>>>> 3) register the host
>>>> 4) add the required host components (based on the node group to which
>>>> it is being added)
>>>> 5) install components (transition state from INIT to INSTALLED)
>>>> 6) start the components (transition state from INSTALLED to STARTED)
>>>>
>>>> I have a couple of questions regarding this approach:
>>>>
>>>> 1) Is it the ideal approach? Is there a more direct way of
>>>> interacting with ambari to provision a new host?
>>>>
>>>> 2) I am having an issue with the final step. During the invocation I
>>>> get the following error:
>>>>
>>>> <html>
>>>> <head>
>>>> <meta http-equiv="Content-Type"
>>>> content="text/html;charset=ISO-8859-1"/>
>>>> <title>Error 500 Invalid desired state for a client component</title>
>>>> </head>
>>>> <body>
>>>> <h2>HTTP ERROR: 500</h2>
>>>> <p>Problem accessing
>>>>
>>>> /api/v1/clusters/my-cluster/hosts/my-cluster-slave-003.novalocal/host_co
>>>> mp
>>>> onents. Reason:
>>>> <pre> Invalid desired state for a client component</pre></p>
>>>> <hr /><i><small>Powered by Jetty://</small></i>
>>>>
>>>> The mix of components that are in the installed state is:
>>>>
>>>> {
>>>> "href" :
>>>>
>>>> "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slav
>>>> e-
>>>> 003.novalocal/host_components?HostRoles/state=INSTALLED",
>>>> "items" : [
>>>> {
>>>> "href" :
>>>>
>>>> "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slav
>>>> e-
>>>> 003.novalocal/host_components/DATANODE",
>>>> "HostRoles" : {
>>>> "cluster_name" : "my-cluster",
>>>> "component_name" : "DATANODE",
>>>> "state" : "INSTALLED",
>>>> "host_name" : "my-cluster-slave-003.novalocal"
>>>> },
>>>> "host" : {
>>>> "href" :
>>>>
>>>> "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slav
>>>> e-
>>>> 003.novalocal"
>>>> }
>>>> },
>>>> {
>>>> "href" :
>>>>
>>>> "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slav
>>>> e-
>>>> 003.novalocal/host_components/MAPREDUCE_CLIENT",
>>>> "HostRoles" : {
>>>> "cluster_name" : "my-cluster",
>>>> "component_name" : "MAPREDUCE_CLIENT",
>>>> "state" : "INSTALLED",
>>>> "host_name" : "my-cluster-slave-003.novalocal"
>>>> },
>>>> "host" : {
>>>> "href" :
>>>>
>>>> "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slav
>>>> e-
>>>> 003.novalocal"
>>>> }
>>>> },
>>>> {
>>>> "href" :
>>>>
>>>> "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slav
>>>> e-
>>>> 003.novalocal/host_components/HDFS_CLIENT",
>>>> "HostRoles" : {
>>>> "cluster_name" : "my-cluster",
>>>> "component_name" : "HDFS_CLIENT",
>>>> "state" : "INSTALLED",
>>>> "host_name" : "my-cluster-slave-003.novalocal"
>>>> },
>>>> "host" : {
>>>> "href" :
>>>>
>>>> "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slav
>>>> e-
>>>> 003.novalocal"
>>>> }
>>>> },
>>>> {
>>>> "href" :
>>>>
>>>> "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slav
>>>> e-
>>>> 003.novalocal/host_components/GANGLIA_MONITOR",
>>>> "HostRoles" : {
>>>> "cluster_name" : "my-cluster",
>>>> "component_name" : "GANGLIA_MONITOR",
>>>> "state" : "INSTALLED",
>>>> "host_name" : "my-cluster-slave-003.novalocal"
>>>> },
>>>> "host" : {
>>>> "href" :
>>>>
>>>> "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slav
>>>> e-
>>>> 003.novalocal"
>>>> }
>>>> },
>>>> {
>>>> "href" :
>>>>
>>>> "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slav
>>>> e-
>>>> 003.novalocal/host_components/TASKTRACKER",
>>>> "HostRoles" : {
>>>> "cluster_name" : "my-cluster",
>>>> "component_name" : "TASKTRACKER",
>>>> "state" : "INSTALLED",
>>>> "host_name" : "my-cluster-slave-003.novalocal"
>>>> },
>>>> "host" : {
>>>> "href" :
>>>>
>>>> "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slav
>>>> e-
>>>> 003.novalocal"
>>>> }
>>>> }
>>>> ]
>>>> }
>>>>
>>>> So it appears that "CLIENT" components are not transitioned into the
>>>> STARTED state? Do I need to iterate thru these and avoid starting
>>>> components with CLIENT in their name?
>>>>
>>>> -- Jon
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity
>>>> to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the
>>>> reader
>>>> of this message is not the intended recipient, you are hereby notified
>>>> that
>>>> any printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you have
>>>> received this communication in error, please contact the sender
>>>> immediately
>>>> and delete it from your system. Thank You.
>>>
>>>
>>>
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or
>>> entity to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the
>>> reader
>>> of this message is not the intended recipient, you are hereby notified
>>> that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender
>>> immediately
>>> and delete it from your system. Thank You.
>>
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified
>> that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender
>> immediately
>> and delete it from your system. Thank You.
>
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: HDP Cluster scaling
Posted by Sumit Mohanty <sm...@hortonworks.com>.
Unfortunately, host delete feature is not yet available. Likely,
availability is after 1.2.5.
The expectation is that host can be deleted after the mapped components
are deleted.
-Sumit
On 8/14/13 3:50 PM, "Jon Maron" <jm...@hortonworks.com> wrote:
>Thanks!
>
>Follow up question:
>
>To scale down, do I simply have to leverage the REST API to DELETE the
>given host?
>
>-- Jon
>
>On Aug 14, 2013, at 5:30 PM, Sumit Mohanty <sm...@hortonworks.com>
>wrote:
>
>> 1) Is it the ideal approach? Is there a more direct way of interacting
>> with ambari to provision a new host?
>>
>> The steps you have is the ideal approach. Alternatively, there is
>> bootstrap API but it will also require Ambari to have SSH access. Even
>> then the number of steps will not be smaller.
>>
>> 2) I am having an issue with the final step. During the invocation I
>> get the following error:
>>
>>
>> You are right. The client components do not transition to STARTED. So
>>you
>> need to only issue INSTALLED-->STARTED for the non client components.
>>
>> -Sumit
>>
>> On 8/14/13 12:57 PM, "Jon Maron" <jm...@hortonworks.com> wrote:
>>
>>> Hi,
>>>
>>> I'm working on the Savanna HDP plugin, currently working on the
>>> capability to scale a cluster (add hosts, remove hosts). The current
>>> approach to adding a host is to:
>>>
>>> 1) Install the ambari agent on the host
>>> 2) start the ambari agent
>>> 3) register the host
>>> 4) add the required host components (based on the node group to which
>>> it is being added)
>>> 5) install components (transition state from INIT to INSTALLED)
>>> 6) start the components (transition state from INSTALLED to STARTED)
>>>
>>> I have a couple of questions regarding this approach:
>>>
>>> 1) Is it the ideal approach? Is there a more direct way of
>>> interacting with ambari to provision a new host?
>>>
>>> 2) I am having an issue with the final step. During the invocation I
>>> get the following error:
>>>
>>> <html>
>>> <head>
>>> <meta http-equiv="Content-Type"
>>>content="text/html;charset=ISO-8859-1"/>
>>> <title>Error 500 Invalid desired state for a client component</title>
>>> </head>
>>> <body>
>>> <h2>HTTP ERROR: 500</h2>
>>> <p>Problem accessing
>>>
>>>/api/v1/clusters/my-cluster/hosts/my-cluster-slave-003.novalocal/host_co
>>>mp
>>> onents. Reason:
>>> <pre> Invalid desired state for a client component</pre></p>
>>> <hr /><i><small>Powered by Jetty://</small></i>
>>>
>>> The mix of components that are in the installed state is:
>>>
>>> {
>>> "href" :
>>>
>>>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slav
>>>e-
>>> 003.novalocal/host_components?HostRoles/state=INSTALLED",
>>> "items" : [
>>> {
>>> "href" :
>>>
>>>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slav
>>>e-
>>> 003.novalocal/host_components/DATANODE",
>>> "HostRoles" : {
>>> "cluster_name" : "my-cluster",
>>> "component_name" : "DATANODE",
>>> "state" : "INSTALLED",
>>> "host_name" : "my-cluster-slave-003.novalocal"
>>> },
>>> "host" : {
>>> "href" :
>>>
>>>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slav
>>>e-
>>> 003.novalocal"
>>> }
>>> },
>>> {
>>> "href" :
>>>
>>>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slav
>>>e-
>>> 003.novalocal/host_components/MAPREDUCE_CLIENT",
>>> "HostRoles" : {
>>> "cluster_name" : "my-cluster",
>>> "component_name" : "MAPREDUCE_CLIENT",
>>> "state" : "INSTALLED",
>>> "host_name" : "my-cluster-slave-003.novalocal"
>>> },
>>> "host" : {
>>> "href" :
>>>
>>>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slav
>>>e-
>>> 003.novalocal"
>>> }
>>> },
>>> {
>>> "href" :
>>>
>>>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slav
>>>e-
>>> 003.novalocal/host_components/HDFS_CLIENT",
>>> "HostRoles" : {
>>> "cluster_name" : "my-cluster",
>>> "component_name" : "HDFS_CLIENT",
>>> "state" : "INSTALLED",
>>> "host_name" : "my-cluster-slave-003.novalocal"
>>> },
>>> "host" : {
>>> "href" :
>>>
>>>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slav
>>>e-
>>> 003.novalocal"
>>> }
>>> },
>>> {
>>> "href" :
>>>
>>>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slav
>>>e-
>>> 003.novalocal/host_components/GANGLIA_MONITOR",
>>> "HostRoles" : {
>>> "cluster_name" : "my-cluster",
>>> "component_name" : "GANGLIA_MONITOR",
>>> "state" : "INSTALLED",
>>> "host_name" : "my-cluster-slave-003.novalocal"
>>> },
>>> "host" : {
>>> "href" :
>>>
>>>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slav
>>>e-
>>> 003.novalocal"
>>> }
>>> },
>>> {
>>> "href" :
>>>
>>>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slav
>>>e-
>>> 003.novalocal/host_components/TASKTRACKER",
>>> "HostRoles" : {
>>> "cluster_name" : "my-cluster",
>>> "component_name" : "TASKTRACKER",
>>> "state" : "INSTALLED",
>>> "host_name" : "my-cluster-slave-003.novalocal"
>>> },
>>> "host" : {
>>> "href" :
>>>
>>>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slav
>>>e-
>>> 003.novalocal"
>>> }
>>> }
>>> ]
>>> }
>>>
>>> So it appears that "CLIENT" components are not transitioned into the
>>> STARTED state? Do I need to iterate thru these and avoid starting
>>> components with CLIENT in their name?
>>>
>>> -- Jon
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or
>>>entity
>>> to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the
>>>reader
>>> of this message is not the intended recipient, you are hereby notified
>>> that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender
>>> immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or
>>entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the
>>reader
>> of this message is not the intended recipient, you are hereby notified
>>that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender
>>immediately
>> and delete it from your system. Thank You.
>
>
>--
>CONFIDENTIALITY NOTICE
>NOTICE: This message is intended for the use of the individual or entity
>to
>which it is addressed and may contain information that is confidential,
>privileged and exempt from disclosure under applicable law. If the reader
>of this message is not the intended recipient, you are hereby notified
>that
>any printing, copying, dissemination, distribution, disclosure or
>forwarding of this communication is strictly prohibited. If you have
>received this communication in error, please contact the sender
>immediately
>and delete it from your system. Thank You.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: HDP Cluster scaling
Posted by Jon Maron <jm...@hortonworks.com>.
Thanks!
Follow up question:
To scale down, do I simply have to leverage the REST API to DELETE the given host?
-- Jon
On Aug 14, 2013, at 5:30 PM, Sumit Mohanty <sm...@hortonworks.com> wrote:
> 1) Is it the ideal approach? Is there a more direct way of interacting
> with ambari to provision a new host?
>
> The steps you have is the ideal approach. Alternatively, there is
> bootstrap API but it will also require Ambari to have SSH access. Even
> then the number of steps will not be smaller.
>
> 2) I am having an issue with the final step. During the invocation I
> get the following error:
>
>
> You are right. The client components do not transition to STARTED. So you
> need to only issue INSTALLED-->STARTED for the non client components.
>
> -Sumit
>
> On 8/14/13 12:57 PM, "Jon Maron" <jm...@hortonworks.com> wrote:
>
>> Hi,
>>
>> I'm working on the Savanna HDP plugin, currently working on the
>> capability to scale a cluster (add hosts, remove hosts). The current
>> approach to adding a host is to:
>>
>> 1) Install the ambari agent on the host
>> 2) start the ambari agent
>> 3) register the host
>> 4) add the required host components (based on the node group to which
>> it is being added)
>> 5) install components (transition state from INIT to INSTALLED)
>> 6) start the components (transition state from INSTALLED to STARTED)
>>
>> I have a couple of questions regarding this approach:
>>
>> 1) Is it the ideal approach? Is there a more direct way of
>> interacting with ambari to provision a new host?
>>
>> 2) I am having an issue with the final step. During the invocation I
>> get the following error:
>>
>> <html>
>> <head>
>> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
>> <title>Error 500 Invalid desired state for a client component</title>
>> </head>
>> <body>
>> <h2>HTTP ERROR: 500</h2>
>> <p>Problem accessing
>> /api/v1/clusters/my-cluster/hosts/my-cluster-slave-003.novalocal/host_comp
>> onents. Reason:
>> <pre> Invalid desired state for a client component</pre></p>
>> <hr /><i><small>Powered by Jetty://</small></i>
>>
>> The mix of components that are in the installed state is:
>>
>> {
>> "href" :
>> "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-
>> 003.novalocal/host_components?HostRoles/state=INSTALLED",
>> "items" : [
>> {
>> "href" :
>> "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-
>> 003.novalocal/host_components/DATANODE",
>> "HostRoles" : {
>> "cluster_name" : "my-cluster",
>> "component_name" : "DATANODE",
>> "state" : "INSTALLED",
>> "host_name" : "my-cluster-slave-003.novalocal"
>> },
>> "host" : {
>> "href" :
>> "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-
>> 003.novalocal"
>> }
>> },
>> {
>> "href" :
>> "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-
>> 003.novalocal/host_components/MAPREDUCE_CLIENT",
>> "HostRoles" : {
>> "cluster_name" : "my-cluster",
>> "component_name" : "MAPREDUCE_CLIENT",
>> "state" : "INSTALLED",
>> "host_name" : "my-cluster-slave-003.novalocal"
>> },
>> "host" : {
>> "href" :
>> "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-
>> 003.novalocal"
>> }
>> },
>> {
>> "href" :
>> "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-
>> 003.novalocal/host_components/HDFS_CLIENT",
>> "HostRoles" : {
>> "cluster_name" : "my-cluster",
>> "component_name" : "HDFS_CLIENT",
>> "state" : "INSTALLED",
>> "host_name" : "my-cluster-slave-003.novalocal"
>> },
>> "host" : {
>> "href" :
>> "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-
>> 003.novalocal"
>> }
>> },
>> {
>> "href" :
>> "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-
>> 003.novalocal/host_components/GANGLIA_MONITOR",
>> "HostRoles" : {
>> "cluster_name" : "my-cluster",
>> "component_name" : "GANGLIA_MONITOR",
>> "state" : "INSTALLED",
>> "host_name" : "my-cluster-slave-003.novalocal"
>> },
>> "host" : {
>> "href" :
>> "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-
>> 003.novalocal"
>> }
>> },
>> {
>> "href" :
>> "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-
>> 003.novalocal/host_components/TASKTRACKER",
>> "HostRoles" : {
>> "cluster_name" : "my-cluster",
>> "component_name" : "TASKTRACKER",
>> "state" : "INSTALLED",
>> "host_name" : "my-cluster-slave-003.novalocal"
>> },
>> "host" : {
>> "href" :
>> "http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-
>> 003.novalocal"
>> }
>> }
>> ]
>> }
>>
>> So it appears that "CLIENT" components are not transitioned into the
>> STARTED state? Do I need to iterate thru these and avoid starting
>> components with CLIENT in their name?
>>
>> -- Jon
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified
>> that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender
>> immediately
>> and delete it from your system. Thank You.
>
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: HDP Cluster scaling
Posted by Sumit Mohanty <sm...@hortonworks.com>.
1) Is it the ideal approach? Is there a more direct way of interacting
with ambari to provision a new host?
The steps you have is the ideal approach. Alternatively, there is
bootstrap API but it will also require Ambari to have SSH access. Even
then the number of steps will not be smaller.
2) I am having an issue with the final step. During the invocation I
get the following error:
You are right. The client components do not transition to STARTED. So you
need to only issue INSTALLED-->STARTED for the non client components.
-Sumit
On 8/14/13 12:57 PM, "Jon Maron" <jm...@hortonworks.com> wrote:
>Hi,
>
> I'm working on the Savanna HDP plugin, currently working on the
>capability to scale a cluster (add hosts, remove hosts). The current
>approach to adding a host is to:
>
> 1) Install the ambari agent on the host
> 2) start the ambari agent
> 3) register the host
> 4) add the required host components (based on the node group to which
>it is being added)
> 5) install components (transition state from INIT to INSTALLED)
> 6) start the components (transition state from INSTALLED to STARTED)
>
> I have a couple of questions regarding this approach:
>
> 1) Is it the ideal approach? Is there a more direct way of
>interacting with ambari to provision a new host?
>
> 2) I am having an issue with the final step. During the invocation I
>get the following error:
>
><html>
><head>
><meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
><title>Error 500 Invalid desired state for a client component</title>
></head>
><body>
><h2>HTTP ERROR: 500</h2>
><p>Problem accessing
>/api/v1/clusters/my-cluster/hosts/my-cluster-slave-003.novalocal/host_comp
>onents. Reason:
><pre> Invalid desired state for a client component</pre></p>
><hr /><i><small>Powered by Jetty://</small></i>
>
> The mix of components that are in the installed state is:
>
>{
> "href" :
>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-
>003.novalocal/host_components?HostRoles/state=INSTALLED",
> "items" : [
> {
> "href" :
>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-
>003.novalocal/host_components/DATANODE",
> "HostRoles" : {
> "cluster_name" : "my-cluster",
> "component_name" : "DATANODE",
> "state" : "INSTALLED",
> "host_name" : "my-cluster-slave-003.novalocal"
> },
> "host" : {
> "href" :
>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-
>003.novalocal"
> }
> },
> {
> "href" :
>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-
>003.novalocal/host_components/MAPREDUCE_CLIENT",
> "HostRoles" : {
> "cluster_name" : "my-cluster",
> "component_name" : "MAPREDUCE_CLIENT",
> "state" : "INSTALLED",
> "host_name" : "my-cluster-slave-003.novalocal"
> },
> "host" : {
> "href" :
>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-
>003.novalocal"
> }
> },
> {
> "href" :
>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-
>003.novalocal/host_components/HDFS_CLIENT",
> "HostRoles" : {
> "cluster_name" : "my-cluster",
> "component_name" : "HDFS_CLIENT",
> "state" : "INSTALLED",
> "host_name" : "my-cluster-slave-003.novalocal"
> },
> "host" : {
> "href" :
>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-
>003.novalocal"
> }
> },
> {
> "href" :
>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-
>003.novalocal/host_components/GANGLIA_MONITOR",
> "HostRoles" : {
> "cluster_name" : "my-cluster",
> "component_name" : "GANGLIA_MONITOR",
> "state" : "INSTALLED",
> "host_name" : "my-cluster-slave-003.novalocal"
> },
> "host" : {
> "href" :
>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-
>003.novalocal"
> }
> },
> {
> "href" :
>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-
>003.novalocal/host_components/TASKTRACKER",
> "HostRoles" : {
> "cluster_name" : "my-cluster",
> "component_name" : "TASKTRACKER",
> "state" : "INSTALLED",
> "host_name" : "my-cluster-slave-003.novalocal"
> },
> "host" : {
> "href" :
>"http://172.18.3.5:8080/api/v1/clusters/my-cluster/hosts/my-cluster-slave-
>003.novalocal"
> }
> }
> ]
>}
>
> So it appears that "CLIENT" components are not transitioned into the
>STARTED state? Do I need to iterate thru these and avoid starting
>components with CLIENT in their name?
>
>-- Jon
>--
>CONFIDENTIALITY NOTICE
>NOTICE: This message is intended for the use of the individual or entity
>to
>which it is addressed and may contain information that is confidential,
>privileged and exempt from disclosure under applicable law. If the reader
>of this message is not the intended recipient, you are hereby notified
>that
>any printing, copying, dissemination, distribution, disclosure or
>forwarding of this communication is strictly prohibited. If you have
>received this communication in error, please contact the sender
>immediately
>and delete it from your system. Thank You.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.