You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ambari.apache.org by Alejandro Fernandez <af...@hortonworks.com> on 2016/01/04 19:13:36 UTC

Re: Services' versions defined in stack ignored when provisioning cluster via blueprint

Hi Michal,

Which version of Ambari are you on?

In AMBARI-13830<https://issues.apache.org/jira/browse/AMBARI-13830> (on trunk), Spark's metainfo.xml file contains,
          <name>SPARK</name>
          <version>1.5.2.2.3</version>
          <extends>common-services/SPARK/1.4.1.2.3</extends>

It may very well be possible that the value is changed depending on which version of HDP 2.3 you install.

Thanks,
Alejandro

From: Michal Siemaszko <mi...@vnomic.com>>
Reply-To: "user@ambari.apache.org<ma...@ambari.apache.org>" <us...@ambari.apache.org>>
Date: Monday, January 4, 2016 at 8:34 AM
To: "user@ambari.apache.org<ma...@ambari.apache.org>" <us...@ambari.apache.org>>
Subject: Services' versions defined in stack ignored when provisioning cluster via blueprint


Hi,

For a project where creation of Hadoop clusters needs to be fully automated, I utilized Ambari's blueprints and auto-discovery features, in addition to bootstraping hosts via REST instead of GUI (so the manual host registration is not necessary anymore).

I run into an issue where versions of services defined in stack used are ignored during cluster provisioning. For example, I specify "stack_name" as "HDP" and "stack_version" as "2.3" in blueprint I use, and "Spark" version associated with that stack is "1.4.1.2.3". However, once cluster is provisioned, even though http://[AmbariServerNode]:8080/#/main/admin/stack/services also shows "Spark" version "1.4.1.2.3" installed, when submitting a sample Spark job via Spark shell on client node, "Spark" version "1.5.2" is shown; `/usr/hdp/2.3.4.0-3485/spark/RELEASE` file on that node also shows "Spark" version "1.5.2.2.3.4.0".

I got around this issue by changing the base URL for repository used to 'http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.2.0' (instead of "http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.4.0") prior to provisioning cluster from blueprint.

I'm wondering whether this is something expected or if it's a bug, i.e. that service version defined in stack is ignored unless such change is made. Shouldn't services' versions defined in stack be respected and applied regardless of base URL? Perhaps I'm missing some other settings that should be applied in blueprint or cluster creation template, but AFAIK it's all as per docs/examples I read.

Thank you for your input.

Regards,
Michal Siemaszko

Re: Services' versions defined in stack ignored when provisioning cluster via blueprint

Posted by Michal Siemaszko <mi...@vnomic.com>.
Hi Sumit,

Since the entire cluster provisioning process must be fully automated, I had to find "a better way", i.e. using Ambari's stack's REST endpoint, I issue (an equivalent of)


$curl -v -i -X PUT --user admin:admin -H "X-Requested-By: ambari" --data '{"Repositories":{"base_url":"http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.2.0","verify_base_url" : false}}' http://[AmbariServerNode]:8080/api/v1/stacks/HDP/versions/2.3/operating_systems/redhat6/repositories/HDP-2.3

(i.e. what I mentioned earlier about changing the base URL for repository used prior to provisioning cluster from blueprint).

So, it seems it's not a bug (you are aware of this). Then stack definition (http://[AmbariServerNode]:8080/#/main/admin/stack/services) for selected stack (i.e. HDP 2.3) cannot be taken as "single source of truth", as there might be other settings (as it turned out) that affect which version of given service is actually installed.

Thank you for your input.

Regards,
Michal



________________________________
From: Sumit Mohanty <sm...@hortonworks.com>
Sent: Monday, January 4, 2016 8:44 PM
To: Michal Siemaszko; Alejandro Fernandez; user@ambari.apache.org; Robert Nettleton
Subject: Re: Services' versions defined in stack ignored when provisioning cluster via blueprint


One way is to modify the repoinfo.xml file for HDP-2.3 to point to the desired repo URL and then modifying

  <latest>http://s3.amazonaws.com/dev.hortonworks.com/HDP/hdp_urlinfo.json</latest>​

to point to an invalid file (e.g. invalid_hdp_urlinfo.json)


And restart Ambari.


There might be a better way.

________________________________
From: Michal Siemaszko <mi...@vnomic.com>
Sent: Monday, January 04, 2016 11:12 AM
To: Alejandro Fernandez; user@ambari.apache.org; Sumit Mohanty
Subject: Re: Services' versions defined in stack ignored when provisioning cluster via blueprint


Hi Alejandro,

We use Ambari 2.1.2.

Blueprint I use to create cluster is pointed to stack "HDP", version "2.3". Definition for that stack viewed via Ambari GUI (i.e. http://[AmbariServerNode]:8080/#/main/admin/stack/services) has Spark v. "1.4.1.2.3" in it.

Since we use some additional services in that cluster that are only tested for certain versions of Spark, we need to make sure what's in stack definition is exactly what is being installed. I did get around this problem by modifying base URL, but I'm not sure how stack definition should be viewed then, if this info can be changed somewhere else (in that metainfo.xml file you mentioned) without this being reflected in Ambari GUI.

Perhaps in that blueprint we should specify stack release version as well? I.e. instead of "2.3", use "2.3.2" (so it doesn't automatically assume latest release version, 2.3.4 in that case)?

Thank you very much for your answer - we wanted to understand how this is supposed to work, perhaps this is deliberate, but I've used stack definition so far to verify services versions to be installed.


Regards,

Michal


________________________________
From: Alejandro Fernandez <af...@hortonworks.com>
Sent: Monday, January 4, 2016 7:13 PM
To: user@ambari.apache.org; Sumit Mohanty; Michal Siemaszko
Subject: Re: Services' versions defined in stack ignored when provisioning cluster via blueprint

Hi Michal,

Which version of Ambari are you on?

In AMBARI-13830<https://issues.apache.org/jira/browse/AMBARI-13830> (on trunk), Spark's metainfo.xml file contains,
          <name>SPARK</name>
          <version>1.5.2.2.3</version>
          <extends>common-services/SPARK/1.4.1.2.3</extends>

It may very well be possible that the value is changed depending on which version of HDP 2.3 you install.

Thanks,
Alejandro

From: Michal Siemaszko <mi...@vnomic.com>>
Reply-To: "user@ambari.apache.org<ma...@ambari.apache.org>" <us...@ambari.apache.org>>
Date: Monday, January 4, 2016 at 8:34 AM
To: "user@ambari.apache.org<ma...@ambari.apache.org>" <us...@ambari.apache.org>>
Subject: Services' versions defined in stack ignored when provisioning cluster via blueprint


Hi,

For a project where creation of Hadoop clusters needs to be fully automated, I utilized Ambari's blueprints and auto-discovery features, in addition to bootstraping hosts via REST instead of GUI (so the manual host registration is not necessary anymore).

I run into an issue where versions of services defined in stack used are ignored during cluster provisioning. For example, I specify "stack_name" as "HDP" and "stack_version" as "2.3" in blueprint I use, and "Spark" version associated with that stack is "1.4.1.2.3". However, once cluster is provisioned, even though http://[AmbariServerNode]:8080/#/main/admin/stack/services also shows "Spark" version "1.4.1.2.3" installed, when submitting a sample Spark job via Spark shell on client node, "Spark" version "1.5.2" is shown; `/usr/hdp/2.3.4.0-3485/spark/RELEASE` file on that node also shows "Spark" version "1.5.2.2.3.4.0".

I got around this issue by changing the base URL for repository used to 'http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.2.0' (instead of "http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.4.0") prior to provisioning cluster from blueprint.

I'm wondering whether this is something expected or if it's a bug, i.e. that service version defined in stack is ignored unless such change is made. Shouldn't services' versions defined in stack be respected and applied regardless of base URL? Perhaps I'm missing some other settings that should be applied in blueprint or cluster creation template, but AFAIK it's all as per docs/examples I read.

Thank you for your input.

Regards,
Michal Siemaszko

Re: Services' versions defined in stack ignored when provisioning cluster via blueprint

Posted by Sumit Mohanty <sm...@hortonworks.com>.
One way is to modify the repoinfo.xml file for HDP-2.3 to point to the desired repo URL and then modifying

  <latest>http://s3.amazonaws.com/dev.hortonworks.com/HDP/hdp_urlinfo.json</latest>​

to point to an invalid file (e.g. invalid_hdp_urlinfo.json)


And restart Ambari.


There might be a better way.

________________________________
From: Michal Siemaszko <mi...@vnomic.com>
Sent: Monday, January 04, 2016 11:12 AM
To: Alejandro Fernandez; user@ambari.apache.org; Sumit Mohanty
Subject: Re: Services' versions defined in stack ignored when provisioning cluster via blueprint


Hi Alejandro,

We use Ambari 2.1.2.

Blueprint I use to create cluster is pointed to stack "HDP", version "2.3". Definition for that stack viewed via Ambari GUI (i.e. http://[AmbariServerNode]:8080/#/main/admin/stack/services) has Spark v. "1.4.1.2.3" in it.

Since we use some additional services in that cluster that are only tested for certain versions of Spark, we need to make sure what's in stack definition is exactly what is being installed. I did get around this problem by modifying base URL, but I'm not sure how stack definition should be viewed then, if this info can be changed somewhere else (in that metainfo.xml file you mentioned) without this being reflected in Ambari GUI.

Perhaps in that blueprint we should specify stack release version as well? I.e. instead of "2.3", use "2.3.2" (so it doesn't automatically assume latest release version, 2.3.4 in that case)?

Thank you very much for your answer - we wanted to understand how this is supposed to work, perhaps this is deliberate, but I've used stack definition so far to verify services versions to be installed.


Regards,

Michal


________________________________
From: Alejandro Fernandez <af...@hortonworks.com>
Sent: Monday, January 4, 2016 7:13 PM
To: user@ambari.apache.org; Sumit Mohanty; Michal Siemaszko
Subject: Re: Services' versions defined in stack ignored when provisioning cluster via blueprint

Hi Michal,

Which version of Ambari are you on?

In AMBARI-13830<https://issues.apache.org/jira/browse/AMBARI-13830> (on trunk), Spark's metainfo.xml file contains,
          <name>SPARK</name>
          <version>1.5.2.2.3</version>
          <extends>common-services/SPARK/1.4.1.2.3</extends>

It may very well be possible that the value is changed depending on which version of HDP 2.3 you install.

Thanks,
Alejandro

From: Michal Siemaszko <mi...@vnomic.com>>
Reply-To: "user@ambari.apache.org<ma...@ambari.apache.org>" <us...@ambari.apache.org>>
Date: Monday, January 4, 2016 at 8:34 AM
To: "user@ambari.apache.org<ma...@ambari.apache.org>" <us...@ambari.apache.org>>
Subject: Services' versions defined in stack ignored when provisioning cluster via blueprint


Hi,

For a project where creation of Hadoop clusters needs to be fully automated, I utilized Ambari's blueprints and auto-discovery features, in addition to bootstraping hosts via REST instead of GUI (so the manual host registration is not necessary anymore).

I run into an issue where versions of services defined in stack used are ignored during cluster provisioning. For example, I specify "stack_name" as "HDP" and "stack_version" as "2.3" in blueprint I use, and "Spark" version associated with that stack is "1.4.1.2.3". However, once cluster is provisioned, even though http://[AmbariServerNode]:8080/#/main/admin/stack/services also shows "Spark" version "1.4.1.2.3" installed, when submitting a sample Spark job via Spark shell on client node, "Spark" version "1.5.2" is shown; `/usr/hdp/2.3.4.0-3485/spark/RELEASE` file on that node also shows "Spark" version "1.5.2.2.3.4.0".

I got around this issue by changing the base URL for repository used to 'http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.2.0' (instead of "http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.4.0") prior to provisioning cluster from blueprint.

I'm wondering whether this is something expected or if it's a bug, i.e. that service version defined in stack is ignored unless such change is made. Shouldn't services' versions defined in stack be respected and applied regardless of base URL? Perhaps I'm missing some other settings that should be applied in blueprint or cluster creation template, but AFAIK it's all as per docs/examples I read.

Thank you for your input.

Regards,
Michal Siemaszko

Re: Services' versions defined in stack ignored when provisioning cluster via blueprint

Posted by Michal Siemaszko <mi...@vnomic.com>.
Hi Alejandro,

We use Ambari 2.1.2.

Blueprint I use to create cluster is pointed to stack "HDP", version "2.3". Definition for that stack viewed via Ambari GUI (i.e. http://[AmbariServerNode]:8080/#/main/admin/stack/services) has Spark v. "1.4.1.2.3" in it.

Since we use some additional services in that cluster that are only tested for certain versions of Spark, we need to make sure what's in stack definition is exactly what is being installed. I did get around this problem by modifying base URL, but I'm not sure how stack definition should be viewed then, if this info can be changed somewhere else (in that metainfo.xml file you mentioned) without this being reflected in Ambari GUI.

Perhaps in that blueprint we should specify stack release version as well? I.e. instead of "2.3", use "2.3.2" (so it doesn't automatically assume latest release version, 2.3.4 in that case)?

Thank you very much for your answer - we wanted to understand how this is supposed to work, perhaps this is deliberate, but I've used stack definition so far to verify services versions to be installed.


Regards,

Michal


________________________________
From: Alejandro Fernandez <af...@hortonworks.com>
Sent: Monday, January 4, 2016 7:13 PM
To: user@ambari.apache.org; Sumit Mohanty; Michal Siemaszko
Subject: Re: Services' versions defined in stack ignored when provisioning cluster via blueprint

Hi Michal,

Which version of Ambari are you on?

In AMBARI-13830<https://issues.apache.org/jira/browse/AMBARI-13830> (on trunk), Spark's metainfo.xml file contains,
          <name>SPARK</name>
          <version>1.5.2.2.3</version>
          <extends>common-services/SPARK/1.4.1.2.3</extends>

It may very well be possible that the value is changed depending on which version of HDP 2.3 you install.

Thanks,
Alejandro

From: Michal Siemaszko <mi...@vnomic.com>>
Reply-To: "user@ambari.apache.org<ma...@ambari.apache.org>" <us...@ambari.apache.org>>
Date: Monday, January 4, 2016 at 8:34 AM
To: "user@ambari.apache.org<ma...@ambari.apache.org>" <us...@ambari.apache.org>>
Subject: Services' versions defined in stack ignored when provisioning cluster via blueprint


Hi,

For a project where creation of Hadoop clusters needs to be fully automated, I utilized Ambari's blueprints and auto-discovery features, in addition to bootstraping hosts via REST instead of GUI (so the manual host registration is not necessary anymore).

I run into an issue where versions of services defined in stack used are ignored during cluster provisioning. For example, I specify "stack_name" as "HDP" and "stack_version" as "2.3" in blueprint I use, and "Spark" version associated with that stack is "1.4.1.2.3". However, once cluster is provisioned, even though http://[AmbariServerNode]:8080/#/main/admin/stack/services also shows "Spark" version "1.4.1.2.3" installed, when submitting a sample Spark job via Spark shell on client node, "Spark" version "1.5.2" is shown; `/usr/hdp/2.3.4.0-3485/spark/RELEASE` file on that node also shows "Spark" version "1.5.2.2.3.4.0".

I got around this issue by changing the base URL for repository used to 'http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.2.0' (instead of "http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.4.0") prior to provisioning cluster from blueprint.

I'm wondering whether this is something expected or if it's a bug, i.e. that service version defined in stack is ignored unless such change is made. Shouldn't services' versions defined in stack be respected and applied regardless of base URL? Perhaps I'm missing some other settings that should be applied in blueprint or cluster creation template, but AFAIK it's all as per docs/examples I read.

Thank you for your input.

Regards,
Michal Siemaszko