You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Juanjo Marron <jm...@us.ibm.com> on 2016/03/16 22:24:43 UTC
Review Request 44933: Apply the stack featurization prototype
detailed on AMBARI-13364 to ZOOKEEPER service.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44933/
-----------------------------------------------------------
Review request for Ambari, Alejandro Fernandez and Jayush Luniya.
Bugs: AMBARI-14435
https://issues.apache.org/jira/browse/AMBARI-14435
Repository: ambari
Description
-------
Apply the stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER service.
This patch removes the hardcoded stack versions in ZK common-services code (centralized version and parameterized stack configurations will be faced in other JIRAS)
stack_features is now a property in .json format in configuration/cluster-env.xml for each stack version.
With the structure:
{
"stack_features": [
{ "name": "feature1", "description" : "Feature1 support", "min_version" : "Y.Y.Y.Y" , "max_version" : "X.X.X.X"}
,
...
]
}
where min_version/max_version are optional constraints.
New resource_management/libraries/functions/stack_fetaures.py has been introduced to parse the json file and called from service code to check if the stack supports the required feature.
Diffs
-----
ambari-common/src/main/python/resource_management/libraries/functions/stack_fetaures.py PRE-CREATION
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py 9ff9125
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py 5712ce4
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_client.py 25ace24
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py d5b6898
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py 6fe0772
ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml f1fa4de
ambari-server/src/main/resources/stacks/HDP/2.3/configuration/cluster-env.xml PRE-CREATION
Diff: https://reviews.apache.org/r/44933/diff/
Testing
-------
Zookeeper fresh installation has been tested using HDP stack 2.3 and AMBARI-13364 branch code.
Thanks,
Juanjo Marron
Re: Review Request 44933: Apply the stack featurization prototype
detailed on AMBARI-13364 to ZOOKEEPER service.
Posted by Juanjo Marron <jm...@us.ibm.com>.
> On March 18, 2016, 9:57 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml, line 24
> > <https://reviews.apache.org/r/44933/diff/1/?file=1301835#file1301835line24>
> >
> > Actually, this file must exist for the entire stack and not inside the #.# folder.
> > The reason being that an EU from HDP 2.1 to 2.3 does the following:
> >
> > 1. Current stack is 2.1
> > 2. Stop all services
> > 3. Change stack to 2.3 and save new configs
> > 4. Start all services on 2.3
> >
> > Any action before step 3 may need to reference the stack_features min_version and max_version of the newer stack it is going to. But that will of course be inaccessible at that time.
> >
> > So we need a single file for the entire stack name.
Thanks Alejandro, I got it.
I added a comment above to confirm how the new approach should work
- Juanjo
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44933/#review124298
-----------------------------------------------------------
On March 16, 2016, 9:24 p.m., Juanjo Marron wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44933/
> -----------------------------------------------------------
>
> (Updated March 16, 2016, 9:24 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Jayush Luniya.
>
>
> Bugs: AMBARI-14435
> https://issues.apache.org/jira/browse/AMBARI-14435
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Apply the stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER service.
> This patch removes the hardcoded stack versions in ZK common-services code (centralized version and parameterized stack configurations will be faced in other JIRAS)
> stack_features is now a property in .json format in configuration/cluster-env.xml for each stack version.
> With the structure:
> {
> "stack_features": [
> { "name": "feature1", "description" : "Feature1 support", "min_version" : "Y.Y.Y.Y" , "max_version" : "X.X.X.X"}
> ,
> ...
> ]
> }
> where min_version/max_version are optional constraints.
> New resource_management/libraries/functions/stack_fetaures.py has been introduced to parse the json file and called from service code to check if the stack supports the required feature.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_fetaures.py PRE-CREATION
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py 9ff9125
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py 5712ce4
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_client.py 25ace24
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py d5b6898
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py 6fe0772
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
> ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml f1fa4de
> ambari-server/src/main/resources/stacks/HDP/2.3/configuration/cluster-env.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/44933/diff/
>
>
> Testing
> -------
>
> Zookeeper fresh installation has been tested using HDP stack 2.3 and AMBARI-13364 branch code.
>
>
> Thanks,
>
> Juanjo Marron
>
>
Re: Review Request 44933: Apply the stack featurization prototype
detailed on AMBARI-13364 to ZOOKEEPER service.
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44933/#review124298
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml (line 24)
<https://reviews.apache.org/r/44933/#comment186772>
Actually, this file must exist for the entire stack and not inside the #.# folder.
The reason being that an EU from HDP 2.1 to 2.3 does the following:
1. Current stack is 2.1
2. Stop all services
3. Change stack to 2.3 and save new configs
4. Start all services on 2.3
Any action before step 3 may need to reference the stack_features min_version and max_version of the newer stack it is going to. But that will of course be inaccessible at that time.
So we need a single file for the entire stack name.
- Alejandro Fernandez
On March 16, 2016, 9:24 p.m., Juanjo Marron wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44933/
> -----------------------------------------------------------
>
> (Updated March 16, 2016, 9:24 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Jayush Luniya.
>
>
> Bugs: AMBARI-14435
> https://issues.apache.org/jira/browse/AMBARI-14435
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Apply the stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER service.
> This patch removes the hardcoded stack versions in ZK common-services code (centralized version and parameterized stack configurations will be faced in other JIRAS)
> stack_features is now a property in .json format in configuration/cluster-env.xml for each stack version.
> With the structure:
> {
> "stack_features": [
> { "name": "feature1", "description" : "Feature1 support", "min_version" : "Y.Y.Y.Y" , "max_version" : "X.X.X.X"}
> ,
> ...
> ]
> }
> where min_version/max_version are optional constraints.
> New resource_management/libraries/functions/stack_fetaures.py has been introduced to parse the json file and called from service code to check if the stack supports the required feature.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_fetaures.py PRE-CREATION
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py 9ff9125
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py 5712ce4
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_client.py 25ace24
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py d5b6898
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py 6fe0772
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
> ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml f1fa4de
> ambari-server/src/main/resources/stacks/HDP/2.3/configuration/cluster-env.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/44933/diff/
>
>
> Testing
> -------
>
> Zookeeper fresh installation has been tested using HDP stack 2.3 and AMBARI-13364 branch code.
>
>
> Thanks,
>
> Juanjo Marron
>
>
Re: Review Request 44933: Apply the stack featurization prototype
detailed on AMBARI-13364 to ZOOKEEPER service.
Posted by Jayush Luniya <jl...@hortonworks.com>.
> On March 21, 2016, 8:01 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py, line 40
> > <https://reviews.apache.org/r/44933/diff/1/?file=1301830#file1301830line40>
> >
> > Add constants to ambari/ambari-common/src/main/python/resource_management/libraries/functions/constants.py
> >
> > class StackFeatures:
> > ROLLING_UPGRADE = "rolling_upgrade"
>
> Juanjo Marron wrote:
> Added new review request with all the new design changes: https://reviews.apache.org/r/45141/
>
> Im not sure how is the right procedure in this case, but it was hard to me to keep track of the diff changes after the new prototype design
Reviewed the patch on the new review request. Can you close this review? Going forward just update the review with the new patch so that we can see the differences between the patches.
- Jayush
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44933/#review124636
-----------------------------------------------------------
On March 22, 2016, 1:03 a.m., Juanjo Marron wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44933/
> -----------------------------------------------------------
>
> (Updated March 22, 2016, 1:03 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Jayush Luniya.
>
>
> Bugs: AMBARI-14435
> https://issues.apache.org/jira/browse/AMBARI-14435
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Apply the stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER service.
> This patch removes the hardcoded stack versions in ZK common-services code (centralized version and parameterized stack configurations will be faced in other JIRAS)
> stack_features is now a property in .json format in configuration/cluster-env.xml for each stack version.
> With the structure:
> {
> "stack_features": [
> { "name": "feature1", "description" : "Feature1 support", "min_version" : "Y.Y.Y.Y" , "max_version" : "X.X.X.X"}
> ,
> ...
> ]
> }
> where min_version/max_version are optional constraints.
> New resource_management/libraries/functions/stack_fetaures.py has been introduced to parse the json file and called from service code to check if the stack supports the required feature.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_fetaures.py PRE-CREATION
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py 9ff9125
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py 5712ce4
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_client.py 25ace24
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py d5b6898
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py 6fe0772
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
> ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml f1fa4de
> ambari-server/src/main/resources/stacks/HDP/2.3/configuration/cluster-env.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/44933/diff/
>
>
> Testing
> -------
>
> Zookeeper fresh installation has been tested using HDP stack 2.3 and AMBARI-13364 branch code.
>
>
> Thanks,
>
> Juanjo Marron
>
>
Re: Review Request 44933: Apply the stack featurization prototype
detailed on AMBARI-13364 to ZOOKEEPER service.
Posted by Juanjo Marron <jm...@us.ibm.com>.
> On March 21, 2016, 8:01 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py, line 40
> > <https://reviews.apache.org/r/44933/diff/1/?file=1301830#file1301830line40>
> >
> > Add constants to ambari/ambari-common/src/main/python/resource_management/libraries/functions/constants.py
> >
> > class StackFeatures:
> > ROLLING_UPGRADE = "rolling_upgrade"
Added new review request with all the new design changes: https://reviews.apache.org/r/45141/
Im not sure how is the right procedure in this case, but it was hard to me to keep track of the diff changes after the new prototype design
- Juanjo
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44933/#review124636
-----------------------------------------------------------
On March 22, 2016, 1:03 a.m., Juanjo Marron wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44933/
> -----------------------------------------------------------
>
> (Updated March 22, 2016, 1:03 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Jayush Luniya.
>
>
> Bugs: AMBARI-14435
> https://issues.apache.org/jira/browse/AMBARI-14435
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Apply the stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER service.
> This patch removes the hardcoded stack versions in ZK common-services code (centralized version and parameterized stack configurations will be faced in other JIRAS)
> stack_features is now a property in .json format in configuration/cluster-env.xml for each stack version.
> With the structure:
> {
> "stack_features": [
> { "name": "feature1", "description" : "Feature1 support", "min_version" : "Y.Y.Y.Y" , "max_version" : "X.X.X.X"}
> ,
> ...
> ]
> }
> where min_version/max_version are optional constraints.
> New resource_management/libraries/functions/stack_fetaures.py has been introduced to parse the json file and called from service code to check if the stack supports the required feature.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_fetaures.py PRE-CREATION
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py 9ff9125
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py 5712ce4
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_client.py 25ace24
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py d5b6898
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py 6fe0772
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
> ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml f1fa4de
> ambari-server/src/main/resources/stacks/HDP/2.3/configuration/cluster-env.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/44933/diff/
>
>
> Testing
> -------
>
> Zookeeper fresh installation has been tested using HDP stack 2.3 and AMBARI-13364 branch code.
>
>
> Thanks,
>
> Juanjo Marron
>
>
Re: Review Request 44933: Apply the stack featurization prototype
detailed on AMBARI-13364 to ZOOKEEPER service.
Posted by Jayush Luniya <jl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44933/#review124636
-----------------------------------------------------------
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py (line 40)
<https://reviews.apache.org/r/44933/#comment187260>
Add constants to ambari/ambari-common/src/main/python/resource_management/libraries/functions/constants.py
class StackFeatures:
ROLLING_UPGRADE = "rolling_upgrade"
- Jayush Luniya
On March 16, 2016, 9:24 p.m., Juanjo Marron wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44933/
> -----------------------------------------------------------
>
> (Updated March 16, 2016, 9:24 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Jayush Luniya.
>
>
> Bugs: AMBARI-14435
> https://issues.apache.org/jira/browse/AMBARI-14435
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Apply the stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER service.
> This patch removes the hardcoded stack versions in ZK common-services code (centralized version and parameterized stack configurations will be faced in other JIRAS)
> stack_features is now a property in .json format in configuration/cluster-env.xml for each stack version.
> With the structure:
> {
> "stack_features": [
> { "name": "feature1", "description" : "Feature1 support", "min_version" : "Y.Y.Y.Y" , "max_version" : "X.X.X.X"}
> ,
> ...
> ]
> }
> where min_version/max_version are optional constraints.
> New resource_management/libraries/functions/stack_fetaures.py has been introduced to parse the json file and called from service code to check if the stack supports the required feature.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_fetaures.py PRE-CREATION
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py 9ff9125
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py 5712ce4
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_client.py 25ace24
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py d5b6898
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py 6fe0772
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
> ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml f1fa4de
> ambari-server/src/main/resources/stacks/HDP/2.3/configuration/cluster-env.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/44933/diff/
>
>
> Testing
> -------
>
> Zookeeper fresh installation has been tested using HDP stack 2.3 and AMBARI-13364 branch code.
>
>
> Thanks,
>
> Juanjo Marron
>
>
Re: Review Request 44933: Apply the stack featurization prototype
detailed on AMBARI-13364 to ZOOKEEPER service.
Posted by Juanjo Marron <jm...@us.ibm.com>.
> On March 21, 2016, 7:55 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml, lines 29-32
> > <https://reviews.apache.org/r/44933/diff/1/?file=1301835#file1301835line29>
> >
> > Should be rolling_upgrade instead of upgrade.
> > express_upgrade is supported from HDP-2.1
> > Also config versioning is supported from HDP-2.3
New approach on https://reviews.apache.org/r/45141 uses diff fetaures for rolling_upgrade, express_upgrade and config_versioning
- Juanjo
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44933/#review124634
-----------------------------------------------------------
On March 22, 2016, 1:03 a.m., Juanjo Marron wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44933/
> -----------------------------------------------------------
>
> (Updated March 22, 2016, 1:03 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Jayush Luniya.
>
>
> Bugs: AMBARI-14435
> https://issues.apache.org/jira/browse/AMBARI-14435
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Apply the stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER service.
> This patch removes the hardcoded stack versions in ZK common-services code (centralized version and parameterized stack configurations will be faced in other JIRAS)
> stack_features is now a property in .json format in configuration/cluster-env.xml for each stack version.
> With the structure:
> {
> "stack_features": [
> { "name": "feature1", "description" : "Feature1 support", "min_version" : "Y.Y.Y.Y" , "max_version" : "X.X.X.X"}
> ,
> ...
> ]
> }
> where min_version/max_version are optional constraints.
> New resource_management/libraries/functions/stack_fetaures.py has been introduced to parse the json file and called from service code to check if the stack supports the required feature.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_fetaures.py PRE-CREATION
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py 9ff9125
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py 5712ce4
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_client.py 25ace24
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py d5b6898
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py 6fe0772
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
> ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml f1fa4de
> ambari-server/src/main/resources/stacks/HDP/2.3/configuration/cluster-env.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/44933/diff/
>
>
> Testing
> -------
>
> Zookeeper fresh installation has been tested using HDP stack 2.3 and AMBARI-13364 branch code.
>
>
> Thanks,
>
> Juanjo Marron
>
>
Re: Review Request 44933: Apply the stack featurization prototype
detailed on AMBARI-13364 to ZOOKEEPER service.
Posted by Jayush Luniya <jl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44933/#review124634
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml (lines 29 - 32)
<https://reviews.apache.org/r/44933/#comment187259>
Should be rolling_upgrade instead of upgrade.
express_upgrade is supported from HDP-2.1
Also config versioning is supported from HDP-2.3
- Jayush Luniya
On March 16, 2016, 9:24 p.m., Juanjo Marron wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44933/
> -----------------------------------------------------------
>
> (Updated March 16, 2016, 9:24 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Jayush Luniya.
>
>
> Bugs: AMBARI-14435
> https://issues.apache.org/jira/browse/AMBARI-14435
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Apply the stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER service.
> This patch removes the hardcoded stack versions in ZK common-services code (centralized version and parameterized stack configurations will be faced in other JIRAS)
> stack_features is now a property in .json format in configuration/cluster-env.xml for each stack version.
> With the structure:
> {
> "stack_features": [
> { "name": "feature1", "description" : "Feature1 support", "min_version" : "Y.Y.Y.Y" , "max_version" : "X.X.X.X"}
> ,
> ...
> ]
> }
> where min_version/max_version are optional constraints.
> New resource_management/libraries/functions/stack_fetaures.py has been introduced to parse the json file and called from service code to check if the stack supports the required feature.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_fetaures.py PRE-CREATION
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py 9ff9125
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py 5712ce4
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_client.py 25ace24
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py d5b6898
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py 6fe0772
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
> ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml f1fa4de
> ambari-server/src/main/resources/stacks/HDP/2.3/configuration/cluster-env.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/44933/diff/
>
>
> Testing
> -------
>
> Zookeeper fresh installation has been tested using HDP stack 2.3 and AMBARI-13364 branch code.
>
>
> Thanks,
>
> Juanjo Marron
>
>
Re: Review Request 44933: Apply the stack featurization prototype
detailed on AMBARI-13364 to ZOOKEEPER service.
Posted by Juanjo Marron <jm...@us.ibm.com>.
> On March 18, 2016, 9:25 p.m., Jayush Luniya wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/stack_fetaures.py, line 1
> > <https://reviews.apache.org/r/44933/diff/1/?file=1301827#file1301827line1>
> >
> > Minor, fix typo in file name
>
> Juanjo Marron wrote:
> OK.Thanks. I'll solve it in the next patch replacing this one
Fixed in: https://reviews.apache.org/r/45141
> On March 18, 2016, 9:25 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.3/configuration/cluster-env.xml, lines 29-35
> > <https://reviews.apache.org/r/44933/diff/1/?file=1301836#file1301836line29>
> >
> > Dont need to provide min_version if the feature was not enabled at from a maint release onwards.
Just HDP/2.0.6/configuration/cluster-env.xml will include stack_features property
> On March 18, 2016, 9:25 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml, line 25
> > <https://reviews.apache.org/r/44933/diff/1/?file=1301834#file1301834line25>
> >
> > FYI, in https://reviews.apache.org/r/44835/ I added a feature to support external properties file for complete structures like these. So you can create a json file at HDP/2.0.6/properties/stack_features.json and reference it in the config property xml file. I will be committing those changes soon.
>
> Juanjo Marron wrote:
> I saw the example of stack_tools.json in https://reviews.apache.org/r/44835/.
> It would work even if the stack_fetaures.json file is defined out of the stack, under HDP/configuration/stack_fetautes for example?
New approach on https://reviews.apache.org/r/45141 uses the new feature to support external properties file
- Juanjo
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44933/#review124285
-----------------------------------------------------------
On March 22, 2016, 1:03 a.m., Juanjo Marron wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44933/
> -----------------------------------------------------------
>
> (Updated March 22, 2016, 1:03 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Jayush Luniya.
>
>
> Bugs: AMBARI-14435
> https://issues.apache.org/jira/browse/AMBARI-14435
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Apply the stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER service.
> This patch removes the hardcoded stack versions in ZK common-services code (centralized version and parameterized stack configurations will be faced in other JIRAS)
> stack_features is now a property in .json format in configuration/cluster-env.xml for each stack version.
> With the structure:
> {
> "stack_features": [
> { "name": "feature1", "description" : "Feature1 support", "min_version" : "Y.Y.Y.Y" , "max_version" : "X.X.X.X"}
> ,
> ...
> ]
> }
> where min_version/max_version are optional constraints.
> New resource_management/libraries/functions/stack_fetaures.py has been introduced to parse the json file and called from service code to check if the stack supports the required feature.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_fetaures.py PRE-CREATION
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py 9ff9125
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py 5712ce4
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_client.py 25ace24
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py d5b6898
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py 6fe0772
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
> ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml f1fa4de
> ambari-server/src/main/resources/stacks/HDP/2.3/configuration/cluster-env.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/44933/diff/
>
>
> Testing
> -------
>
> Zookeeper fresh installation has been tested using HDP stack 2.3 and AMBARI-13364 branch code.
>
>
> Thanks,
>
> Juanjo Marron
>
>
Re: Review Request 44933: Apply the stack featurization prototype
detailed on AMBARI-13364 to ZOOKEEPER service.
Posted by Jayush Luniya <jl...@hortonworks.com>.
> On March 18, 2016, 9:25 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py, line 53
> > <https://reviews.apache.org/r/44933/diff/1/?file=1301828#file1301828line53>
> >
> > Will things might work out in regular install, I think there is going to be a problem here during upgrades.
> >
> > Lets say we are doing EU from HDP-2.1 to HDP-2.2
> > the command.json file for the command will have
> > hostLevelParams/current_version = 2.1.0.0-1234
> > hostLevelParams/version = 2.2.0.0-5678
> > commandParams/stack_version = 2.2
> > configurations/cluster-env/stack_features = "snappy" (HDP-2.1 version of stack_features)
> >
> > So the check will fail.
> >
> > @Alejandro is my understanding correct? I can try out the upgrade to verify too.
> >
> > BTW, @Juanjo Script.is_stack_greater* functions use commandParams/stack_version whereas you are changing it to current_version
>
> Jayush Luniya wrote:
> @Juanjo,
> Alejandro and I had a chat on this. Given the upgrade scenario, it would be best to have a single stack_features list for a stack and maintain a min_version and max_version for the feature instead of breaking down the list of stack features for each stack version. That way we will not run into issues with upgrades where we would sometimes need the upgradeFromVersion stack_features list and sometimes we would need the upgradeToVersion stack_features list.
>
> Juanjo Marron wrote:
> @Jayush,
> @Alejandro
> So If I understand properly, the idea is to create a xml file, let's say stack_features.xml under ambari-server/src/main/resources/stacks/HDP/configuration, for example. It will be common to all the HDP stacks.
> Inside the xml I can add the json file content and read it using the new feature to support external properties file whicj Jayush is creating in https://reviews.apache.org/r/44835.
>
> <configuration>
> <property>
> <name>stack_features</name>
> <value></value>
> <description>List of features supported by the stack</description>
> <property-type>VALUE_FROM_PROPERTY_FILE</property-type>
> <value-attributes>
> <property-file-name>stack_features.json</property-file-name>
> <property-file-type>json</property-file-type>
> </value-attributes>
> </property>
> </configuration>
>
> And stack_fetaures.json will be:
>
> {
> "stack_features": [
> { "name": "snappy", "description" : "Snappy compressor/decompressor support" , "min_version" : "2.0.0.0" , "max_version" : "2.2.0.0" }
> ,
> { "name": "upgrade", "description" : "Upgrade support", "min_version" : "2.2.0.0" }
> ,
> { "name": "ranger", "description" : "Ranger Service support", "min_version" : "2.2.0.0" }
> ]
> }
>
>
> This stack_features.xml file will be read from params.py and stack_features.py (new resource_management script) will make sure the stack_version for a specific feature is in the range [min_version - max_version]. Ill let max_version as optional argument since the re will be features working from min_version until now and the next versions.
>
>
> Also, regarding the use of commandParams/stack_version. I would need to declare it? it is needed? I mean is it diff from stack_version_unformatted, stack_version_formatted , current_version or version that are already defined?
>
> Jayush Luniya wrote:
> @Juanjo,
> I meant adding the config to the base stack HDP/2.0.6/configuration/cluster-env.xml so that it is applicable to all stack versions.
> Both min_version and max_version should be optional.
>
> If you see /var/lib/ambari-agent/data/command-XXX.json it is the json file that is passed from the server to the agent commandParams/stack_version is set there. Similarly, hostLevelParams/current_version and hostLevelParams/version
>
> params.py declare python variables and are mostly set based on the config parameters specified in this json file.
Re-read my comment as
hostLevelParams/current_version = 2.1.0.0-1234
commandParams/version = 2.2.0.0-5678
hostLevelParams/stack_version = 2.2
- Jayush
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44933/#review124285
-----------------------------------------------------------
On March 16, 2016, 9:24 p.m., Juanjo Marron wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44933/
> -----------------------------------------------------------
>
> (Updated March 16, 2016, 9:24 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Jayush Luniya.
>
>
> Bugs: AMBARI-14435
> https://issues.apache.org/jira/browse/AMBARI-14435
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Apply the stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER service.
> This patch removes the hardcoded stack versions in ZK common-services code (centralized version and parameterized stack configurations will be faced in other JIRAS)
> stack_features is now a property in .json format in configuration/cluster-env.xml for each stack version.
> With the structure:
> {
> "stack_features": [
> { "name": "feature1", "description" : "Feature1 support", "min_version" : "Y.Y.Y.Y" , "max_version" : "X.X.X.X"}
> ,
> ...
> ]
> }
> where min_version/max_version are optional constraints.
> New resource_management/libraries/functions/stack_fetaures.py has been introduced to parse the json file and called from service code to check if the stack supports the required feature.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_fetaures.py PRE-CREATION
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py 9ff9125
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py 5712ce4
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_client.py 25ace24
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py d5b6898
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py 6fe0772
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
> ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml f1fa4de
> ambari-server/src/main/resources/stacks/HDP/2.3/configuration/cluster-env.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/44933/diff/
>
>
> Testing
> -------
>
> Zookeeper fresh installation has been tested using HDP stack 2.3 and AMBARI-13364 branch code.
>
>
> Thanks,
>
> Juanjo Marron
>
>
Re: Review Request 44933: Apply the stack featurization prototype
detailed on AMBARI-13364 to ZOOKEEPER service.
Posted by Jayush Luniya <jl...@hortonworks.com>.
> On March 18, 2016, 9:25 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py, line 53
> > <https://reviews.apache.org/r/44933/diff/1/?file=1301828#file1301828line53>
> >
> > Will things might work out in regular install, I think there is going to be a problem here during upgrades.
> >
> > Lets say we are doing EU from HDP-2.1 to HDP-2.2
> > the command.json file for the command will have
> > hostLevelParams/current_version = 2.1.0.0-1234
> > hostLevelParams/version = 2.2.0.0-5678
> > commandParams/stack_version = 2.2
> > configurations/cluster-env/stack_features = "snappy" (HDP-2.1 version of stack_features)
> >
> > So the check will fail.
> >
> > @Alejandro is my understanding correct? I can try out the upgrade to verify too.
> >
> > BTW, @Juanjo Script.is_stack_greater* functions use commandParams/stack_version whereas you are changing it to current_version
>
> Jayush Luniya wrote:
> @Juanjo,
> Alejandro and I had a chat on this. Given the upgrade scenario, it would be best to have a single stack_features list for a stack and maintain a min_version and max_version for the feature instead of breaking down the list of stack features for each stack version. That way we will not run into issues with upgrades where we would sometimes need the upgradeFromVersion stack_features list and sometimes we would need the upgradeToVersion stack_features list.
>
> Juanjo Marron wrote:
> @Jayush,
> @Alejandro
> So If I understand properly, the idea is to create a xml file, let's say stack_features.xml under ambari-server/src/main/resources/stacks/HDP/configuration, for example. It will be common to all the HDP stacks.
> Inside the xml I can add the json file content and read it using the new feature to support external properties file whicj Jayush is creating in https://reviews.apache.org/r/44835.
>
> <configuration>
> <property>
> <name>stack_features</name>
> <value></value>
> <description>List of features supported by the stack</description>
> <property-type>VALUE_FROM_PROPERTY_FILE</property-type>
> <value-attributes>
> <property-file-name>stack_features.json</property-file-name>
> <property-file-type>json</property-file-type>
> </value-attributes>
> </property>
> </configuration>
>
> And stack_fetaures.json will be:
>
> {
> "stack_features": [
> { "name": "snappy", "description" : "Snappy compressor/decompressor support" , "min_version" : "2.0.0.0" , "max_version" : "2.2.0.0" }
> ,
> { "name": "upgrade", "description" : "Upgrade support", "min_version" : "2.2.0.0" }
> ,
> { "name": "ranger", "description" : "Ranger Service support", "min_version" : "2.2.0.0" }
> ]
> }
>
>
> This stack_features.xml file will be read from params.py and stack_features.py (new resource_management script) will make sure the stack_version for a specific feature is in the range [min_version - max_version]. Ill let max_version as optional argument since the re will be features working from min_version until now and the next versions.
>
>
> Also, regarding the use of commandParams/stack_version. I would need to declare it? it is needed? I mean is it diff from stack_version_unformatted, stack_version_formatted , current_version or version that are already defined?
@Juanjo,
I meant adding the config to the base stack HDP/2.0.6/configuration/cluster-env.xml so that it is applicable to all stack versions.
Both min_version and max_version should be optional.
If you see /var/lib/ambari-agent/data/command-XXX.json it is the json file that is passed from the server to the agent commandParams/stack_version is set there. Similarly, hostLevelParams/current_version and hostLevelParams/version
params.py declare python variables and are mostly set based on the config parameters specified in this json file.
- Jayush
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44933/#review124285
-----------------------------------------------------------
On March 16, 2016, 9:24 p.m., Juanjo Marron wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44933/
> -----------------------------------------------------------
>
> (Updated March 16, 2016, 9:24 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Jayush Luniya.
>
>
> Bugs: AMBARI-14435
> https://issues.apache.org/jira/browse/AMBARI-14435
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Apply the stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER service.
> This patch removes the hardcoded stack versions in ZK common-services code (centralized version and parameterized stack configurations will be faced in other JIRAS)
> stack_features is now a property in .json format in configuration/cluster-env.xml for each stack version.
> With the structure:
> {
> "stack_features": [
> { "name": "feature1", "description" : "Feature1 support", "min_version" : "Y.Y.Y.Y" , "max_version" : "X.X.X.X"}
> ,
> ...
> ]
> }
> where min_version/max_version are optional constraints.
> New resource_management/libraries/functions/stack_fetaures.py has been introduced to parse the json file and called from service code to check if the stack supports the required feature.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_fetaures.py PRE-CREATION
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py 9ff9125
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py 5712ce4
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_client.py 25ace24
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py d5b6898
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py 6fe0772
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
> ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml f1fa4de
> ambari-server/src/main/resources/stacks/HDP/2.3/configuration/cluster-env.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/44933/diff/
>
>
> Testing
> -------
>
> Zookeeper fresh installation has been tested using HDP stack 2.3 and AMBARI-13364 branch code.
>
>
> Thanks,
>
> Juanjo Marron
>
>
Re: Review Request 44933: Apply the stack featurization prototype
detailed on AMBARI-13364 to ZOOKEEPER service.
Posted by Jayush Luniya <jl...@hortonworks.com>.
> On March 18, 2016, 9:25 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py, line 53
> > <https://reviews.apache.org/r/44933/diff/1/?file=1301828#file1301828line53>
> >
> > Will things might work out in regular install, I think there is going to be a problem here during upgrades.
> >
> > Lets say we are doing EU from HDP-2.1 to HDP-2.2
> > the command.json file for the command will have
> > hostLevelParams/current_version = 2.1.0.0-1234
> > hostLevelParams/version = 2.2.0.0-5678
> > commandParams/stack_version = 2.2
> > configurations/cluster-env/stack_features = "snappy" (HDP-2.1 version of stack_features)
> >
> > So the check will fail.
> >
> > @Alejandro is my understanding correct? I can try out the upgrade to verify too.
> >
> > BTW, @Juanjo Script.is_stack_greater* functions use commandParams/stack_version whereas you are changing it to current_version
@Juanjo,
Alejandro and I had a chat on this. Given the upgrade scenario, it would be best to have a single stack_features list for a stack and maintain a min_version and max_version for the feature instead of breaking down the list of stack features for each stack version. That way we will not run into issues with upgrades where we would sometimes need the upgradeFromVersion stack_features list and sometimes we would need the upgradeToVersion stack_features list.
- Jayush
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44933/#review124285
-----------------------------------------------------------
On March 16, 2016, 9:24 p.m., Juanjo Marron wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44933/
> -----------------------------------------------------------
>
> (Updated March 16, 2016, 9:24 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Jayush Luniya.
>
>
> Bugs: AMBARI-14435
> https://issues.apache.org/jira/browse/AMBARI-14435
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Apply the stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER service.
> This patch removes the hardcoded stack versions in ZK common-services code (centralized version and parameterized stack configurations will be faced in other JIRAS)
> stack_features is now a property in .json format in configuration/cluster-env.xml for each stack version.
> With the structure:
> {
> "stack_features": [
> { "name": "feature1", "description" : "Feature1 support", "min_version" : "Y.Y.Y.Y" , "max_version" : "X.X.X.X"}
> ,
> ...
> ]
> }
> where min_version/max_version are optional constraints.
> New resource_management/libraries/functions/stack_fetaures.py has been introduced to parse the json file and called from service code to check if the stack supports the required feature.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_fetaures.py PRE-CREATION
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py 9ff9125
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py 5712ce4
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_client.py 25ace24
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py d5b6898
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py 6fe0772
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
> ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml f1fa4de
> ambari-server/src/main/resources/stacks/HDP/2.3/configuration/cluster-env.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/44933/diff/
>
>
> Testing
> -------
>
> Zookeeper fresh installation has been tested using HDP stack 2.3 and AMBARI-13364 branch code.
>
>
> Thanks,
>
> Juanjo Marron
>
>
Re: Review Request 44933: Apply the stack featurization prototype
detailed on AMBARI-13364 to ZOOKEEPER service.
Posted by Juanjo Marron <jm...@us.ibm.com>.
> On March 18, 2016, 9:25 p.m., Jayush Luniya wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/stack_fetaures.py, line 1
> > <https://reviews.apache.org/r/44933/diff/1/?file=1301827#file1301827line1>
> >
> > Minor, fix typo in file name
OK.Thanks. I'll solve it in the next patch replacing this one
> On March 18, 2016, 9:25 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py, line 53
> > <https://reviews.apache.org/r/44933/diff/1/?file=1301828#file1301828line53>
> >
> > Will things might work out in regular install, I think there is going to be a problem here during upgrades.
> >
> > Lets say we are doing EU from HDP-2.1 to HDP-2.2
> > the command.json file for the command will have
> > hostLevelParams/current_version = 2.1.0.0-1234
> > hostLevelParams/version = 2.2.0.0-5678
> > commandParams/stack_version = 2.2
> > configurations/cluster-env/stack_features = "snappy" (HDP-2.1 version of stack_features)
> >
> > So the check will fail.
> >
> > @Alejandro is my understanding correct? I can try out the upgrade to verify too.
> >
> > BTW, @Juanjo Script.is_stack_greater* functions use commandParams/stack_version whereas you are changing it to current_version
>
> Jayush Luniya wrote:
> @Juanjo,
> Alejandro and I had a chat on this. Given the upgrade scenario, it would be best to have a single stack_features list for a stack and maintain a min_version and max_version for the feature instead of breaking down the list of stack features for each stack version. That way we will not run into issues with upgrades where we would sometimes need the upgradeFromVersion stack_features list and sometimes we would need the upgradeToVersion stack_features list.
@Jayush,
@Alejandro
So If I understand properly, the idea is to create a xml file, let's say stack_features.xml under ambari-server/src/main/resources/stacks/HDP/configuration, for example. It will be common to all the HDP stacks.
Inside the xml I can add the json file content and read it using the new feature to support external properties file whicj Jayush is creating in https://reviews.apache.org/r/44835.
<configuration>
<property>
<name>stack_features</name>
<value></value>
<description>List of features supported by the stack</description>
<property-type>VALUE_FROM_PROPERTY_FILE</property-type>
<value-attributes>
<property-file-name>stack_features.json</property-file-name>
<property-file-type>json</property-file-type>
</value-attributes>
</property>
</configuration>
And stack_fetaures.json will be:
{
"stack_features": [
{ "name": "snappy", "description" : "Snappy compressor/decompressor support" , "min_version" : "2.0.0.0" , "max_version" : "2.2.0.0" }
,
{ "name": "upgrade", "description" : "Upgrade support", "min_version" : "2.2.0.0" }
,
{ "name": "ranger", "description" : "Ranger Service support", "min_version" : "2.2.0.0" }
]
}
This stack_features.xml file will be read from params.py and stack_features.py (new resource_management script) will make sure the stack_version for a specific feature is in the range [min_version - max_version]. Ill let max_version as optional argument since the re will be features working from min_version until now and the next versions.
Also, regarding the use of commandParams/stack_version. I would need to declare it? it is needed? I mean is it diff from stack_version_unformatted, stack_version_formatted , current_version or version that are already defined?
> On March 18, 2016, 9:25 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml, line 25
> > <https://reviews.apache.org/r/44933/diff/1/?file=1301834#file1301834line25>
> >
> > FYI, in https://reviews.apache.org/r/44835/ I added a feature to support external properties file for complete structures like these. So you can create a json file at HDP/2.0.6/properties/stack_features.json and reference it in the config property xml file. I will be committing those changes soon.
I saw the example of stack_tools.json in https://reviews.apache.org/r/44835/.
It would work even if the stack_fetaures.json file is defined out of the stack, under HDP/configuration/stack_fetautes for example?
- Juanjo
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44933/#review124285
-----------------------------------------------------------
On March 16, 2016, 9:24 p.m., Juanjo Marron wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44933/
> -----------------------------------------------------------
>
> (Updated March 16, 2016, 9:24 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Jayush Luniya.
>
>
> Bugs: AMBARI-14435
> https://issues.apache.org/jira/browse/AMBARI-14435
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Apply the stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER service.
> This patch removes the hardcoded stack versions in ZK common-services code (centralized version and parameterized stack configurations will be faced in other JIRAS)
> stack_features is now a property in .json format in configuration/cluster-env.xml for each stack version.
> With the structure:
> {
> "stack_features": [
> { "name": "feature1", "description" : "Feature1 support", "min_version" : "Y.Y.Y.Y" , "max_version" : "X.X.X.X"}
> ,
> ...
> ]
> }
> where min_version/max_version are optional constraints.
> New resource_management/libraries/functions/stack_fetaures.py has been introduced to parse the json file and called from service code to check if the stack supports the required feature.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_fetaures.py PRE-CREATION
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py 9ff9125
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py 5712ce4
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_client.py 25ace24
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py d5b6898
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py 6fe0772
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
> ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml f1fa4de
> ambari-server/src/main/resources/stacks/HDP/2.3/configuration/cluster-env.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/44933/diff/
>
>
> Testing
> -------
>
> Zookeeper fresh installation has been tested using HDP stack 2.3 and AMBARI-13364 branch code.
>
>
> Thanks,
>
> Juanjo Marron
>
>
Re: Review Request 44933: Apply the stack featurization prototype
detailed on AMBARI-13364 to ZOOKEEPER service.
Posted by Jayush Luniya <jl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44933/#review124285
-----------------------------------------------------------
ambari-common/src/main/python/resource_management/libraries/functions/stack_fetaures.py (line 1)
<https://reviews.apache.org/r/44933/#comment186765>
Minor, fix typo in file name
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py (line 53)
<https://reviews.apache.org/r/44933/#comment186764>
Will things might work out in regular install, I think there is going to be a problem here during upgrades.
Lets say we are doing EU from HDP-2.1 to HDP-2.2
the command.json file for the command will have
hostLevelParams/current_version = 2.1.0.0-1234
hostLevelParams/version = 2.2.0.0-5678
commandParams/stack_version = 2.2
configurations/cluster-env/stack_features = "snappy" (HDP-2.1 version of stack_features)
So the check will fail.
@Alejandro is my understanding correct? I can try out the upgrade to verify too.
BTW, @Juanjo Script.is_stack_greater* functions use commandParams/stack_version whereas you are changing it to current_version
ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml (line 25)
<https://reviews.apache.org/r/44933/#comment186762>
FYI, in https://reviews.apache.org/r/44835/ I added a feature to support external properties file for complete structures like these. So you can create a json file at HDP/2.0.6/properties/stack_features.json and reference it in the config property xml file. I will be committing those changes soon.
ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml (lines 29 - 32)
<https://reviews.apache.org/r/44933/#comment186761>
Dont need to provide min_version if the feature was not enabled at from a maint release onwards.
ambari-server/src/main/resources/stacks/HDP/2.3/configuration/cluster-env.xml (lines 29 - 35)
<https://reviews.apache.org/r/44933/#comment186760>
Dont need to provide min_version if the feature was not enabled at from a maint release onwards.
- Jayush Luniya
On March 16, 2016, 9:24 p.m., Juanjo Marron wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44933/
> -----------------------------------------------------------
>
> (Updated March 16, 2016, 9:24 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Jayush Luniya.
>
>
> Bugs: AMBARI-14435
> https://issues.apache.org/jira/browse/AMBARI-14435
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Apply the stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER service.
> This patch removes the hardcoded stack versions in ZK common-services code (centralized version and parameterized stack configurations will be faced in other JIRAS)
> stack_features is now a property in .json format in configuration/cluster-env.xml for each stack version.
> With the structure:
> {
> "stack_features": [
> { "name": "feature1", "description" : "Feature1 support", "min_version" : "Y.Y.Y.Y" , "max_version" : "X.X.X.X"}
> ,
> ...
> ]
> }
> where min_version/max_version are optional constraints.
> New resource_management/libraries/functions/stack_fetaures.py has been introduced to parse the json file and called from service code to check if the stack supports the required feature.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_fetaures.py PRE-CREATION
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py 9ff9125
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py 5712ce4
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_client.py 25ace24
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py d5b6898
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py 6fe0772
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
> ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml f1fa4de
> ambari-server/src/main/resources/stacks/HDP/2.3/configuration/cluster-env.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/44933/diff/
>
>
> Testing
> -------
>
> Zookeeper fresh installation has been tested using HDP stack 2.3 and AMBARI-13364 branch code.
>
>
> Thanks,
>
> Juanjo Marron
>
>
Re: Review Request 44933: Apply the stack featurization prototype
detailed on AMBARI-13364 to ZOOKEEPER service.
Posted by Juanjo Marron <jm...@us.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44933/
-----------------------------------------------------------
(Updated March 22, 2016, 1:03 a.m.)
Review request for Ambari, Alejandro Fernandez and Jayush Luniya.
Bugs: AMBARI-14435
https://issues.apache.org/jira/browse/AMBARI-14435
Repository: ambari
Description
-------
Apply the stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER service.
This patch removes the hardcoded stack versions in ZK common-services code (centralized version and parameterized stack configurations will be faced in other JIRAS)
stack_features is now a property in .json format in configuration/cluster-env.xml for each stack version.
With the structure:
{
"stack_features": [
{ "name": "feature1", "description" : "Feature1 support", "min_version" : "Y.Y.Y.Y" , "max_version" : "X.X.X.X"}
,
...
]
}
where min_version/max_version are optional constraints.
New resource_management/libraries/functions/stack_fetaures.py has been introduced to parse the json file and called from service code to check if the stack supports the required feature.
Diffs
-----
ambari-common/src/main/python/resource_management/libraries/functions/stack_fetaures.py PRE-CREATION
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py 9ff9125
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py 5712ce4
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_client.py 25ace24
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py d5b6898
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py 6fe0772
ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml f1fa4de
ambari-server/src/main/resources/stacks/HDP/2.3/configuration/cluster-env.xml PRE-CREATION
Diff: https://reviews.apache.org/r/44933/diff/
Testing
-------
Zookeeper fresh installation has been tested using HDP stack 2.3 and AMBARI-13364 branch code.
Thanks,
Juanjo Marron
Re: Review Request 44933: Apply the stack featurization prototype
detailed on AMBARI-13364 to ZOOKEEPER service.
Posted by Juanjo Marron <jm...@us.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44933/
-----------------------------------------------------------
(Updated March 22, 2016, 1:01 a.m.)
Review request for Ambari, Alejandro Fernandez and Jayush Luniya.
Changes
-------
New prototype based on the last comments:
- It Uses class StackFeature in resource_management/libraries/functions/constants.py
- New stack_features porperty in HDP/2.0.6/configuration/cluster-env.xml reading value from json file using VALUE_FROM_PROPERTY_FILE
- Created new stack_features.json in HDP/2.0.6/properties/
- It uses hostLevelParams/stack_version = stack_version_formatted in params_linux.py
Bugs: AMBARI-14435
https://issues.apache.org/jira/browse/AMBARI-14435
Repository: ambari
Description
-------
Apply the stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER service.
This patch removes the hardcoded stack versions in ZK common-services code (centralized version and parameterized stack configurations will be faced in other JIRAS)
stack_features is now a property in .json format in configuration/cluster-env.xml for each stack version.
With the structure:
{
"stack_features": [
{ "name": "feature1", "description" : "Feature1 support", "min_version" : "Y.Y.Y.Y" , "max_version" : "X.X.X.X"}
,
...
]
}
where min_version/max_version are optional constraints.
New resource_management/libraries/functions/stack_fetaures.py has been introduced to parse the json file and called from service code to check if the stack supports the required feature.
Diffs
-----
ambari-common/src/main/python/resource_management/libraries/functions/stack_fetaures.py PRE-CREATION
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py 9ff9125
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py 5712ce4
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_client.py 25ace24
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py d5b6898
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py 6fe0772
ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml f1fa4de
ambari-server/src/main/resources/stacks/HDP/2.3/configuration/cluster-env.xml PRE-CREATION
Diff: https://reviews.apache.org/r/44933/diff/
Testing
-------
Zookeeper fresh installation has been tested using HDP stack 2.3 and AMBARI-13364 branch code.
File Attachments (updated)
----------------
AMBARI-14435.patch
https://reviews.apache.org/media/uploaded/files/2016/03/22/ca5af0ea-dcdc-4ded-9e36-eda14e7b6ecf__AMBARI-14435.patch
Thanks,
Juanjo Marron
Re: Review Request 44933: Apply the stack featurization prototype
detailed on AMBARI-13364 to ZOOKEEPER service.
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44933/#review124071
-----------------------------------------------------------
Ship it!
Ship It!
- Alejandro Fernandez
On March 16, 2016, 9:24 p.m., Juanjo Marron wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44933/
> -----------------------------------------------------------
>
> (Updated March 16, 2016, 9:24 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Jayush Luniya.
>
>
> Bugs: AMBARI-14435
> https://issues.apache.org/jira/browse/AMBARI-14435
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Apply the stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER service.
> This patch removes the hardcoded stack versions in ZK common-services code (centralized version and parameterized stack configurations will be faced in other JIRAS)
> stack_features is now a property in .json format in configuration/cluster-env.xml for each stack version.
> With the structure:
> {
> "stack_features": [
> { "name": "feature1", "description" : "Feature1 support", "min_version" : "Y.Y.Y.Y" , "max_version" : "X.X.X.X"}
> ,
> ...
> ]
> }
> where min_version/max_version are optional constraints.
> New resource_management/libraries/functions/stack_fetaures.py has been introduced to parse the json file and called from service code to check if the stack supports the required feature.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/stack_fetaures.py PRE-CREATION
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py 9ff9125
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py d18e4d7
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py 5712ce4
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_client.py 25ace24
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py d5b6898
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py 6fe0772
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 3fb82e9
> ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml f1fa4de
> ambari-server/src/main/resources/stacks/HDP/2.3/configuration/cluster-env.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/44933/diff/
>
>
> Testing
> -------
>
> Zookeeper fresh installation has been tested using HDP stack 2.3 and AMBARI-13364 branch code.
>
>
> Thanks,
>
> Juanjo Marron
>
>