You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Di Li <di...@ca.ibm.com> on 2015/09/10 23:37:28 UTC
Review Request 38269: AMBARI-12949: Downgrade fails with err: 500
status
code received on POST method for API: /api/vi/clusters/mycluster/upgrades
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38269/
-----------------------------------------------------------
Review request for Ambari and Alejandro Fernandez.
Bugs: AMBARI-12949
https://issues.apache.org/jira/browse/AMBARI-12949
Repository: ambari
Description
-------
This seems to be an issue that happens when the "latest" configuration stored in the "clusterconfig" table is for a conf group instead of the default configurations. The named query "ClusterConfigEntity.findLatestConfigsByStack" returns the latest configuration to be set with "selected = 1" in the clusterconfigmapping table. When the "latest" config is for a conf group, this record will not have a corresponding entry in the clusterconfigmapping table. So the clusterconfigmapping table is left with a type that does not have the selected column set to 1.
My proposed fix is the new named query "ClusterConfigEntity.findLatestConfigsOfClusterConfigMappingEntriesByStack" in ClusterConfigEntity.java, which searches the clusterconfig table for the latest version of a type_name and version_tag specified in the clusterconfigmapping table. This is different from the current "ClusterConfigEntity.findLatestConfigsByStack" query, which simply pulls the latest config of a type_name from the clusterconfig table regardless whether the corresponding version_tag is in the clusterconfigmapping table or not.
Investigation:
I used Oozie for debugging.
When a user clicks the Downgrade button from the Finalize page. The request eventually hits ClusterImpl.applyLatestConfigurations method where the Ambari server updates the database to reverse back to the "latest" configuration of each type for the older stack (HDP 2.2)
The workflow goes as
1. Set ALL entries in clusterconfigmapping table to have selected value set to 0
2. Ambari server uses ClusterImpl.applyLatestConfigurations method to get the "latest" configuration of each type for the older stack (HDP 2.2)
3. Loop through all the entries in the clusterconfigmapping table, and set the selected to 1 if this entry also happens to be in the "latest" configuration returned in step 2.
The ClusterImpl.applyLatestConfigurations method calls ClusterDAO.java getLatestConfigurations to get the latest configuration for each type for the older stack. The DAO class simply runs the following SQL query (cluster id and stack id are the parameters) and passes the results back.
SELECT clusterConfig1 FROM clusterconfig clusterConfig1 WHERE clusterConfig1.cluster_id='2' AND clusterConfig1.create_timestamp = (SELECT MAX(clusterConfig2.create_timestamp) FROM clusterconfig clusterConfig2 WHERE clusterConfig2.cluster_id='2' AND clusterConfig2.stack_id= '3' AND clusterConfig2.type_name = clusterConfig1.type_name);
For downgrade, in our particular use case, the oozie-env configuration stored in the clusterconfig table, when the stack id is the HDP 2.2 version, only contains TWO conf, the original version1 and the one for the config group.
The query above will return the entry corresponding to the oozie config group, as it was the last configuration entry added to the clusterconfig table for oozie-site with stack id = 3. However, because the config group entry is NEVER stored into the clusterconfigmapping table, the logic in ClusterImpl.applyLatestConfigurations method to re-select a version for a type in the clusterconfigmapping will not be able to update the clusterconfigmapping table. Thus no oozie-site is set with selected = 1 in the clusterconfigmapping table.
for( ClusterConfigEntity clusterConfigToMakeSelected : clusterConfigsToMakeSelected ){
LOG.info("Checking " + clusterConfigToMakeSelected.getType() + " with tag " + clusterConfigToMakeSelected.getTag());
for (ClusterConfigMappingEntity configMappingEntity : configMappingEntities) {
String tag = configMappingEntity.getTag();
String type = configMappingEntity.getType();
if (clusterConfigToMakeSelected.getTag().equals(tag)
&& clusterConfigToMakeSelected.getType().equals(type))
{ LOG.info(type + " with tag " + tag + " is selected"); configMappingEntity.setSelected(1); }
}
}
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterDAO.java 6e55128
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java 899ffe8
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java a48fb54
ambari-server/src/test/java/org/apache/ambari/server/orm/dao/ServiceConfigDAOTest.java 4c186b5
Diff: https://reviews.apache.org/r/38269/diff/
Testing
-------
unit test: added more test cases to ambari-server code
test on a rolling upgrade then downgrade.
Thanks,
Di Li
Re: Review Request 38269: AMBARI-12949: Downgrade fails with err: 500
status
code received on POST method for API: /api/vi/clusters/mycluster/upgrades
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38269/#review98871
-----------------------------------------------------------
This looks good. Thank you for the contribution.
Please also include Nate Cole and Jonathan Hurley in the code review.
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterDAO.java (line 190)
<https://reviews.apache.org/r/38269/#comment155504>
If this is no longer used, can remove it.
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java (line 54)
<https://reviews.apache.org/r/38269/#comment155502>
If this is no longer used, can remove it.
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java (line 2845)
<https://reviews.apache.org/r/38269/#comment155501>
Can remove this.
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java (line 2867)
<https://reviews.apache.org/r/38269/#comment155505>
Typo, has => have.
Let's change the logging level to ERROR, and note that it was in the "applyLatestConfigurations" method.
ambari-server/src/test/java/org/apache/ambari/server/orm/dao/ServiceConfigDAOTest.java (line 346)
<https://reviews.apache.org/r/38269/#comment155507>
Please add some doc outlining the situation this is testing.
- Alejandro Fernandez
On Sept. 10, 2015, 9:37 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38269/
> -----------------------------------------------------------
>
> (Updated Sept. 10, 2015, 9:37 p.m.)
>
>
> Review request for Ambari and Alejandro Fernandez.
>
>
> Bugs: AMBARI-12949
> https://issues.apache.org/jira/browse/AMBARI-12949
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This seems to be an issue that happens when the "latest" configuration stored in the "clusterconfig" table is for a conf group instead of the default configurations. The named query "ClusterConfigEntity.findLatestConfigsByStack" returns the latest configuration to be set with "selected = 1" in the clusterconfigmapping table. When the "latest" config is for a conf group, this record will not have a corresponding entry in the clusterconfigmapping table. So the clusterconfigmapping table is left with a type that does not have the selected column set to 1.
>
> My proposed fix is the new named query "ClusterConfigEntity.findLatestConfigsOfClusterConfigMappingEntriesByStack" in ClusterConfigEntity.java, which searches the clusterconfig table for the latest version of a type_name and version_tag specified in the clusterconfigmapping table. This is different from the current "ClusterConfigEntity.findLatestConfigsByStack" query, which simply pulls the latest config of a type_name from the clusterconfig table regardless whether the corresponding version_tag is in the clusterconfigmapping table or not.
>
>
> Investigation:
>
> I used Oozie for debugging.
>
> When a user clicks the Downgrade button from the Finalize page. The request eventually hits ClusterImpl.applyLatestConfigurations method where the Ambari server updates the database to reverse back to the "latest" configuration of each type for the older stack (HDP 2.2)
>
> The workflow goes as
> 1. Set ALL entries in clusterconfigmapping table to have selected value set to 0
> 2. Ambari server uses ClusterImpl.applyLatestConfigurations method to get the "latest" configuration of each type for the older stack (HDP 2.2)
> 3. Loop through all the entries in the clusterconfigmapping table, and set the selected to 1 if this entry also happens to be in the "latest" configuration returned in step 2.
>
> The ClusterImpl.applyLatestConfigurations method calls ClusterDAO.java getLatestConfigurations to get the latest configuration for each type for the older stack. The DAO class simply runs the following SQL query (cluster id and stack id are the parameters) and passes the results back.
>
> SELECT clusterConfig1 FROM clusterconfig clusterConfig1 WHERE clusterConfig1.cluster_id='2' AND clusterConfig1.create_timestamp = (SELECT MAX(clusterConfig2.create_timestamp) FROM clusterconfig clusterConfig2 WHERE clusterConfig2.cluster_id='2' AND clusterConfig2.stack_id= '3' AND clusterConfig2.type_name = clusterConfig1.type_name);
>
> For downgrade, in our particular use case, the oozie-env configuration stored in the clusterconfig table, when the stack id is the HDP 2.2 version, only contains TWO conf, the original version1 and the one for the config group.
>
> The query above will return the entry corresponding to the oozie config group, as it was the last configuration entry added to the clusterconfig table for oozie-site with stack id = 3. However, because the config group entry is NEVER stored into the clusterconfigmapping table, the logic in ClusterImpl.applyLatestConfigurations method to re-select a version for a type in the clusterconfigmapping will not be able to update the clusterconfigmapping table. Thus no oozie-site is set with selected = 1 in the clusterconfigmapping table.
>
> for( ClusterConfigEntity clusterConfigToMakeSelected : clusterConfigsToMakeSelected ){
> LOG.info("Checking " + clusterConfigToMakeSelected.getType() + " with tag " + clusterConfigToMakeSelected.getTag());
> for (ClusterConfigMappingEntity configMappingEntity : configMappingEntities) {
> String tag = configMappingEntity.getTag();
> String type = configMappingEntity.getType();
>
> if (clusterConfigToMakeSelected.getTag().equals(tag)
> && clusterConfigToMakeSelected.getType().equals(type))
> { LOG.info(type + " with tag " + tag + " is selected"); configMappingEntity.setSelected(1); }
>
> }
> }
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterDAO.java 6e55128
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java 899ffe8
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java a48fb54
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/ServiceConfigDAOTest.java 4c186b5
>
> Diff: https://reviews.apache.org/r/38269/diff/
>
>
> Testing
> -------
>
> unit test: added more test cases to ambari-server code
> test on a rolling upgrade then downgrade.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 38269: AMBARI-12949: Downgrade fails with err: 500
status
code received on POST method for API: /api/vi/clusters/mycluster/upgrades
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38269/#review98913
-----------------------------------------------------------
Ship it!
I suggest getting another Ship It from Nate or Jonathan.
- Alejandro Fernandez
On Sept. 14, 2015, 7:09 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38269/
> -----------------------------------------------------------
>
> (Updated Sept. 14, 2015, 7:09 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate Cole.
>
>
> Bugs: AMBARI-12949
> https://issues.apache.org/jira/browse/AMBARI-12949
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This seems to be an issue that happens when the "latest" configuration stored in the "clusterconfig" table is for a conf group instead of the default configurations. The named query "ClusterConfigEntity.findLatestConfigsByStack" returns the latest configuration to be set with "selected = 1" in the clusterconfigmapping table. When the "latest" config is for a conf group, this record will not have a corresponding entry in the clusterconfigmapping table. So the clusterconfigmapping table is left with a type that does not have the selected column set to 1.
>
> My proposed fix is the new named query "ClusterConfigEntity.findLatestConfigsOfClusterConfigMappingEntriesByStack" in ClusterConfigEntity.java, which searches the clusterconfig table for the latest version of a type_name and version_tag specified in the clusterconfigmapping table. This is different from the current "ClusterConfigEntity.findLatestConfigsByStack" query, which simply pulls the latest config of a type_name from the clusterconfig table regardless whether the corresponding version_tag is in the clusterconfigmapping table or not.
>
>
> Investigation:
>
> I used Oozie for debugging.
>
> When a user clicks the Downgrade button from the Finalize page. The request eventually hits ClusterImpl.applyLatestConfigurations method where the Ambari server updates the database to reverse back to the "latest" configuration of each type for the older stack (HDP 2.2)
>
> The workflow goes as
> 1. Set ALL entries in clusterconfigmapping table to have selected value set to 0
> 2. Ambari server uses ClusterImpl.applyLatestConfigurations method to get the "latest" configuration of each type for the older stack (HDP 2.2)
> 3. Loop through all the entries in the clusterconfigmapping table, and set the selected to 1 if this entry also happens to be in the "latest" configuration returned in step 2.
>
> The ClusterImpl.applyLatestConfigurations method calls ClusterDAO.java getLatestConfigurations to get the latest configuration for each type for the older stack. The DAO class simply runs the following SQL query (cluster id and stack id are the parameters) and passes the results back.
>
> SELECT clusterConfig1 FROM clusterconfig clusterConfig1 WHERE clusterConfig1.cluster_id='2' AND clusterConfig1.create_timestamp = (SELECT MAX(clusterConfig2.create_timestamp) FROM clusterconfig clusterConfig2 WHERE clusterConfig2.cluster_id='2' AND clusterConfig2.stack_id= '3' AND clusterConfig2.type_name = clusterConfig1.type_name);
>
> For downgrade, in our particular use case, the oozie-env configuration stored in the clusterconfig table, when the stack id is the HDP 2.2 version, only contains TWO conf, the original version1 and the one for the config group.
>
> The query above will return the entry corresponding to the oozie config group, as it was the last configuration entry added to the clusterconfig table for oozie-site with stack id = 3. However, because the config group entry is NEVER stored into the clusterconfigmapping table, the logic in ClusterImpl.applyLatestConfigurations method to re-select a version for a type in the clusterconfigmapping will not be able to update the clusterconfigmapping table. Thus no oozie-site is set with selected = 1 in the clusterconfigmapping table.
>
> for( ClusterConfigEntity clusterConfigToMakeSelected : clusterConfigsToMakeSelected ){
> LOG.info("Checking " + clusterConfigToMakeSelected.getType() + " with tag " + clusterConfigToMakeSelected.getTag());
> for (ClusterConfigMappingEntity configMappingEntity : configMappingEntities) {
> String tag = configMappingEntity.getTag();
> String type = configMappingEntity.getType();
>
> if (clusterConfigToMakeSelected.getTag().equals(tag)
> && clusterConfigToMakeSelected.getType().equals(type))
> { LOG.info(type + " with tag " + tag + " is selected"); configMappingEntity.setSelected(1); }
>
> }
> }
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterDAO.java 6e55128
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java 899ffe8
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java a48fb54
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/ServiceConfigDAOTest.java 4c186b5
>
> Diff: https://reviews.apache.org/r/38269/diff/
>
>
> Testing
> -------
>
> unit test: added more test cases to ambari-server code
> test on a rolling upgrade then downgrade.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 38269: AMBARI-12949: Downgrade fails with err: 500
status
code received on POST method for API: /api/vi/clusters/mycluster/upgrades
Posted by Di Li <di...@ca.ibm.com>.
> On Sept. 15, 2015, 12:36 p.m., Nate Cole wrote:
> >
Hello Nate,
Could you please help review the latest code change with changes per your feedback?
Thanks.
Di
- Di
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38269/#review99016
-----------------------------------------------------------
On Sept. 14, 2015, 7:09 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38269/
> -----------------------------------------------------------
>
> (Updated Sept. 14, 2015, 7:09 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate Cole.
>
>
> Bugs: AMBARI-12949
> https://issues.apache.org/jira/browse/AMBARI-12949
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This seems to be an issue that happens when the "latest" configuration stored in the "clusterconfig" table is for a conf group instead of the default configurations. The named query "ClusterConfigEntity.findLatestConfigsByStack" returns the latest configuration to be set with "selected = 1" in the clusterconfigmapping table. When the "latest" config is for a conf group, this record will not have a corresponding entry in the clusterconfigmapping table. So the clusterconfigmapping table is left with a type that does not have the selected column set to 1.
>
> My proposed fix is the new named query "ClusterConfigEntity.findLatestConfigsOfClusterConfigMappingEntriesByStack" in ClusterConfigEntity.java, which searches the clusterconfig table for the latest version of a type_name and version_tag specified in the clusterconfigmapping table. This is different from the current "ClusterConfigEntity.findLatestConfigsByStack" query, which simply pulls the latest config of a type_name from the clusterconfig table regardless whether the corresponding version_tag is in the clusterconfigmapping table or not.
>
>
> Investigation:
>
> I used Oozie for debugging.
>
> When a user clicks the Downgrade button from the Finalize page. The request eventually hits ClusterImpl.applyLatestConfigurations method where the Ambari server updates the database to reverse back to the "latest" configuration of each type for the older stack (HDP 2.2)
>
> The workflow goes as
> 1. Set ALL entries in clusterconfigmapping table to have selected value set to 0
> 2. Ambari server uses ClusterImpl.applyLatestConfigurations method to get the "latest" configuration of each type for the older stack (HDP 2.2)
> 3. Loop through all the entries in the clusterconfigmapping table, and set the selected to 1 if this entry also happens to be in the "latest" configuration returned in step 2.
>
> The ClusterImpl.applyLatestConfigurations method calls ClusterDAO.java getLatestConfigurations to get the latest configuration for each type for the older stack. The DAO class simply runs the following SQL query (cluster id and stack id are the parameters) and passes the results back.
>
> SELECT clusterConfig1 FROM clusterconfig clusterConfig1 WHERE clusterConfig1.cluster_id='2' AND clusterConfig1.create_timestamp = (SELECT MAX(clusterConfig2.create_timestamp) FROM clusterconfig clusterConfig2 WHERE clusterConfig2.cluster_id='2' AND clusterConfig2.stack_id= '3' AND clusterConfig2.type_name = clusterConfig1.type_name);
>
> For downgrade, in our particular use case, the oozie-env configuration stored in the clusterconfig table, when the stack id is the HDP 2.2 version, only contains TWO conf, the original version1 and the one for the config group.
>
> The query above will return the entry corresponding to the oozie config group, as it was the last configuration entry added to the clusterconfig table for oozie-site with stack id = 3. However, because the config group entry is NEVER stored into the clusterconfigmapping table, the logic in ClusterImpl.applyLatestConfigurations method to re-select a version for a type in the clusterconfigmapping will not be able to update the clusterconfigmapping table. Thus no oozie-site is set with selected = 1 in the clusterconfigmapping table.
>
> for( ClusterConfigEntity clusterConfigToMakeSelected : clusterConfigsToMakeSelected ){
> LOG.info("Checking " + clusterConfigToMakeSelected.getType() + " with tag " + clusterConfigToMakeSelected.getTag());
> for (ClusterConfigMappingEntity configMappingEntity : configMappingEntities) {
> String tag = configMappingEntity.getTag();
> String type = configMappingEntity.getType();
>
> if (clusterConfigToMakeSelected.getTag().equals(tag)
> && clusterConfigToMakeSelected.getType().equals(type))
> { LOG.info(type + " with tag " + tag + " is selected"); configMappingEntity.setSelected(1); }
>
> }
> }
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterDAO.java 6e55128
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java 899ffe8
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java a48fb54
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/ServiceConfigDAOTest.java 4c186b5
>
> Diff: https://reviews.apache.org/r/38269/diff/
>
>
> Testing
> -------
>
> unit test: added more test cases to ambari-server code
> test on a rolling upgrade then downgrade.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 38269: AMBARI-12949: Downgrade fails with err: 500
status
code received on POST method for API: /api/vi/clusters/mycluster/upgrades
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38269/#review99016
-----------------------------------------------------------
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterDAO.java (line 190)
<https://reviews.apache.org/r/38269/#comment155824>
Method name is a little long. The previous name indicated purpose sufficiently.
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java (line 54)
<https://reviews.apache.org/r/38269/#comment155825>
Just change the query - no other clarification (crazy long method name) is needed.
ambari-server/src/test/java/org/apache/ambari/server/orm/dao/ServiceConfigDAOTest.java (line 50)
<https://reviews.apache.org/r/38269/#comment155826>
Unused
ambari-server/src/test/java/org/apache/ambari/server/orm/dao/ServiceConfigDAOTest.java (line 384)
<https://reviews.apache.org/r/38269/#comment155827>
May also want to test creating a new entity, then recalling the dao method to confirm it's working
- Nate Cole
On Sept. 14, 2015, 3:09 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38269/
> -----------------------------------------------------------
>
> (Updated Sept. 14, 2015, 3:09 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate Cole.
>
>
> Bugs: AMBARI-12949
> https://issues.apache.org/jira/browse/AMBARI-12949
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This seems to be an issue that happens when the "latest" configuration stored in the "clusterconfig" table is for a conf group instead of the default configurations. The named query "ClusterConfigEntity.findLatestConfigsByStack" returns the latest configuration to be set with "selected = 1" in the clusterconfigmapping table. When the "latest" config is for a conf group, this record will not have a corresponding entry in the clusterconfigmapping table. So the clusterconfigmapping table is left with a type that does not have the selected column set to 1.
>
> My proposed fix is the new named query "ClusterConfigEntity.findLatestConfigsOfClusterConfigMappingEntriesByStack" in ClusterConfigEntity.java, which searches the clusterconfig table for the latest version of a type_name and version_tag specified in the clusterconfigmapping table. This is different from the current "ClusterConfigEntity.findLatestConfigsByStack" query, which simply pulls the latest config of a type_name from the clusterconfig table regardless whether the corresponding version_tag is in the clusterconfigmapping table or not.
>
>
> Investigation:
>
> I used Oozie for debugging.
>
> When a user clicks the Downgrade button from the Finalize page. The request eventually hits ClusterImpl.applyLatestConfigurations method where the Ambari server updates the database to reverse back to the "latest" configuration of each type for the older stack (HDP 2.2)
>
> The workflow goes as
> 1. Set ALL entries in clusterconfigmapping table to have selected value set to 0
> 2. Ambari server uses ClusterImpl.applyLatestConfigurations method to get the "latest" configuration of each type for the older stack (HDP 2.2)
> 3. Loop through all the entries in the clusterconfigmapping table, and set the selected to 1 if this entry also happens to be in the "latest" configuration returned in step 2.
>
> The ClusterImpl.applyLatestConfigurations method calls ClusterDAO.java getLatestConfigurations to get the latest configuration for each type for the older stack. The DAO class simply runs the following SQL query (cluster id and stack id are the parameters) and passes the results back.
>
> SELECT clusterConfig1 FROM clusterconfig clusterConfig1 WHERE clusterConfig1.cluster_id='2' AND clusterConfig1.create_timestamp = (SELECT MAX(clusterConfig2.create_timestamp) FROM clusterconfig clusterConfig2 WHERE clusterConfig2.cluster_id='2' AND clusterConfig2.stack_id= '3' AND clusterConfig2.type_name = clusterConfig1.type_name);
>
> For downgrade, in our particular use case, the oozie-env configuration stored in the clusterconfig table, when the stack id is the HDP 2.2 version, only contains TWO conf, the original version1 and the one for the config group.
>
> The query above will return the entry corresponding to the oozie config group, as it was the last configuration entry added to the clusterconfig table for oozie-site with stack id = 3. However, because the config group entry is NEVER stored into the clusterconfigmapping table, the logic in ClusterImpl.applyLatestConfigurations method to re-select a version for a type in the clusterconfigmapping will not be able to update the clusterconfigmapping table. Thus no oozie-site is set with selected = 1 in the clusterconfigmapping table.
>
> for( ClusterConfigEntity clusterConfigToMakeSelected : clusterConfigsToMakeSelected ){
> LOG.info("Checking " + clusterConfigToMakeSelected.getType() + " with tag " + clusterConfigToMakeSelected.getTag());
> for (ClusterConfigMappingEntity configMappingEntity : configMappingEntities) {
> String tag = configMappingEntity.getTag();
> String type = configMappingEntity.getType();
>
> if (clusterConfigToMakeSelected.getTag().equals(tag)
> && clusterConfigToMakeSelected.getType().equals(type))
> { LOG.info(type + " with tag " + tag + " is selected"); configMappingEntity.setSelected(1); }
>
> }
> }
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterDAO.java 6e55128
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java 899ffe8
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java a48fb54
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/ServiceConfigDAOTest.java 4c186b5
>
> Diff: https://reviews.apache.org/r/38269/diff/
>
>
> Testing
> -------
>
> unit test: added more test cases to ambari-server code
> test on a rolling upgrade then downgrade.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 38269: AMBARI-12949: Downgrade fails with err: 500
status
code received on POST method for API: /api/vi/clusters/mycluster/upgrades
Posted by Di Li <di...@ca.ibm.com>.
> On Sept. 16, 2015, 3:41 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java, line 54
> > <https://reviews.apache.org/r/38269/diff/3/?file=1074459#file1074459line54>
> >
> > Instead of double a triple nested SELECT, would it instead make sense to just iterate over the clusterocnfigmapping listings and take the latest of each? So instead of using the clusterconfig table, we're using the clusterconfigmapping table to "apply" the latest configs?
I guess the alternative approach suggested here is the Java way of what that named query is trying to get from the database. Actually, I thought about iterating the clusterconfigmapping table initially. But I opted to updating the named query as I think changing the query introduces fewer changes to the code base while achieving the same result.
Please correct me if I am wrong, still new and learning...
- Di
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38269/#review99227
-----------------------------------------------------------
On Sept. 15, 2015, 9:12 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38269/
> -----------------------------------------------------------
>
> (Updated Sept. 15, 2015, 9:12 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate Cole.
>
>
> Bugs: AMBARI-12949
> https://issues.apache.org/jira/browse/AMBARI-12949
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This seems to be an issue that happens when the "latest" configuration stored in the "clusterconfig" table is for a conf group instead of the default configurations. The named query "ClusterConfigEntity.findLatestConfigsByStack" returns the latest configuration to be set with "selected = 1" in the clusterconfigmapping table. When the "latest" config is for a conf group, this record will not have a corresponding entry in the clusterconfigmapping table. So the clusterconfigmapping table is left with a type that does not have the selected column set to 1.
>
> My proposed fix is the new named query "ClusterConfigEntity.findLatestConfigsOfClusterConfigMappingEntriesByStack" in ClusterConfigEntity.java, which searches the clusterconfig table for the latest version of a type_name and version_tag specified in the clusterconfigmapping table. This is different from the current "ClusterConfigEntity.findLatestConfigsByStack" query, which simply pulls the latest config of a type_name from the clusterconfig table regardless whether the corresponding version_tag is in the clusterconfigmapping table or not.
>
>
> Investigation:
>
> I used Oozie for debugging.
>
> When a user clicks the Downgrade button from the Finalize page. The request eventually hits ClusterImpl.applyLatestConfigurations method where the Ambari server updates the database to reverse back to the "latest" configuration of each type for the older stack (HDP 2.2)
>
> The workflow goes as
> 1. Set ALL entries in clusterconfigmapping table to have selected value set to 0
> 2. Ambari server uses ClusterImpl.applyLatestConfigurations method to get the "latest" configuration of each type for the older stack (HDP 2.2)
> 3. Loop through all the entries in the clusterconfigmapping table, and set the selected to 1 if this entry also happens to be in the "latest" configuration returned in step 2.
>
> The ClusterImpl.applyLatestConfigurations method calls ClusterDAO.java getLatestConfigurations to get the latest configuration for each type for the older stack. The DAO class simply runs the following SQL query (cluster id and stack id are the parameters) and passes the results back.
>
> SELECT clusterConfig1 FROM clusterconfig clusterConfig1 WHERE clusterConfig1.cluster_id='2' AND clusterConfig1.create_timestamp = (SELECT MAX(clusterConfig2.create_timestamp) FROM clusterconfig clusterConfig2 WHERE clusterConfig2.cluster_id='2' AND clusterConfig2.stack_id= '3' AND clusterConfig2.type_name = clusterConfig1.type_name);
>
> For downgrade, in our particular use case, the oozie-env configuration stored in the clusterconfig table, when the stack id is the HDP 2.2 version, only contains TWO conf, the original version1 and the one for the config group.
>
> The query above will return the entry corresponding to the oozie config group, as it was the last configuration entry added to the clusterconfig table for oozie-site with stack id = 3. However, because the config group entry is NEVER stored into the clusterconfigmapping table, the logic in ClusterImpl.applyLatestConfigurations method to re-select a version for a type in the clusterconfigmapping will not be able to update the clusterconfigmapping table. Thus no oozie-site is set with selected = 1 in the clusterconfigmapping table.
>
> for( ClusterConfigEntity clusterConfigToMakeSelected : clusterConfigsToMakeSelected ){
> LOG.info("Checking " + clusterConfigToMakeSelected.getType() + " with tag " + clusterConfigToMakeSelected.getTag());
> for (ClusterConfigMappingEntity configMappingEntity : configMappingEntities) {
> String tag = configMappingEntity.getTag();
> String type = configMappingEntity.getType();
>
> if (clusterConfigToMakeSelected.getTag().equals(tag)
> && clusterConfigToMakeSelected.getType().equals(type))
> { LOG.info(type + " with tag " + tag + " is selected"); configMappingEntity.setSelected(1); }
>
> }
> }
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterDAO.java 6e55128
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java 899ffe8
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java a48fb54
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/ServiceConfigDAOTest.java 4c186b5
>
> Diff: https://reviews.apache.org/r/38269/diff/
>
>
> Testing
> -------
>
> unit test: added more test cases to ambari-server code
> test on a rolling upgrade then downgrade.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 38269: AMBARI-12949: Downgrade fails with err: 500
status
code received on POST method for API: /api/vi/clusters/mycluster/upgrades
Posted by Di Li <di...@ca.ibm.com>.
> On Sept. 16, 2015, 3:41 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java, line 54
> > <https://reviews.apache.org/r/38269/diff/3/?file=1074459#file1074459line54>
> >
> > Instead of double a triple nested SELECT, would it instead make sense to just iterate over the clusterocnfigmapping listings and take the latest of each? So instead of using the clusterconfig table, we're using the clusterconfigmapping table to "apply" the latest configs?
>
> Di Li wrote:
> I guess the alternative approach suggested here is the Java way of what that named query is trying to get from the database. Actually, I thought about iterating the clusterconfigmapping table initially. But I opted to updating the named query as I think changing the query introduces fewer changes to the code base while achieving the same result.
>
> Please correct me if I am wrong, still new and learning...
>
> Jonathan Hurley wrote:
> Your solution will work - it's just that when you have 2 nested select statements then there might be a better way to accomplish what you're trying to do. In this case, we need to find the latest applicable configuration for a given stack - not the latest configuration. Since the config group change is a new entry in clusterconfig but it's the mapping in clusterconfigmapping we're after, we probably don't want to do all of the SELECT filtering on clusterconfig.
>
> I think in this case, it's probably best to introduce a new named query that gets you the data you want directly from the clusterconfigmapping table and then use the tags from that list to mark the appropirate entities as "selected".
Hello Jonathan,
Please find the latest patch uploaded as my attempt to use entities from the clusterconfigmapping table and iterate thru them in Java in order to set the selected to 1 for the latest config of a give type of a given stack id.
Thanks.
- Di
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38269/#review99227
-----------------------------------------------------------
On Sept. 16, 2015, 5:14 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38269/
> -----------------------------------------------------------
>
> (Updated Sept. 16, 2015, 5:14 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate Cole.
>
>
> Bugs: AMBARI-12949
> https://issues.apache.org/jira/browse/AMBARI-12949
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This seems to be an issue that happens when the "latest" configuration stored in the "clusterconfig" table is for a conf group instead of the default configurations. The named query "ClusterConfigEntity.findLatestConfigsByStack" returns the latest configuration to be set with "selected = 1" in the clusterconfigmapping table. When the "latest" config is for a conf group, this record will not have a corresponding entry in the clusterconfigmapping table. So the clusterconfigmapping table is left with a type that does not have the selected column set to 1.
>
> My proposed fix is the new named query "ClusterConfigEntity.findLatestConfigsOfClusterConfigMappingEntriesByStack" in ClusterConfigEntity.java, which searches the clusterconfig table for the latest version of a type_name and version_tag specified in the clusterconfigmapping table. This is different from the current "ClusterConfigEntity.findLatestConfigsByStack" query, which simply pulls the latest config of a type_name from the clusterconfig table regardless whether the corresponding version_tag is in the clusterconfigmapping table or not.
>
>
> Investigation:
>
> I used Oozie for debugging.
>
> When a user clicks the Downgrade button from the Finalize page. The request eventually hits ClusterImpl.applyLatestConfigurations method where the Ambari server updates the database to reverse back to the "latest" configuration of each type for the older stack (HDP 2.2)
>
> The workflow goes as
> 1. Set ALL entries in clusterconfigmapping table to have selected value set to 0
> 2. Ambari server uses ClusterImpl.applyLatestConfigurations method to get the "latest" configuration of each type for the older stack (HDP 2.2)
> 3. Loop through all the entries in the clusterconfigmapping table, and set the selected to 1 if this entry also happens to be in the "latest" configuration returned in step 2.
>
> The ClusterImpl.applyLatestConfigurations method calls ClusterDAO.java getLatestConfigurations to get the latest configuration for each type for the older stack. The DAO class simply runs the following SQL query (cluster id and stack id are the parameters) and passes the results back.
>
> SELECT clusterConfig1 FROM clusterconfig clusterConfig1 WHERE clusterConfig1.cluster_id='2' AND clusterConfig1.create_timestamp = (SELECT MAX(clusterConfig2.create_timestamp) FROM clusterconfig clusterConfig2 WHERE clusterConfig2.cluster_id='2' AND clusterConfig2.stack_id= '3' AND clusterConfig2.type_name = clusterConfig1.type_name);
>
> For downgrade, in our particular use case, the oozie-env configuration stored in the clusterconfig table, when the stack id is the HDP 2.2 version, only contains TWO conf, the original version1 and the one for the config group.
>
> The query above will return the entry corresponding to the oozie config group, as it was the last configuration entry added to the clusterconfig table for oozie-site with stack id = 3. However, because the config group entry is NEVER stored into the clusterconfigmapping table, the logic in ClusterImpl.applyLatestConfigurations method to re-select a version for a type in the clusterconfigmapping will not be able to update the clusterconfigmapping table. Thus no oozie-site is set with selected = 1 in the clusterconfigmapping table.
>
> for( ClusterConfigEntity clusterConfigToMakeSelected : clusterConfigsToMakeSelected ){
> LOG.info("Checking " + clusterConfigToMakeSelected.getType() + " with tag " + clusterConfigToMakeSelected.getTag());
> for (ClusterConfigMappingEntity configMappingEntity : configMappingEntities) {
> String tag = configMappingEntity.getTag();
> String type = configMappingEntity.getType();
>
> if (clusterConfigToMakeSelected.getTag().equals(tag)
> && clusterConfigToMakeSelected.getType().equals(type))
> { LOG.info(type + " with tag " + tag + " is selected"); configMappingEntity.setSelected(1); }
>
> }
> }
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterDAO.java 6e55128
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java 899ffe8
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java a48fb54
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/ServiceConfigDAOTest.java 4c186b5
>
> Diff: https://reviews.apache.org/r/38269/diff/
>
>
> Testing
> -------
>
> unit test: added more test cases to ambari-server code
> test on a rolling upgrade then downgrade.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 38269: AMBARI-12949: Downgrade fails with err: 500
status
code received on POST method for API: /api/vi/clusters/mycluster/upgrades
Posted by Jonathan Hurley <jh...@hortonworks.com>.
> On Sept. 16, 2015, 11:41 a.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java, line 54
> > <https://reviews.apache.org/r/38269/diff/3/?file=1074459#file1074459line54>
> >
> > Instead of double a triple nested SELECT, would it instead make sense to just iterate over the clusterocnfigmapping listings and take the latest of each? So instead of using the clusterconfig table, we're using the clusterconfigmapping table to "apply" the latest configs?
>
> Di Li wrote:
> I guess the alternative approach suggested here is the Java way of what that named query is trying to get from the database. Actually, I thought about iterating the clusterconfigmapping table initially. But I opted to updating the named query as I think changing the query introduces fewer changes to the code base while achieving the same result.
>
> Please correct me if I am wrong, still new and learning...
Your solution will work - it's just that when you have 2 nested select statements then there might be a better way to accomplish what you're trying to do. In this case, we need to find the latest applicable configuration for a given stack - not the latest configuration. Since the config group change is a new entry in clusterconfig but it's the mapping in clusterconfigmapping we're after, we probably don't want to do all of the SELECT filtering on clusterconfig.
I think in this case, it's probably best to introduce a new named query that gets you the data you want directly from the clusterconfigmapping table and then use the tags from that list to mark the appropirate entities as "selected".
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38269/#review99227
-----------------------------------------------------------
On Sept. 16, 2015, 1:14 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38269/
> -----------------------------------------------------------
>
> (Updated Sept. 16, 2015, 1:14 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate Cole.
>
>
> Bugs: AMBARI-12949
> https://issues.apache.org/jira/browse/AMBARI-12949
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This seems to be an issue that happens when the "latest" configuration stored in the "clusterconfig" table is for a conf group instead of the default configurations. The named query "ClusterConfigEntity.findLatestConfigsByStack" returns the latest configuration to be set with "selected = 1" in the clusterconfigmapping table. When the "latest" config is for a conf group, this record will not have a corresponding entry in the clusterconfigmapping table. So the clusterconfigmapping table is left with a type that does not have the selected column set to 1.
>
> My proposed fix is the new named query "ClusterConfigEntity.findLatestConfigsOfClusterConfigMappingEntriesByStack" in ClusterConfigEntity.java, which searches the clusterconfig table for the latest version of a type_name and version_tag specified in the clusterconfigmapping table. This is different from the current "ClusterConfigEntity.findLatestConfigsByStack" query, which simply pulls the latest config of a type_name from the clusterconfig table regardless whether the corresponding version_tag is in the clusterconfigmapping table or not.
>
>
> Investigation:
>
> I used Oozie for debugging.
>
> When a user clicks the Downgrade button from the Finalize page. The request eventually hits ClusterImpl.applyLatestConfigurations method where the Ambari server updates the database to reverse back to the "latest" configuration of each type for the older stack (HDP 2.2)
>
> The workflow goes as
> 1. Set ALL entries in clusterconfigmapping table to have selected value set to 0
> 2. Ambari server uses ClusterImpl.applyLatestConfigurations method to get the "latest" configuration of each type for the older stack (HDP 2.2)
> 3. Loop through all the entries in the clusterconfigmapping table, and set the selected to 1 if this entry also happens to be in the "latest" configuration returned in step 2.
>
> The ClusterImpl.applyLatestConfigurations method calls ClusterDAO.java getLatestConfigurations to get the latest configuration for each type for the older stack. The DAO class simply runs the following SQL query (cluster id and stack id are the parameters) and passes the results back.
>
> SELECT clusterConfig1 FROM clusterconfig clusterConfig1 WHERE clusterConfig1.cluster_id='2' AND clusterConfig1.create_timestamp = (SELECT MAX(clusterConfig2.create_timestamp) FROM clusterconfig clusterConfig2 WHERE clusterConfig2.cluster_id='2' AND clusterConfig2.stack_id= '3' AND clusterConfig2.type_name = clusterConfig1.type_name);
>
> For downgrade, in our particular use case, the oozie-env configuration stored in the clusterconfig table, when the stack id is the HDP 2.2 version, only contains TWO conf, the original version1 and the one for the config group.
>
> The query above will return the entry corresponding to the oozie config group, as it was the last configuration entry added to the clusterconfig table for oozie-site with stack id = 3. However, because the config group entry is NEVER stored into the clusterconfigmapping table, the logic in ClusterImpl.applyLatestConfigurations method to re-select a version for a type in the clusterconfigmapping will not be able to update the clusterconfigmapping table. Thus no oozie-site is set with selected = 1 in the clusterconfigmapping table.
>
> for( ClusterConfigEntity clusterConfigToMakeSelected : clusterConfigsToMakeSelected ){
> LOG.info("Checking " + clusterConfigToMakeSelected.getType() + " with tag " + clusterConfigToMakeSelected.getTag());
> for (ClusterConfigMappingEntity configMappingEntity : configMappingEntities) {
> String tag = configMappingEntity.getTag();
> String type = configMappingEntity.getType();
>
> if (clusterConfigToMakeSelected.getTag().equals(tag)
> && clusterConfigToMakeSelected.getType().equals(type))
> { LOG.info(type + " with tag " + tag + " is selected"); configMappingEntity.setSelected(1); }
>
> }
> }
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterDAO.java 6e55128
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java 899ffe8
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java a48fb54
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/ServiceConfigDAOTest.java 4c186b5
>
> Diff: https://reviews.apache.org/r/38269/diff/
>
>
> Testing
> -------
>
> unit test: added more test cases to ambari-server code
> test on a rolling upgrade then downgrade.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 38269: AMBARI-12949: Downgrade fails with err: 500
status
code received on POST method for API: /api/vi/clusters/mycluster/upgrades
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38269/#review99227
-----------------------------------------------------------
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java (line 54)
<https://reviews.apache.org/r/38269/#comment156163>
Instead of double a triple nested SELECT, would it instead make sense to just iterate over the clusterocnfigmapping listings and take the latest of each? So instead of using the clusterconfig table, we're using the clusterconfigmapping table to "apply" the latest configs?
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java (line 2837)
<https://reviews.apache.org/r/38269/#comment156159>
Don't use + when logging statements as it constructs a string. You can use {} as a placeholder instead.
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java (line 2841)
<https://reviews.apache.org/r/38269/#comment156161>
Don't use + when logging statements as it constructs a string. You can use {} as a placeholder instead.
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java (line 2850)
<https://reviews.apache.org/r/38269/#comment156160>
Don't use + when logging statements as it constructs a string. You can use {} as a placeholder instead.
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java (line 2857)
<https://reviews.apache.org/r/38269/#comment156162>
Don't use + when logging statements as it constructs a string. You can use {} as a placeholder instead.
- Jonathan Hurley
On Sept. 15, 2015, 5:12 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38269/
> -----------------------------------------------------------
>
> (Updated Sept. 15, 2015, 5:12 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate Cole.
>
>
> Bugs: AMBARI-12949
> https://issues.apache.org/jira/browse/AMBARI-12949
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This seems to be an issue that happens when the "latest" configuration stored in the "clusterconfig" table is for a conf group instead of the default configurations. The named query "ClusterConfigEntity.findLatestConfigsByStack" returns the latest configuration to be set with "selected = 1" in the clusterconfigmapping table. When the "latest" config is for a conf group, this record will not have a corresponding entry in the clusterconfigmapping table. So the clusterconfigmapping table is left with a type that does not have the selected column set to 1.
>
> My proposed fix is the new named query "ClusterConfigEntity.findLatestConfigsOfClusterConfigMappingEntriesByStack" in ClusterConfigEntity.java, which searches the clusterconfig table for the latest version of a type_name and version_tag specified in the clusterconfigmapping table. This is different from the current "ClusterConfigEntity.findLatestConfigsByStack" query, which simply pulls the latest config of a type_name from the clusterconfig table regardless whether the corresponding version_tag is in the clusterconfigmapping table or not.
>
>
> Investigation:
>
> I used Oozie for debugging.
>
> When a user clicks the Downgrade button from the Finalize page. The request eventually hits ClusterImpl.applyLatestConfigurations method where the Ambari server updates the database to reverse back to the "latest" configuration of each type for the older stack (HDP 2.2)
>
> The workflow goes as
> 1. Set ALL entries in clusterconfigmapping table to have selected value set to 0
> 2. Ambari server uses ClusterImpl.applyLatestConfigurations method to get the "latest" configuration of each type for the older stack (HDP 2.2)
> 3. Loop through all the entries in the clusterconfigmapping table, and set the selected to 1 if this entry also happens to be in the "latest" configuration returned in step 2.
>
> The ClusterImpl.applyLatestConfigurations method calls ClusterDAO.java getLatestConfigurations to get the latest configuration for each type for the older stack. The DAO class simply runs the following SQL query (cluster id and stack id are the parameters) and passes the results back.
>
> SELECT clusterConfig1 FROM clusterconfig clusterConfig1 WHERE clusterConfig1.cluster_id='2' AND clusterConfig1.create_timestamp = (SELECT MAX(clusterConfig2.create_timestamp) FROM clusterconfig clusterConfig2 WHERE clusterConfig2.cluster_id='2' AND clusterConfig2.stack_id= '3' AND clusterConfig2.type_name = clusterConfig1.type_name);
>
> For downgrade, in our particular use case, the oozie-env configuration stored in the clusterconfig table, when the stack id is the HDP 2.2 version, only contains TWO conf, the original version1 and the one for the config group.
>
> The query above will return the entry corresponding to the oozie config group, as it was the last configuration entry added to the clusterconfig table for oozie-site with stack id = 3. However, because the config group entry is NEVER stored into the clusterconfigmapping table, the logic in ClusterImpl.applyLatestConfigurations method to re-select a version for a type in the clusterconfigmapping will not be able to update the clusterconfigmapping table. Thus no oozie-site is set with selected = 1 in the clusterconfigmapping table.
>
> for( ClusterConfigEntity clusterConfigToMakeSelected : clusterConfigsToMakeSelected ){
> LOG.info("Checking " + clusterConfigToMakeSelected.getType() + " with tag " + clusterConfigToMakeSelected.getTag());
> for (ClusterConfigMappingEntity configMappingEntity : configMappingEntities) {
> String tag = configMappingEntity.getTag();
> String type = configMappingEntity.getType();
>
> if (clusterConfigToMakeSelected.getTag().equals(tag)
> && clusterConfigToMakeSelected.getType().equals(type))
> { LOG.info(type + " with tag " + tag + " is selected"); configMappingEntity.setSelected(1); }
>
> }
> }
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterDAO.java 6e55128
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java 899ffe8
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java a48fb54
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/ServiceConfigDAOTest.java 4c186b5
>
> Diff: https://reviews.apache.org/r/38269/diff/
>
>
> Testing
> -------
>
> unit test: added more test cases to ambari-server code
> test on a rolling upgrade then downgrade.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 38269: AMBARI-12949: Downgrade fails with err: 500
status
code received on POST method for API: /api/vi/clusters/mycluster/upgrades
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38269/#review99219
-----------------------------------------------------------
Ship it!
Ship It!
- Nate Cole
On Sept. 15, 2015, 5:12 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38269/
> -----------------------------------------------------------
>
> (Updated Sept. 15, 2015, 5:12 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate Cole.
>
>
> Bugs: AMBARI-12949
> https://issues.apache.org/jira/browse/AMBARI-12949
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This seems to be an issue that happens when the "latest" configuration stored in the "clusterconfig" table is for a conf group instead of the default configurations. The named query "ClusterConfigEntity.findLatestConfigsByStack" returns the latest configuration to be set with "selected = 1" in the clusterconfigmapping table. When the "latest" config is for a conf group, this record will not have a corresponding entry in the clusterconfigmapping table. So the clusterconfigmapping table is left with a type that does not have the selected column set to 1.
>
> My proposed fix is the new named query "ClusterConfigEntity.findLatestConfigsOfClusterConfigMappingEntriesByStack" in ClusterConfigEntity.java, which searches the clusterconfig table for the latest version of a type_name and version_tag specified in the clusterconfigmapping table. This is different from the current "ClusterConfigEntity.findLatestConfigsByStack" query, which simply pulls the latest config of a type_name from the clusterconfig table regardless whether the corresponding version_tag is in the clusterconfigmapping table or not.
>
>
> Investigation:
>
> I used Oozie for debugging.
>
> When a user clicks the Downgrade button from the Finalize page. The request eventually hits ClusterImpl.applyLatestConfigurations method where the Ambari server updates the database to reverse back to the "latest" configuration of each type for the older stack (HDP 2.2)
>
> The workflow goes as
> 1. Set ALL entries in clusterconfigmapping table to have selected value set to 0
> 2. Ambari server uses ClusterImpl.applyLatestConfigurations method to get the "latest" configuration of each type for the older stack (HDP 2.2)
> 3. Loop through all the entries in the clusterconfigmapping table, and set the selected to 1 if this entry also happens to be in the "latest" configuration returned in step 2.
>
> The ClusterImpl.applyLatestConfigurations method calls ClusterDAO.java getLatestConfigurations to get the latest configuration for each type for the older stack. The DAO class simply runs the following SQL query (cluster id and stack id are the parameters) and passes the results back.
>
> SELECT clusterConfig1 FROM clusterconfig clusterConfig1 WHERE clusterConfig1.cluster_id='2' AND clusterConfig1.create_timestamp = (SELECT MAX(clusterConfig2.create_timestamp) FROM clusterconfig clusterConfig2 WHERE clusterConfig2.cluster_id='2' AND clusterConfig2.stack_id= '3' AND clusterConfig2.type_name = clusterConfig1.type_name);
>
> For downgrade, in our particular use case, the oozie-env configuration stored in the clusterconfig table, when the stack id is the HDP 2.2 version, only contains TWO conf, the original version1 and the one for the config group.
>
> The query above will return the entry corresponding to the oozie config group, as it was the last configuration entry added to the clusterconfig table for oozie-site with stack id = 3. However, because the config group entry is NEVER stored into the clusterconfigmapping table, the logic in ClusterImpl.applyLatestConfigurations method to re-select a version for a type in the clusterconfigmapping will not be able to update the clusterconfigmapping table. Thus no oozie-site is set with selected = 1 in the clusterconfigmapping table.
>
> for( ClusterConfigEntity clusterConfigToMakeSelected : clusterConfigsToMakeSelected ){
> LOG.info("Checking " + clusterConfigToMakeSelected.getType() + " with tag " + clusterConfigToMakeSelected.getTag());
> for (ClusterConfigMappingEntity configMappingEntity : configMappingEntities) {
> String tag = configMappingEntity.getTag();
> String type = configMappingEntity.getType();
>
> if (clusterConfigToMakeSelected.getTag().equals(tag)
> && clusterConfigToMakeSelected.getType().equals(type))
> { LOG.info(type + " with tag " + tag + " is selected"); configMappingEntity.setSelected(1); }
>
> }
> }
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterDAO.java 6e55128
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java 899ffe8
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java a48fb54
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/ServiceConfigDAOTest.java 4c186b5
>
> Diff: https://reviews.apache.org/r/38269/diff/
>
>
> Testing
> -------
>
> unit test: added more test cases to ambari-server code
> test on a rolling upgrade then downgrade.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 38269: AMBARI-12949: Downgrade fails with err: 500
status
code received on POST method for API: /api/vi/clusters/mycluster/upgrades
Posted by Di Li <di...@ca.ibm.com>.
> On Sept. 18, 2015, 12:36 p.m., Jonathan Hurley wrote:
> > Ship It!
Thanks Jonathan. Could you please help commit the change to trunk? I don't have permissions to do it.
- Di
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38269/#review99527
-----------------------------------------------------------
On Sept. 17, 2015, 5:07 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38269/
> -----------------------------------------------------------
>
> (Updated Sept. 17, 2015, 5:07 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate Cole.
>
>
> Bugs: AMBARI-12949
> https://issues.apache.org/jira/browse/AMBARI-12949
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This seems to be an issue that happens when the "latest" configuration stored in the "clusterconfig" table is for a conf group instead of the default configurations. The named query "ClusterConfigEntity.findLatestConfigsByStack" returns the latest configuration to be set with "selected = 1" in the clusterconfigmapping table. When the "latest" config is for a conf group, this record will not have a corresponding entry in the clusterconfigmapping table. So the clusterconfigmapping table is left with a type that does not have the selected column set to 1.
>
> My proposed fix is the new named query "ClusterConfigEntity.findLatestConfigsOfClusterConfigMappingEntriesByStack" in ClusterConfigEntity.java, which searches the clusterconfig table for the latest version of a type_name and version_tag specified in the clusterconfigmapping table. This is different from the current "ClusterConfigEntity.findLatestConfigsByStack" query, which simply pulls the latest config of a type_name from the clusterconfig table regardless whether the corresponding version_tag is in the clusterconfigmapping table or not.
>
>
> Investigation:
>
> I used Oozie for debugging.
>
> When a user clicks the Downgrade button from the Finalize page. The request eventually hits ClusterImpl.applyLatestConfigurations method where the Ambari server updates the database to reverse back to the "latest" configuration of each type for the older stack (HDP 2.2)
>
> The workflow goes as
> 1. Set ALL entries in clusterconfigmapping table to have selected value set to 0
> 2. Ambari server uses ClusterImpl.applyLatestConfigurations method to get the "latest" configuration of each type for the older stack (HDP 2.2)
> 3. Loop through all the entries in the clusterconfigmapping table, and set the selected to 1 if this entry also happens to be in the "latest" configuration returned in step 2.
>
> The ClusterImpl.applyLatestConfigurations method calls ClusterDAO.java getLatestConfigurations to get the latest configuration for each type for the older stack. The DAO class simply runs the following SQL query (cluster id and stack id are the parameters) and passes the results back.
>
> SELECT clusterConfig1 FROM clusterconfig clusterConfig1 WHERE clusterConfig1.cluster_id='2' AND clusterConfig1.create_timestamp = (SELECT MAX(clusterConfig2.create_timestamp) FROM clusterconfig clusterConfig2 WHERE clusterConfig2.cluster_id='2' AND clusterConfig2.stack_id= '3' AND clusterConfig2.type_name = clusterConfig1.type_name);
>
> For downgrade, in our particular use case, the oozie-env configuration stored in the clusterconfig table, when the stack id is the HDP 2.2 version, only contains TWO conf, the original version1 and the one for the config group.
>
> The query above will return the entry corresponding to the oozie config group, as it was the last configuration entry added to the clusterconfig table for oozie-site with stack id = 3. However, because the config group entry is NEVER stored into the clusterconfigmapping table, the logic in ClusterImpl.applyLatestConfigurations method to re-select a version for a type in the clusterconfigmapping will not be able to update the clusterconfigmapping table. Thus no oozie-site is set with selected = 1 in the clusterconfigmapping table.
>
> for( ClusterConfigEntity clusterConfigToMakeSelected : clusterConfigsToMakeSelected ){
> LOG.info("Checking " + clusterConfigToMakeSelected.getType() + " with tag " + clusterConfigToMakeSelected.getTag());
> for (ClusterConfigMappingEntity configMappingEntity : configMappingEntities) {
> String tag = configMappingEntity.getTag();
> String type = configMappingEntity.getType();
>
> if (clusterConfigToMakeSelected.getTag().equals(tag)
> && clusterConfigToMakeSelected.getType().equals(type))
> { LOG.info(type + " with tag " + tag + " is selected"); configMappingEntity.setSelected(1); }
>
> }
> }
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterDAO.java 6e55128
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java 899ffe8
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java 4b20cba
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/ServiceConfigDAOTest.java 4c186b5
>
> Diff: https://reviews.apache.org/r/38269/diff/
>
>
> Testing
> -------
>
> unit test: added more test cases to ambari-server code
> test on a rolling upgrade then downgrade.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 38269: AMBARI-12949: Downgrade fails with err: 500
status
code received on POST method for API: /api/vi/clusters/mycluster/upgrades
Posted by Jonathan Hurley <jh...@hortonworks.com>.
> On Sept. 18, 2015, 8:36 a.m., Jonathan Hurley wrote:
> > Ship It!
>
> Di Li wrote:
> Thanks Jonathan. Could you please help commit the change to trunk? I don't have permissions to do it.
Pushed. Please close the review.
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38269/#review99527
-----------------------------------------------------------
On Sept. 17, 2015, 1:07 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38269/
> -----------------------------------------------------------
>
> (Updated Sept. 17, 2015, 1:07 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate Cole.
>
>
> Bugs: AMBARI-12949
> https://issues.apache.org/jira/browse/AMBARI-12949
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This seems to be an issue that happens when the "latest" configuration stored in the "clusterconfig" table is for a conf group instead of the default configurations. The named query "ClusterConfigEntity.findLatestConfigsByStack" returns the latest configuration to be set with "selected = 1" in the clusterconfigmapping table. When the "latest" config is for a conf group, this record will not have a corresponding entry in the clusterconfigmapping table. So the clusterconfigmapping table is left with a type that does not have the selected column set to 1.
>
> My proposed fix is the new named query "ClusterConfigEntity.findLatestConfigsOfClusterConfigMappingEntriesByStack" in ClusterConfigEntity.java, which searches the clusterconfig table for the latest version of a type_name and version_tag specified in the clusterconfigmapping table. This is different from the current "ClusterConfigEntity.findLatestConfigsByStack" query, which simply pulls the latest config of a type_name from the clusterconfig table regardless whether the corresponding version_tag is in the clusterconfigmapping table or not.
>
>
> Investigation:
>
> I used Oozie for debugging.
>
> When a user clicks the Downgrade button from the Finalize page. The request eventually hits ClusterImpl.applyLatestConfigurations method where the Ambari server updates the database to reverse back to the "latest" configuration of each type for the older stack (HDP 2.2)
>
> The workflow goes as
> 1. Set ALL entries in clusterconfigmapping table to have selected value set to 0
> 2. Ambari server uses ClusterImpl.applyLatestConfigurations method to get the "latest" configuration of each type for the older stack (HDP 2.2)
> 3. Loop through all the entries in the clusterconfigmapping table, and set the selected to 1 if this entry also happens to be in the "latest" configuration returned in step 2.
>
> The ClusterImpl.applyLatestConfigurations method calls ClusterDAO.java getLatestConfigurations to get the latest configuration for each type for the older stack. The DAO class simply runs the following SQL query (cluster id and stack id are the parameters) and passes the results back.
>
> SELECT clusterConfig1 FROM clusterconfig clusterConfig1 WHERE clusterConfig1.cluster_id='2' AND clusterConfig1.create_timestamp = (SELECT MAX(clusterConfig2.create_timestamp) FROM clusterconfig clusterConfig2 WHERE clusterConfig2.cluster_id='2' AND clusterConfig2.stack_id= '3' AND clusterConfig2.type_name = clusterConfig1.type_name);
>
> For downgrade, in our particular use case, the oozie-env configuration stored in the clusterconfig table, when the stack id is the HDP 2.2 version, only contains TWO conf, the original version1 and the one for the config group.
>
> The query above will return the entry corresponding to the oozie config group, as it was the last configuration entry added to the clusterconfig table for oozie-site with stack id = 3. However, because the config group entry is NEVER stored into the clusterconfigmapping table, the logic in ClusterImpl.applyLatestConfigurations method to re-select a version for a type in the clusterconfigmapping will not be able to update the clusterconfigmapping table. Thus no oozie-site is set with selected = 1 in the clusterconfigmapping table.
>
> for( ClusterConfigEntity clusterConfigToMakeSelected : clusterConfigsToMakeSelected ){
> LOG.info("Checking " + clusterConfigToMakeSelected.getType() + " with tag " + clusterConfigToMakeSelected.getTag());
> for (ClusterConfigMappingEntity configMappingEntity : configMappingEntities) {
> String tag = configMappingEntity.getTag();
> String type = configMappingEntity.getType();
>
> if (clusterConfigToMakeSelected.getTag().equals(tag)
> && clusterConfigToMakeSelected.getType().equals(type))
> { LOG.info(type + " with tag " + tag + " is selected"); configMappingEntity.setSelected(1); }
>
> }
> }
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterDAO.java 6e55128
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java 899ffe8
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java 4b20cba
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/ServiceConfigDAOTest.java 4c186b5
>
> Diff: https://reviews.apache.org/r/38269/diff/
>
>
> Testing
> -------
>
> unit test: added more test cases to ambari-server code
> test on a rolling upgrade then downgrade.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 38269: AMBARI-12949: Downgrade fails with err: 500
status
code received on POST method for API: /api/vi/clusters/mycluster/upgrades
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38269/#review99527
-----------------------------------------------------------
Ship it!
Ship It!
- Jonathan Hurley
On Sept. 17, 2015, 1:07 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38269/
> -----------------------------------------------------------
>
> (Updated Sept. 17, 2015, 1:07 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate Cole.
>
>
> Bugs: AMBARI-12949
> https://issues.apache.org/jira/browse/AMBARI-12949
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This seems to be an issue that happens when the "latest" configuration stored in the "clusterconfig" table is for a conf group instead of the default configurations. The named query "ClusterConfigEntity.findLatestConfigsByStack" returns the latest configuration to be set with "selected = 1" in the clusterconfigmapping table. When the "latest" config is for a conf group, this record will not have a corresponding entry in the clusterconfigmapping table. So the clusterconfigmapping table is left with a type that does not have the selected column set to 1.
>
> My proposed fix is the new named query "ClusterConfigEntity.findLatestConfigsOfClusterConfigMappingEntriesByStack" in ClusterConfigEntity.java, which searches the clusterconfig table for the latest version of a type_name and version_tag specified in the clusterconfigmapping table. This is different from the current "ClusterConfigEntity.findLatestConfigsByStack" query, which simply pulls the latest config of a type_name from the clusterconfig table regardless whether the corresponding version_tag is in the clusterconfigmapping table or not.
>
>
> Investigation:
>
> I used Oozie for debugging.
>
> When a user clicks the Downgrade button from the Finalize page. The request eventually hits ClusterImpl.applyLatestConfigurations method where the Ambari server updates the database to reverse back to the "latest" configuration of each type for the older stack (HDP 2.2)
>
> The workflow goes as
> 1. Set ALL entries in clusterconfigmapping table to have selected value set to 0
> 2. Ambari server uses ClusterImpl.applyLatestConfigurations method to get the "latest" configuration of each type for the older stack (HDP 2.2)
> 3. Loop through all the entries in the clusterconfigmapping table, and set the selected to 1 if this entry also happens to be in the "latest" configuration returned in step 2.
>
> The ClusterImpl.applyLatestConfigurations method calls ClusterDAO.java getLatestConfigurations to get the latest configuration for each type for the older stack. The DAO class simply runs the following SQL query (cluster id and stack id are the parameters) and passes the results back.
>
> SELECT clusterConfig1 FROM clusterconfig clusterConfig1 WHERE clusterConfig1.cluster_id='2' AND clusterConfig1.create_timestamp = (SELECT MAX(clusterConfig2.create_timestamp) FROM clusterconfig clusterConfig2 WHERE clusterConfig2.cluster_id='2' AND clusterConfig2.stack_id= '3' AND clusterConfig2.type_name = clusterConfig1.type_name);
>
> For downgrade, in our particular use case, the oozie-env configuration stored in the clusterconfig table, when the stack id is the HDP 2.2 version, only contains TWO conf, the original version1 and the one for the config group.
>
> The query above will return the entry corresponding to the oozie config group, as it was the last configuration entry added to the clusterconfig table for oozie-site with stack id = 3. However, because the config group entry is NEVER stored into the clusterconfigmapping table, the logic in ClusterImpl.applyLatestConfigurations method to re-select a version for a type in the clusterconfigmapping will not be able to update the clusterconfigmapping table. Thus no oozie-site is set with selected = 1 in the clusterconfigmapping table.
>
> for( ClusterConfigEntity clusterConfigToMakeSelected : clusterConfigsToMakeSelected ){
> LOG.info("Checking " + clusterConfigToMakeSelected.getType() + " with tag " + clusterConfigToMakeSelected.getTag());
> for (ClusterConfigMappingEntity configMappingEntity : configMappingEntities) {
> String tag = configMappingEntity.getTag();
> String type = configMappingEntity.getType();
>
> if (clusterConfigToMakeSelected.getTag().equals(tag)
> && clusterConfigToMakeSelected.getType().equals(type))
> { LOG.info(type + " with tag " + tag + " is selected"); configMappingEntity.setSelected(1); }
>
> }
> }
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterDAO.java 6e55128
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java 899ffe8
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java 4b20cba
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/ServiceConfigDAOTest.java 4c186b5
>
> Diff: https://reviews.apache.org/r/38269/diff/
>
>
> Testing
> -------
>
> unit test: added more test cases to ambari-server code
> test on a rolling upgrade then downgrade.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 38269: AMBARI-12949: Downgrade fails with err: 500
status
code received on POST method for API: /api/vi/clusters/mycluster/upgrades
Posted by Di Li <di...@ca.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38269/
-----------------------------------------------------------
(Updated Sept. 17, 2015, 5:07 p.m.)
Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate Cole.
Bugs: AMBARI-12949
https://issues.apache.org/jira/browse/AMBARI-12949
Repository: ambari
Description
-------
This seems to be an issue that happens when the "latest" configuration stored in the "clusterconfig" table is for a conf group instead of the default configurations. The named query "ClusterConfigEntity.findLatestConfigsByStack" returns the latest configuration to be set with "selected = 1" in the clusterconfigmapping table. When the "latest" config is for a conf group, this record will not have a corresponding entry in the clusterconfigmapping table. So the clusterconfigmapping table is left with a type that does not have the selected column set to 1.
My proposed fix is the new named query "ClusterConfigEntity.findLatestConfigsOfClusterConfigMappingEntriesByStack" in ClusterConfigEntity.java, which searches the clusterconfig table for the latest version of a type_name and version_tag specified in the clusterconfigmapping table. This is different from the current "ClusterConfigEntity.findLatestConfigsByStack" query, which simply pulls the latest config of a type_name from the clusterconfig table regardless whether the corresponding version_tag is in the clusterconfigmapping table or not.
Investigation:
I used Oozie for debugging.
When a user clicks the Downgrade button from the Finalize page. The request eventually hits ClusterImpl.applyLatestConfigurations method where the Ambari server updates the database to reverse back to the "latest" configuration of each type for the older stack (HDP 2.2)
The workflow goes as
1. Set ALL entries in clusterconfigmapping table to have selected value set to 0
2. Ambari server uses ClusterImpl.applyLatestConfigurations method to get the "latest" configuration of each type for the older stack (HDP 2.2)
3. Loop through all the entries in the clusterconfigmapping table, and set the selected to 1 if this entry also happens to be in the "latest" configuration returned in step 2.
The ClusterImpl.applyLatestConfigurations method calls ClusterDAO.java getLatestConfigurations to get the latest configuration for each type for the older stack. The DAO class simply runs the following SQL query (cluster id and stack id are the parameters) and passes the results back.
SELECT clusterConfig1 FROM clusterconfig clusterConfig1 WHERE clusterConfig1.cluster_id='2' AND clusterConfig1.create_timestamp = (SELECT MAX(clusterConfig2.create_timestamp) FROM clusterconfig clusterConfig2 WHERE clusterConfig2.cluster_id='2' AND clusterConfig2.stack_id= '3' AND clusterConfig2.type_name = clusterConfig1.type_name);
For downgrade, in our particular use case, the oozie-env configuration stored in the clusterconfig table, when the stack id is the HDP 2.2 version, only contains TWO conf, the original version1 and the one for the config group.
The query above will return the entry corresponding to the oozie config group, as it was the last configuration entry added to the clusterconfig table for oozie-site with stack id = 3. However, because the config group entry is NEVER stored into the clusterconfigmapping table, the logic in ClusterImpl.applyLatestConfigurations method to re-select a version for a type in the clusterconfigmapping will not be able to update the clusterconfigmapping table. Thus no oozie-site is set with selected = 1 in the clusterconfigmapping table.
for( ClusterConfigEntity clusterConfigToMakeSelected : clusterConfigsToMakeSelected ){
LOG.info("Checking " + clusterConfigToMakeSelected.getType() + " with tag " + clusterConfigToMakeSelected.getTag());
for (ClusterConfigMappingEntity configMappingEntity : configMappingEntities) {
String tag = configMappingEntity.getTag();
String type = configMappingEntity.getType();
if (clusterConfigToMakeSelected.getTag().equals(tag)
&& clusterConfigToMakeSelected.getType().equals(type))
{ LOG.info(type + " with tag " + tag + " is selected"); configMappingEntity.setSelected(1); }
}
}
Diffs (updated)
-----
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterDAO.java 6e55128
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java 899ffe8
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java 4b20cba
ambari-server/src/test/java/org/apache/ambari/server/orm/dao/ServiceConfigDAOTest.java 4c186b5
Diff: https://reviews.apache.org/r/38269/diff/
Testing
-------
unit test: added more test cases to ambari-server code
test on a rolling upgrade then downgrade.
Thanks,
Di Li
Re: Review Request 38269: AMBARI-12949: Downgrade fails with err: 500
status
code received on POST method for API: /api/vi/clusters/mycluster/upgrades
Posted by Di Li <di...@ca.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38269/
-----------------------------------------------------------
(Updated Sept. 16, 2015, 5:14 p.m.)
Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate Cole.
Changes
-------
used {} placeholders in the LOG output.
Bugs: AMBARI-12949
https://issues.apache.org/jira/browse/AMBARI-12949
Repository: ambari
Description
-------
This seems to be an issue that happens when the "latest" configuration stored in the "clusterconfig" table is for a conf group instead of the default configurations. The named query "ClusterConfigEntity.findLatestConfigsByStack" returns the latest configuration to be set with "selected = 1" in the clusterconfigmapping table. When the "latest" config is for a conf group, this record will not have a corresponding entry in the clusterconfigmapping table. So the clusterconfigmapping table is left with a type that does not have the selected column set to 1.
My proposed fix is the new named query "ClusterConfigEntity.findLatestConfigsOfClusterConfigMappingEntriesByStack" in ClusterConfigEntity.java, which searches the clusterconfig table for the latest version of a type_name and version_tag specified in the clusterconfigmapping table. This is different from the current "ClusterConfigEntity.findLatestConfigsByStack" query, which simply pulls the latest config of a type_name from the clusterconfig table regardless whether the corresponding version_tag is in the clusterconfigmapping table or not.
Investigation:
I used Oozie for debugging.
When a user clicks the Downgrade button from the Finalize page. The request eventually hits ClusterImpl.applyLatestConfigurations method where the Ambari server updates the database to reverse back to the "latest" configuration of each type for the older stack (HDP 2.2)
The workflow goes as
1. Set ALL entries in clusterconfigmapping table to have selected value set to 0
2. Ambari server uses ClusterImpl.applyLatestConfigurations method to get the "latest" configuration of each type for the older stack (HDP 2.2)
3. Loop through all the entries in the clusterconfigmapping table, and set the selected to 1 if this entry also happens to be in the "latest" configuration returned in step 2.
The ClusterImpl.applyLatestConfigurations method calls ClusterDAO.java getLatestConfigurations to get the latest configuration for each type for the older stack. The DAO class simply runs the following SQL query (cluster id and stack id are the parameters) and passes the results back.
SELECT clusterConfig1 FROM clusterconfig clusterConfig1 WHERE clusterConfig1.cluster_id='2' AND clusterConfig1.create_timestamp = (SELECT MAX(clusterConfig2.create_timestamp) FROM clusterconfig clusterConfig2 WHERE clusterConfig2.cluster_id='2' AND clusterConfig2.stack_id= '3' AND clusterConfig2.type_name = clusterConfig1.type_name);
For downgrade, in our particular use case, the oozie-env configuration stored in the clusterconfig table, when the stack id is the HDP 2.2 version, only contains TWO conf, the original version1 and the one for the config group.
The query above will return the entry corresponding to the oozie config group, as it was the last configuration entry added to the clusterconfig table for oozie-site with stack id = 3. However, because the config group entry is NEVER stored into the clusterconfigmapping table, the logic in ClusterImpl.applyLatestConfigurations method to re-select a version for a type in the clusterconfigmapping will not be able to update the clusterconfigmapping table. Thus no oozie-site is set with selected = 1 in the clusterconfigmapping table.
for( ClusterConfigEntity clusterConfigToMakeSelected : clusterConfigsToMakeSelected ){
LOG.info("Checking " + clusterConfigToMakeSelected.getType() + " with tag " + clusterConfigToMakeSelected.getTag());
for (ClusterConfigMappingEntity configMappingEntity : configMappingEntities) {
String tag = configMappingEntity.getTag();
String type = configMappingEntity.getType();
if (clusterConfigToMakeSelected.getTag().equals(tag)
&& clusterConfigToMakeSelected.getType().equals(type))
{ LOG.info(type + " with tag " + tag + " is selected"); configMappingEntity.setSelected(1); }
}
}
Diffs (updated)
-----
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterDAO.java 6e55128
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java 899ffe8
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java a48fb54
ambari-server/src/test/java/org/apache/ambari/server/orm/dao/ServiceConfigDAOTest.java 4c186b5
Diff: https://reviews.apache.org/r/38269/diff/
Testing
-------
unit test: added more test cases to ambari-server code
test on a rolling upgrade then downgrade.
Thanks,
Di Li
Re: Review Request 38269: AMBARI-12949: Downgrade fails with err: 500
status
code received on POST method for API: /api/vi/clusters/mycluster/upgrades
Posted by Di Li <di...@ca.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38269/
-----------------------------------------------------------
(Updated Sept. 15, 2015, 9:12 p.m.)
Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate Cole.
Changes
-------
renamed the query, and java methods. Junit test.
Bugs: AMBARI-12949
https://issues.apache.org/jira/browse/AMBARI-12949
Repository: ambari
Description
-------
This seems to be an issue that happens when the "latest" configuration stored in the "clusterconfig" table is for a conf group instead of the default configurations. The named query "ClusterConfigEntity.findLatestConfigsByStack" returns the latest configuration to be set with "selected = 1" in the clusterconfigmapping table. When the "latest" config is for a conf group, this record will not have a corresponding entry in the clusterconfigmapping table. So the clusterconfigmapping table is left with a type that does not have the selected column set to 1.
My proposed fix is the new named query "ClusterConfigEntity.findLatestConfigsOfClusterConfigMappingEntriesByStack" in ClusterConfigEntity.java, which searches the clusterconfig table for the latest version of a type_name and version_tag specified in the clusterconfigmapping table. This is different from the current "ClusterConfigEntity.findLatestConfigsByStack" query, which simply pulls the latest config of a type_name from the clusterconfig table regardless whether the corresponding version_tag is in the clusterconfigmapping table or not.
Investigation:
I used Oozie for debugging.
When a user clicks the Downgrade button from the Finalize page. The request eventually hits ClusterImpl.applyLatestConfigurations method where the Ambari server updates the database to reverse back to the "latest" configuration of each type for the older stack (HDP 2.2)
The workflow goes as
1. Set ALL entries in clusterconfigmapping table to have selected value set to 0
2. Ambari server uses ClusterImpl.applyLatestConfigurations method to get the "latest" configuration of each type for the older stack (HDP 2.2)
3. Loop through all the entries in the clusterconfigmapping table, and set the selected to 1 if this entry also happens to be in the "latest" configuration returned in step 2.
The ClusterImpl.applyLatestConfigurations method calls ClusterDAO.java getLatestConfigurations to get the latest configuration for each type for the older stack. The DAO class simply runs the following SQL query (cluster id and stack id are the parameters) and passes the results back.
SELECT clusterConfig1 FROM clusterconfig clusterConfig1 WHERE clusterConfig1.cluster_id='2' AND clusterConfig1.create_timestamp = (SELECT MAX(clusterConfig2.create_timestamp) FROM clusterconfig clusterConfig2 WHERE clusterConfig2.cluster_id='2' AND clusterConfig2.stack_id= '3' AND clusterConfig2.type_name = clusterConfig1.type_name);
For downgrade, in our particular use case, the oozie-env configuration stored in the clusterconfig table, when the stack id is the HDP 2.2 version, only contains TWO conf, the original version1 and the one for the config group.
The query above will return the entry corresponding to the oozie config group, as it was the last configuration entry added to the clusterconfig table for oozie-site with stack id = 3. However, because the config group entry is NEVER stored into the clusterconfigmapping table, the logic in ClusterImpl.applyLatestConfigurations method to re-select a version for a type in the clusterconfigmapping will not be able to update the clusterconfigmapping table. Thus no oozie-site is set with selected = 1 in the clusterconfigmapping table.
for( ClusterConfigEntity clusterConfigToMakeSelected : clusterConfigsToMakeSelected ){
LOG.info("Checking " + clusterConfigToMakeSelected.getType() + " with tag " + clusterConfigToMakeSelected.getTag());
for (ClusterConfigMappingEntity configMappingEntity : configMappingEntities) {
String tag = configMappingEntity.getTag();
String type = configMappingEntity.getType();
if (clusterConfigToMakeSelected.getTag().equals(tag)
&& clusterConfigToMakeSelected.getType().equals(type))
{ LOG.info(type + " with tag " + tag + " is selected"); configMappingEntity.setSelected(1); }
}
}
Diffs (updated)
-----
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterDAO.java 6e55128
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java 899ffe8
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java a48fb54
ambari-server/src/test/java/org/apache/ambari/server/orm/dao/ServiceConfigDAOTest.java 4c186b5
Diff: https://reviews.apache.org/r/38269/diff/
Testing
-------
unit test: added more test cases to ambari-server code
test on a rolling upgrade then downgrade.
Thanks,
Di Li
Re: Review Request 38269: AMBARI-12949: Downgrade fails with err: 500
status
code received on POST method for API: /api/vi/clusters/mycluster/upgrades
Posted by Di Li <di...@ca.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38269/
-----------------------------------------------------------
(Updated Sept. 14, 2015, 7:09 p.m.)
Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Nate Cole.
Changes
-------
code clean up per comments
Bugs: AMBARI-12949
https://issues.apache.org/jira/browse/AMBARI-12949
Repository: ambari
Description
-------
This seems to be an issue that happens when the "latest" configuration stored in the "clusterconfig" table is for a conf group instead of the default configurations. The named query "ClusterConfigEntity.findLatestConfigsByStack" returns the latest configuration to be set with "selected = 1" in the clusterconfigmapping table. When the "latest" config is for a conf group, this record will not have a corresponding entry in the clusterconfigmapping table. So the clusterconfigmapping table is left with a type that does not have the selected column set to 1.
My proposed fix is the new named query "ClusterConfigEntity.findLatestConfigsOfClusterConfigMappingEntriesByStack" in ClusterConfigEntity.java, which searches the clusterconfig table for the latest version of a type_name and version_tag specified in the clusterconfigmapping table. This is different from the current "ClusterConfigEntity.findLatestConfigsByStack" query, which simply pulls the latest config of a type_name from the clusterconfig table regardless whether the corresponding version_tag is in the clusterconfigmapping table or not.
Investigation:
I used Oozie for debugging.
When a user clicks the Downgrade button from the Finalize page. The request eventually hits ClusterImpl.applyLatestConfigurations method where the Ambari server updates the database to reverse back to the "latest" configuration of each type for the older stack (HDP 2.2)
The workflow goes as
1. Set ALL entries in clusterconfigmapping table to have selected value set to 0
2. Ambari server uses ClusterImpl.applyLatestConfigurations method to get the "latest" configuration of each type for the older stack (HDP 2.2)
3. Loop through all the entries in the clusterconfigmapping table, and set the selected to 1 if this entry also happens to be in the "latest" configuration returned in step 2.
The ClusterImpl.applyLatestConfigurations method calls ClusterDAO.java getLatestConfigurations to get the latest configuration for each type for the older stack. The DAO class simply runs the following SQL query (cluster id and stack id are the parameters) and passes the results back.
SELECT clusterConfig1 FROM clusterconfig clusterConfig1 WHERE clusterConfig1.cluster_id='2' AND clusterConfig1.create_timestamp = (SELECT MAX(clusterConfig2.create_timestamp) FROM clusterconfig clusterConfig2 WHERE clusterConfig2.cluster_id='2' AND clusterConfig2.stack_id= '3' AND clusterConfig2.type_name = clusterConfig1.type_name);
For downgrade, in our particular use case, the oozie-env configuration stored in the clusterconfig table, when the stack id is the HDP 2.2 version, only contains TWO conf, the original version1 and the one for the config group.
The query above will return the entry corresponding to the oozie config group, as it was the last configuration entry added to the clusterconfig table for oozie-site with stack id = 3. However, because the config group entry is NEVER stored into the clusterconfigmapping table, the logic in ClusterImpl.applyLatestConfigurations method to re-select a version for a type in the clusterconfigmapping will not be able to update the clusterconfigmapping table. Thus no oozie-site is set with selected = 1 in the clusterconfigmapping table.
for( ClusterConfigEntity clusterConfigToMakeSelected : clusterConfigsToMakeSelected ){
LOG.info("Checking " + clusterConfigToMakeSelected.getType() + " with tag " + clusterConfigToMakeSelected.getTag());
for (ClusterConfigMappingEntity configMappingEntity : configMappingEntities) {
String tag = configMappingEntity.getTag();
String type = configMappingEntity.getType();
if (clusterConfigToMakeSelected.getTag().equals(tag)
&& clusterConfigToMakeSelected.getType().equals(type))
{ LOG.info(type + " with tag " + tag + " is selected"); configMappingEntity.setSelected(1); }
}
}
Diffs (updated)
-----
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterDAO.java 6e55128
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterConfigEntity.java 899ffe8
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java a48fb54
ambari-server/src/test/java/org/apache/ambari/server/orm/dao/ServiceConfigDAOTest.java 4c186b5
Diff: https://reviews.apache.org/r/38269/diff/
Testing
-------
unit test: added more test cases to ambari-server code
test on a rolling upgrade then downgrade.
Thanks,
Di Li