You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Jonathan Hurley <jh...@hortonworks.com> on 2015/12/04 21:48:47 UTC
Review Request 40986: Express Upgrades Fail To Stop Services Because
Of Destroyed Configurations
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40986/
-----------------------------------------------------------
Review request for Ambari, Jayush Luniya, Nate Cole, and Sumit Mohanty.
Bugs: AMBARI-14220
https://issues.apache.org/jira/browse/AMBARI-14220
Repository: ambari
Description
-------
AMBARI-13824 introduced a regression caused by additions of code to conf_select which destroyed configuration directories. Long story short is that conf_select should be told to create symlinks and move configurations around; it should not do it on every select() call.
I removed the offending code and have install_packages determine if symlinking should be done in conf_select.
Diffs
-----
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 24ca151
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java d299602
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 69709e5
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 0170806
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py 3961897
Diff: https://reviews.apache.org/r/40986/diff/
Testing
-------
Upgrades from 2.1 to 2.3, 2.2 to 2.3, and 2.3 to 2.3; no side effects yet.
Working on tests...
Thanks,
Jonathan Hurley
Re: Review Request 40986: Express Upgrades Fail To Stop Services
Because Of Destroyed Configurations
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40986/#review109406
-----------------------------------------------------------
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py (line 250)
<https://reviews.apache.org/r/40986/#comment168900>
This file contains a fair bit of changes to this logic. It's unfortunate that we regressed here while trying to fix another issue.
Let's keep an eye out for EU/RU on all of the paths.
- Alejandro Fernandez
On Dec. 4, 2015, 10:02 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40986/
> -----------------------------------------------------------
>
> (Updated Dec. 4, 2015, 10:02 p.m.)
>
>
> Review request for Ambari, Jayush Luniya, Nate Cole, and Sumit Mohanty.
>
>
> Bugs: AMBARI-14220
> https://issues.apache.org/jira/browse/AMBARI-14220
>
>
> Repository: ambari
>
>
> Description
> -------
>
> AMBARI-13824 introduced a regression caused by additions of code to conf_select which destroyed configuration directories. Long story short is that conf_select should be told to create symlinks and move configurations around; it should not do it on every select() call.
>
> I removed the offending code and have install_packages determine if symlinking should be done in conf_select.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 24ca151
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java d299602
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 69709e5
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 0170806
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py 3961897
> ambari-server/src/test/python/stacks/2.0.6/HBASE/test_hbase_client.py 3d55c31
> ambari-server/src/test/python/stacks/2.0.6/HBASE/test_hbase_master.py 2d422d4
> ambari-server/src/test/python/stacks/2.0.6/HBASE/test_hbase_regionserver.py 3ec3b83
> ambari-server/src/test/python/stacks/2.0.6/HBASE/test_phoenix_queryserver.py 56900c8
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_datanode.py c94948c
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py eade3b9
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_journalnode.py f21f3dd
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_nfsgateway.py 47cb378
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_client.py 2788a04
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 69db9ee
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_webhcat_server.py af16a33
> ambari-server/src/test/python/stacks/2.0.6/PIG/test_pig_client.py 85fbbad
> ambari-server/src/test/python/stacks/2.0.6/SQOOP/test_sqoop.py cf94b48
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_historyserver.py 88fbba7
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_mapreduce2_client.py 361376d
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_nodemanager.py 6973ccf
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_resourcemanager.py 2d24059
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_yarn_client.py 80d4ef1
> ambari-server/src/test/python/stacks/2.0.6/ZOOKEEPER/test_zookeeper_client.py 63c6fd8
> ambari-server/src/test/python/stacks/2.0.6/ZOOKEEPER/test_zookeeper_server.py f83df6f
> ambari-server/src/test/python/stacks/2.1/FALCON/test_falcon_client.py c10ee8a
> ambari-server/src/test/python/stacks/2.1/FALCON/test_falcon_server.py 6eb60ce
> ambari-server/src/test/python/stacks/2.1/HIVE/test_hive_metastore.py 514633c
> ambari-server/src/test/python/stacks/2.1/STORM/test_storm_drpc_server.py cef5b97
> ambari-server/src/test/python/stacks/2.1/STORM/test_storm_nimbus.py 8ede94c
> ambari-server/src/test/python/stacks/2.1/STORM/test_storm_nimbus_prod.py 2aa3df5
> ambari-server/src/test/python/stacks/2.1/STORM/test_storm_supervisor.py d8c79ac
> ambari-server/src/test/python/stacks/2.1/STORM/test_storm_supervisor_prod.py ff43e43
> ambari-server/src/test/python/stacks/2.1/STORM/test_storm_ui_server.py dc00139
> ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py de9e7cc
> ambari-server/src/test/python/stacks/2.1/YARN/test_apptimelineserver.py 6118cc8
> ambari-server/src/test/python/stacks/2.2/ACCUMULO/test_accumulo_client.py f3f8d67
> ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 2259b47
> ambari-server/src/test/python/stacks/2.2/KNOX/test_knox_gateway.py 4edb7eb
> ambari-server/src/test/python/stacks/2.2/RANGER/test_ranger_admin.py 80bbbaf
> ambari-server/src/test/python/stacks/2.2/RANGER/test_ranger_usersync.py 5cba731
> ambari-server/src/test/python/stacks/2.2/SLIDER/test_slider_client.py 3d01993
> ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py a3319aa
> ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py 385b2d7
> ambari-server/src/test/python/stacks/2.3/MAHOUT/test_mahout_client.py 06645a9
> ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 1968f3b
>
> Diff: https://reviews.apache.org/r/40986/diff/
>
>
> Testing
> -------
>
> Upgrades from 2.1 to 2.3, 2.2 to 2.3, and 2.3 to 2.3; no side effects yet.
>
> OK
> ----------------------------------------------------------------------
> Total run:845
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 40986: Express Upgrades Fail To Stop Services
Because Of Destroyed Configurations
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40986/
-----------------------------------------------------------
(Updated Dec. 4, 2015, 5:02 p.m.)
Review request for Ambari, Jayush Luniya, Nate Cole, and Sumit Mohanty.
Bugs: AMBARI-14220
https://issues.apache.org/jira/browse/AMBARI-14220
Repository: ambari
Description
-------
AMBARI-13824 introduced a regression caused by additions of code to conf_select which destroyed configuration directories. Long story short is that conf_select should be told to create symlinks and move configurations around; it should not do it on every select() call.
I removed the offending code and have install_packages determine if symlinking should be done in conf_select.
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 24ca151
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java d299602
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 69709e5
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 0170806
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py 3961897
ambari-server/src/test/python/stacks/2.0.6/HBASE/test_hbase_client.py 3d55c31
ambari-server/src/test/python/stacks/2.0.6/HBASE/test_hbase_master.py 2d422d4
ambari-server/src/test/python/stacks/2.0.6/HBASE/test_hbase_regionserver.py 3ec3b83
ambari-server/src/test/python/stacks/2.0.6/HBASE/test_phoenix_queryserver.py 56900c8
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_datanode.py c94948c
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py eade3b9
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_journalnode.py f21f3dd
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_nfsgateway.py 47cb378
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_client.py 2788a04
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 69db9ee
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_webhcat_server.py af16a33
ambari-server/src/test/python/stacks/2.0.6/PIG/test_pig_client.py 85fbbad
ambari-server/src/test/python/stacks/2.0.6/SQOOP/test_sqoop.py cf94b48
ambari-server/src/test/python/stacks/2.0.6/YARN/test_historyserver.py 88fbba7
ambari-server/src/test/python/stacks/2.0.6/YARN/test_mapreduce2_client.py 361376d
ambari-server/src/test/python/stacks/2.0.6/YARN/test_nodemanager.py 6973ccf
ambari-server/src/test/python/stacks/2.0.6/YARN/test_resourcemanager.py 2d24059
ambari-server/src/test/python/stacks/2.0.6/YARN/test_yarn_client.py 80d4ef1
ambari-server/src/test/python/stacks/2.0.6/ZOOKEEPER/test_zookeeper_client.py 63c6fd8
ambari-server/src/test/python/stacks/2.0.6/ZOOKEEPER/test_zookeeper_server.py f83df6f
ambari-server/src/test/python/stacks/2.1/FALCON/test_falcon_client.py c10ee8a
ambari-server/src/test/python/stacks/2.1/FALCON/test_falcon_server.py 6eb60ce
ambari-server/src/test/python/stacks/2.1/HIVE/test_hive_metastore.py 514633c
ambari-server/src/test/python/stacks/2.1/STORM/test_storm_drpc_server.py cef5b97
ambari-server/src/test/python/stacks/2.1/STORM/test_storm_nimbus.py 8ede94c
ambari-server/src/test/python/stacks/2.1/STORM/test_storm_nimbus_prod.py 2aa3df5
ambari-server/src/test/python/stacks/2.1/STORM/test_storm_supervisor.py d8c79ac
ambari-server/src/test/python/stacks/2.1/STORM/test_storm_supervisor_prod.py ff43e43
ambari-server/src/test/python/stacks/2.1/STORM/test_storm_ui_server.py dc00139
ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py de9e7cc
ambari-server/src/test/python/stacks/2.1/YARN/test_apptimelineserver.py 6118cc8
ambari-server/src/test/python/stacks/2.2/ACCUMULO/test_accumulo_client.py f3f8d67
ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 2259b47
ambari-server/src/test/python/stacks/2.2/KNOX/test_knox_gateway.py 4edb7eb
ambari-server/src/test/python/stacks/2.2/RANGER/test_ranger_admin.py 80bbbaf
ambari-server/src/test/python/stacks/2.2/RANGER/test_ranger_usersync.py 5cba731
ambari-server/src/test/python/stacks/2.2/SLIDER/test_slider_client.py 3d01993
ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py a3319aa
ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py 385b2d7
ambari-server/src/test/python/stacks/2.3/MAHOUT/test_mahout_client.py 06645a9
ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 1968f3b
Diff: https://reviews.apache.org/r/40986/diff/
Testing (updated)
-------
Upgrades from 2.1 to 2.3, 2.2 to 2.3, and 2.3 to 2.3; no side effects yet.
OK
----------------------------------------------------------------------
Total run:845
Total errors:0
Total failures:0
OK
Thanks,
Jonathan Hurley
Re: Review Request 40986: Express Upgrades Fail To Stop Services
Because Of Destroyed Configurations
Posted by Jonathan Hurley <jh...@hortonworks.com>.
> On Dec. 4, 2015, 4:51 p.m., Nate Cole wrote:
> > RU has a task that will rollback directories:
> >
> > <execute-stage title="Restore configuration directories and remove HDP 2.3 symlinks">
> > <task xsi:type="execute">
> > <script>scripts/ru_set_all.py</script>
> > <function>unlink_all_configs</function>
> > </task>
> > </execute-stage>
> >
> > Do we need anything special in this case when downgrading to HDP-2.1 and 2.2? (I doubt it, but just bringing it up)
There's no downgrade to 2.1 because you deleted it.
For downgrade to 2.2, I believe we're OK but I will test this.
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40986/#review109034
-----------------------------------------------------------
On Dec. 4, 2015, 3:48 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40986/
> -----------------------------------------------------------
>
> (Updated Dec. 4, 2015, 3:48 p.m.)
>
>
> Review request for Ambari, Jayush Luniya, Nate Cole, and Sumit Mohanty.
>
>
> Bugs: AMBARI-14220
> https://issues.apache.org/jira/browse/AMBARI-14220
>
>
> Repository: ambari
>
>
> Description
> -------
>
> AMBARI-13824 introduced a regression caused by additions of code to conf_select which destroyed configuration directories. Long story short is that conf_select should be told to create symlinks and move configurations around; it should not do it on every select() call.
>
> I removed the offending code and have install_packages determine if symlinking should be done in conf_select.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 24ca151
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java d299602
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 69709e5
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 0170806
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py 3961897
>
> Diff: https://reviews.apache.org/r/40986/diff/
>
>
> Testing
> -------
>
> Upgrades from 2.1 to 2.3, 2.2 to 2.3, and 2.3 to 2.3; no side effects yet.
>
> Working on tests...
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 40986: Express Upgrades Fail To Stop Services
Because Of Destroyed Configurations
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40986/#review109034
-----------------------------------------------------------
Ship it!
RU has a task that will rollback directories:
<execute-stage title="Restore configuration directories and remove HDP 2.3 symlinks">
<task xsi:type="execute">
<script>scripts/ru_set_all.py</script>
<function>unlink_all_configs</function>
</task>
</execute-stage>
Do we need anything special in this case when downgrading to HDP-2.1 and 2.2? (I doubt it, but just bringing it up)
- Nate Cole
On Dec. 4, 2015, 3:48 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40986/
> -----------------------------------------------------------
>
> (Updated Dec. 4, 2015, 3:48 p.m.)
>
>
> Review request for Ambari, Jayush Luniya, Nate Cole, and Sumit Mohanty.
>
>
> Bugs: AMBARI-14220
> https://issues.apache.org/jira/browse/AMBARI-14220
>
>
> Repository: ambari
>
>
> Description
> -------
>
> AMBARI-13824 introduced a regression caused by additions of code to conf_select which destroyed configuration directories. Long story short is that conf_select should be told to create symlinks and move configurations around; it should not do it on every select() call.
>
> I removed the offending code and have install_packages determine if symlinking should be done in conf_select.
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py 24ca151
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java d299602
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 69709e5
> ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 0170806
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py 3961897
>
> Diff: https://reviews.apache.org/r/40986/diff/
>
>
> Testing
> -------
>
> Upgrades from 2.1 to 2.3, 2.2 to 2.3, and 2.3 to 2.3; no side effects yet.
>
> Working on tests...
>
>
> Thanks,
>
> Jonathan Hurley
>
>