You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Nate Cole <nc...@hortonworks.com> on 2015/05/16 04:50:19 UTC
Review Request 34311: Hive Upgrade Pack For HDP-2.2 to HDP-2.3
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34311/
-----------------------------------------------------------
Review request for Ambari, Alejandro Fernandez and Jonathan Hurley.
Bugs: AMBARI-11176
https://issues.apache.org/jira/browse/AMBARI-11176
Repository: ambari
Description
-------
* Changes to Hive scripts to correctly decide paths during an RU to start with the right jars.
* Changes to service check to load with new parameter to work in 2.3
* Added ability to replace properties in a ConfigureTask (to be expanded in the future - just straight string replacement for now.
* Changed base test python class to accept arguments - this allows us to test start(rolling_upgrade=True) instead of the torture of testing restart()
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java ca72934
ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/ConfigureAction.java 729c36a
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ConfigureTask.java 06d3108
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/webhcat-env.xml 14a473f
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/files/startMetastore.sh da0f60b
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/files/templetonSmoke.sh 2083312
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_service_check.py e8bad81
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py c5d30a1
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py a41632e
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py af9d5d3
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/webhcat_server.py 735590b
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/webhcat_service.py 7fb82da
ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml 43db5b0
ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/ConfigureActionTest.java 50e0a17
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_metastore.py 6bc9797
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 04db874
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_webhcat_server.py f2d7edb
ambari-server/src/test/python/stacks/2.1/HIVE/test_hive_metastore.py 2ed5f14
ambari-server/src/test/python/stacks/utils/RMFTestCase.py 0bbde77
Diff: https://reviews.apache.org/r/34311/diff/
Testing
-------
Manual cluster testing; about 1000 upgrades to get working correctly. (Jonathan knows)
Java automated: pending.
Python:
----------------------------------------------------------------------
Ran 233 tests in 8.563s
OK
----------------------------------------------------------------------
Total run:741
Total errors:0
Total failures:0
OK
Thanks,
Nate Cole
Re: Review Request 34311: Hive Upgrade Pack For HDP-2.2 to HDP-2.3
Posted by Nate Cole <nc...@hortonworks.com>.
> On May 16, 2015, 8:36 p.m., Jonathan Hurley wrote:
> > Are there any alert scripts for Hive that still hard code /etc/hive/conf ? If so, we should track those changes as well since now we defer to /usr/hdp/current/hive-server/conf
I'll open a tracker JIRA for it. We need a checklist of sh!t to do when we code: Does it conform to HA? Does it affect alerts? Does it need Metrics? Does it need to be non-root (ambari-sudo)? Does it need specific version for Upgrades? Does it upgrade: period? Does it upgrade: rolling? etc etc
> On May 16, 2015, 8:36 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java, line 1026
> > <https://reviews.apache.org/r/34311/diff/1/?file=961968#file961968line1026>
> >
> > At what point do we just start asking ConfigureTask if there's any work to do :)
Thanks for reviewing. Ha! I know right; I felt bad putting even more checks in there, but my back was against the wall.
- Nate
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34311/#review84047
-----------------------------------------------------------
On May 16, 2015, 12:18 p.m., Nate Cole wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34311/
> -----------------------------------------------------------
>
> (Updated May 16, 2015, 12:18 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Jonathan Hurley.
>
>
> Bugs: AMBARI-11176
> https://issues.apache.org/jira/browse/AMBARI-11176
>
>
> Repository: ambari
>
>
> Description
> -------
>
> * Changes to Hive scripts to correctly decide paths during an RU to start with the right jars.
> * Changes to service check to load with new parameter to work in 2.3
> * Added ability to replace properties in a ConfigureTask (to be expanded in the future - just straight string replacement for now.
> * Changed base test python class to accept arguments - this allows us to test start(rolling_upgrade=True) instead of the torture of testing restart()
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java ca72934
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/ConfigureAction.java 729c36a
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ConfigureTask.java 06d3108
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/webhcat-env.xml 14a473f
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/files/startMetastore.sh da0f60b
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/files/templetonSmoke.sh 2083312
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_service_check.py e8bad81
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py c5d30a1
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py a41632e
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py af9d5d3
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/webhcat_server.py 735590b
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/webhcat_service.py 7fb82da
> ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml 43db5b0
> ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/ConfigureActionTest.java 50e0a17
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_metastore.py 6bc9797
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 04db874
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_webhcat_server.py f2d7edb
> ambari-server/src/test/python/stacks/2.1/HIVE/test_hive_metastore.py 2ed5f14
> ambari-server/src/test/python/stacks/utils/RMFTestCase.py 0bbde77
>
> Diff: https://reviews.apache.org/r/34311/diff/
>
>
> Testing
> -------
>
> Manual cluster testing; about 1000 upgrades to get working correctly. (Jonathan knows)
>
> Java automated: pending.
>
> ----------------------------------------------------------------------
> Ran 233 tests in 6.741s
>
> OK
> ----------------------------------------------------------------------
> Total run:741
> Total errors:0
> Total failures:0
> OK
> log
> log
> StackAdvisor implementation for stack HDP1, version 2.0.6 was not found
> Returning DefaultStackAdvisor implementation
> StackAdvisor implementation for stack XYZ, version 1.0.0 was loaded
> StackAdvisor implementation for stack XYZ, version 1.0.1 was loaded
> Returning XYZ101StackAdvisor implementation
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 50:59.198s
> [INFO] Finished at: Sat May 16 10:17:52 EDT 2015
> [INFO] Final Memory: 31M/405M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Nate Cole
>
>
Re: Review Request 34311: Hive Upgrade Pack For HDP-2.2 to HDP-2.3
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34311/#review84047
-----------------------------------------------------------
Ship it!
Are there any alert scripts for Hive that still hard code /etc/hive/conf ? If so, we should track those changes as well since now we defer to /usr/hdp/current/hive-server/conf
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
<https://reviews.apache.org/r/34311/#comment135175>
At what point do we just start asking ConfigureTask if there's any work to do :)
- Jonathan Hurley
On May 16, 2015, 12:18 p.m., Nate Cole wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34311/
> -----------------------------------------------------------
>
> (Updated May 16, 2015, 12:18 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Jonathan Hurley.
>
>
> Bugs: AMBARI-11176
> https://issues.apache.org/jira/browse/AMBARI-11176
>
>
> Repository: ambari
>
>
> Description
> -------
>
> * Changes to Hive scripts to correctly decide paths during an RU to start with the right jars.
> * Changes to service check to load with new parameter to work in 2.3
> * Added ability to replace properties in a ConfigureTask (to be expanded in the future - just straight string replacement for now.
> * Changed base test python class to accept arguments - this allows us to test start(rolling_upgrade=True) instead of the torture of testing restart()
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java ca72934
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/ConfigureAction.java 729c36a
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ConfigureTask.java 06d3108
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/webhcat-env.xml 14a473f
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/files/startMetastore.sh da0f60b
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/files/templetonSmoke.sh 2083312
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_service_check.py e8bad81
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py c5d30a1
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py a41632e
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py af9d5d3
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/webhcat_server.py 735590b
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/webhcat_service.py 7fb82da
> ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml 43db5b0
> ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/ConfigureActionTest.java 50e0a17
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_metastore.py 6bc9797
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 04db874
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_webhcat_server.py f2d7edb
> ambari-server/src/test/python/stacks/2.1/HIVE/test_hive_metastore.py 2ed5f14
> ambari-server/src/test/python/stacks/utils/RMFTestCase.py 0bbde77
>
> Diff: https://reviews.apache.org/r/34311/diff/
>
>
> Testing
> -------
>
> Manual cluster testing; about 1000 upgrades to get working correctly. (Jonathan knows)
>
> Java automated: pending.
>
> ----------------------------------------------------------------------
> Ran 233 tests in 6.741s
>
> OK
> ----------------------------------------------------------------------
> Total run:741
> Total errors:0
> Total failures:0
> OK
> log
> log
> StackAdvisor implementation for stack HDP1, version 2.0.6 was not found
> Returning DefaultStackAdvisor implementation
> StackAdvisor implementation for stack XYZ, version 1.0.0 was loaded
> StackAdvisor implementation for stack XYZ, version 1.0.1 was loaded
> Returning XYZ101StackAdvisor implementation
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 50:59.198s
> [INFO] Finished at: Sat May 16 10:17:52 EDT 2015
> [INFO] Final Memory: 31M/405M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Nate Cole
>
>
Re: Review Request 34311: Hive Upgrade Pack For HDP-2.2 to HDP-2.3
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34311/
-----------------------------------------------------------
(Updated May 16, 2015, 12:18 p.m.)
Review request for Ambari, Alejandro Fernandez and Jonathan Hurley.
Changes
-------
Updated test results
Bugs: AMBARI-11176
https://issues.apache.org/jira/browse/AMBARI-11176
Repository: ambari
Description
-------
* Changes to Hive scripts to correctly decide paths during an RU to start with the right jars.
* Changes to service check to load with new parameter to work in 2.3
* Added ability to replace properties in a ConfigureTask (to be expanded in the future - just straight string replacement for now.
* Changed base test python class to accept arguments - this allows us to test start(rolling_upgrade=True) instead of the torture of testing restart()
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java ca72934
ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/ConfigureAction.java 729c36a
ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ConfigureTask.java 06d3108
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/webhcat-env.xml 14a473f
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/files/startMetastore.sh da0f60b
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/files/templetonSmoke.sh 2083312
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_service_check.py e8bad81
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py c5d30a1
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py a41632e
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py af9d5d3
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/webhcat_server.py 735590b
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/webhcat_service.py 7fb82da
ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml 43db5b0
ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/ConfigureActionTest.java 50e0a17
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_metastore.py 6bc9797
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 04db874
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_webhcat_server.py f2d7edb
ambari-server/src/test/python/stacks/2.1/HIVE/test_hive_metastore.py 2ed5f14
ambari-server/src/test/python/stacks/utils/RMFTestCase.py 0bbde77
Diff: https://reviews.apache.org/r/34311/diff/
Testing (updated)
-------
Manual cluster testing; about 1000 upgrades to get working correctly. (Jonathan knows)
Java automated: pending.
----------------------------------------------------------------------
Ran 233 tests in 6.741s
OK
----------------------------------------------------------------------
Total run:741
Total errors:0
Total failures:0
OK
log
log
StackAdvisor implementation for stack HDP1, version 2.0.6 was not found
Returning DefaultStackAdvisor implementation
StackAdvisor implementation for stack XYZ, version 1.0.0 was loaded
StackAdvisor implementation for stack XYZ, version 1.0.1 was loaded
Returning XYZ101StackAdvisor implementation
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 50:59.198s
[INFO] Finished at: Sat May 16 10:17:52 EDT 2015
[INFO] Final Memory: 31M/405M
[INFO] ------------------------------------------------------------------------
Thanks,
Nate Cole