You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Sid Wagle <sw...@hortonworks.com> on 2014/10/31 21:02:11 UTC
Review Request 27443: Provide validated Oozie configs for Ambari
deployments
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27443/
-----------------------------------------------------------
Review request for Ambari and Alejandro Fernandez.
Bugs: AMBARI-8074
https://issues.apache.org/jira/browse/AMBARI-8074
Repository: ambari
Description
-------
Here are the list of changes that we need to have. Some are in place already - marked by *
* Have adminusers.txt with oozie install user *
* oozie.service.AuthorizationService.security.enabled = true *
For client retries during oozie server failure during restarts (during rolling upgrade for example), oozie clients use a retry parameter. The default is 4. This is controlled by the system property oozie.connection.retry.count. The sleep time between retries is not configurable and determined by 2^(retriynumber) seconds
So, so client will retry after 2, 4, 8 and 16 seconds delay. If we need more delays than the 30 seconds (cumulative) that we get now, we should set the system property (-Doozie.connection.retry.count=<value>) in <oozie-insall-dir>/bin/oozie.distro file by setting OOZIE_CLIENT_OPTS env variable
We can also pass "-Doozie.connection.retry.count=<value>" on the command line when submitting jobs.
- Sharelib handling
For rolling upgrade, we need to set these to be manually handled by the user - Even though oozie guarantees to keep one older version, we would need to disable the automatic deletion of older sharelibs. The default is 1
oozie.service.ShareLibService.purge.interval=<numdays>
Oozie sharelib is refreshed by explicit command or on oozie server restart. We can't disable the oozie-server restart refresh. This can cause issues during rolling upgrade. Today, ambari installs the oozie sharelib as part of oozie install. When multiple oozie servers are involved, all oozie servers should be stopped before oozie upgrade is begun. And all oozie server binaries and config should be updated and the sharelib updated just before the oozie servers are started with the new version bits
- Handling Hive/Tez.
As it stands today, to ease the effort of users shipping hive and tez files, we should let the installers create per action default config file prepopulated with the contents of hive and tez (for hive actions) and pig and tez (for pig actions) so that workflows need not change between rolling upgrades. See BUG-26227 for more details on our attempts to fix it and why this is necessary
Diffs
-----
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/OOZIE/package/scripts/oozie.py 19c334c
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/OOZIE/package/scripts/params.py 6768014
ambari-server/src/main/resources/stacks/HDP/2.2/services/OOZIE/configuration/oozie-env.xml PRE-CREATION
ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_client.py 67482b4
ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py d9f3ef6
Diff: https://reviews.apache.org/r/27443/diff/
Testing
-------
----------------------------------------------------------------------
Ran 242 tests in 3.706s
OK
----------------------------------------------------------------------
Total run:681
Total errors:0
Total failures:0
OK
Thanks,
Sid Wagle
Re: Review Request 27443: Provide validated Oozie configs for Ambari
deployments
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27443/#review59395
-----------------------------------------------------------
Ship it!
Ship It!
- Alejandro Fernandez
On Oct. 31, 2014, 8:02 p.m., Sid Wagle wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27443/
> -----------------------------------------------------------
>
> (Updated Oct. 31, 2014, 8:02 p.m.)
>
>
> Review request for Ambari and Alejandro Fernandez.
>
>
> Bugs: AMBARI-8074
> https://issues.apache.org/jira/browse/AMBARI-8074
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Here are the list of changes that we need to have. Some are in place already - marked by *
>
> * Have adminusers.txt with oozie install user *
> * oozie.service.AuthorizationService.security.enabled = true *
>
> For client retries during oozie server failure during restarts (during rolling upgrade for example), oozie clients use a retry parameter. The default is 4. This is controlled by the system property oozie.connection.retry.count. The sleep time between retries is not configurable and determined by 2^(retriynumber) seconds
> So, so client will retry after 2, 4, 8 and 16 seconds delay. If we need more delays than the 30 seconds (cumulative) that we get now, we should set the system property (-Doozie.connection.retry.count=<value>) in <oozie-insall-dir>/bin/oozie.distro file by setting OOZIE_CLIENT_OPTS env variable
>
> We can also pass "-Doozie.connection.retry.count=<value>" on the command line when submitting jobs.
>
> - Sharelib handling
>
> For rolling upgrade, we need to set these to be manually handled by the user - Even though oozie guarantees to keep one older version, we would need to disable the automatic deletion of older sharelibs. The default is 1
>
> oozie.service.ShareLibService.purge.interval=<numdays>
>
> Oozie sharelib is refreshed by explicit command or on oozie server restart. We can't disable the oozie-server restart refresh. This can cause issues during rolling upgrade. Today, ambari installs the oozie sharelib as part of oozie install. When multiple oozie servers are involved, all oozie servers should be stopped before oozie upgrade is begun. And all oozie server binaries and config should be updated and the sharelib updated just before the oozie servers are started with the new version bits
>
> - Handling Hive/Tez.
> As it stands today, to ease the effort of users shipping hive and tez files, we should let the installers create per action default config file prepopulated with the contents of hive and tez (for hive actions) and pig and tez (for pig actions) so that workflows need not change between rolling upgrades. See BUG-26227 for more details on our attempts to fix it and why this is necessary
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/OOZIE/package/scripts/oozie.py 19c334c
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/OOZIE/package/scripts/params.py 6768014
> ambari-server/src/main/resources/stacks/HDP/2.2/services/OOZIE/configuration/oozie-env.xml PRE-CREATION
> ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_client.py 67482b4
> ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py d9f3ef6
>
> Diff: https://reviews.apache.org/r/27443/diff/
>
>
> Testing
> -------
>
> ----------------------------------------------------------------------
> Ran 242 tests in 3.706s
>
> OK
> ----------------------------------------------------------------------
> Total run:681
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Sid Wagle
>
>