You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Alejandro Fernandez <af...@hortonworks.com> on 2015/05/20 02:31:45 UTC
Review Request 34451: YARN Service Check job and Oozie job don't run
because
of whitespace in yarn.scheduler.capacity.root.default-node-label-expression
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34451/
-----------------------------------------------------------
Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, Sumit Mohanty, and Sid Wagle.
Bugs: AMBARI-11257
https://issues.apache.org/jira/browse/AMBARI-11257
Repository: ambari
Description
-------
STR:
Create a cluster with Ambari 2.1.0, at build 1638
Install HDP 2.3.0.0 at build 2030, with HDFS, MR, YARN, ZK, OOZIE
Run Yarn Service Check
Run Oozie Service Check
The Resource Manager UI shows the apps as waiting for AM to schedule them, and eventually they get killed.
For Oozie, it is able to launch the MR jobs, but of course, they timeout.
The problem is that the yarn.scheduler.capacity.root.default-node-label-expression property is set to " " instead of "", and this behavior changed in HDP 2.3
Diffs
-----
ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/configuration/capacity-scheduler.xml f1d4b48
ambari-server/src/main/resources/stacks/HDP/2.3.GlusterFS/services/YARN/configuration/capacity-scheduler.xml 63d56b0
ambari-server/src/main/resources/stacks/HDPWIN/2.2/services/YARN/configuration/capacity-scheduler.xml 63d56b0
ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_1.3_to_2.2.json f1d08b9
ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_2.0_to_2.2.2.json 86521d1
ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_2.0_to_2.2.4.json 7721c67
ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_2.1_to_2.2.2.json a040268
ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_2.1_to_2.2.4.json ba2b3d7
ambari-server/src/test/python/stacks/2.0.6/configs/client-upgrade.json aeecb79
ambari-server/src/test/python/stacks/2.0.6/configs/hbase-2.2.json fa48c6d
ambari-server/src/test/python/stacks/2.0.6/configs/hbase-check-2.2.json 88017b5
ambari-server/src/test/python/stacks/2.0.6/configs/hbase-rs-2.2.json 3109b7a
ambari-server/src/test/python/stacks/2.1/configs/client-upgrade.json 1534619
ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade-hdfs-secure.json 3b6a2fa
ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade.json 9d8cb39
ambari-server/src/test/python/stacks/2.2/configs/pig-service-check-secure.json a7a6121
ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json 3e6307d
ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json caec8ca
Diff: https://reviews.apache.org/r/34451/diff/
Testing
-------
Installed ambari-server
Patched the file capacity-scheduler.xml
Deployed a cluster with HDP 2.3 and HDFS, YARN, MR, ZK
Verified the property change in the configs, and that the YARN Service Check passed.
----------------------------------------------------------------------
Total run:741
Total errors:0
Total failures:0
OK
Thanks,
Alejandro Fernandez
Re: Review Request 34451: YARN Service Check job and Oozie job don't
run because of whitespace in
yarn.scheduler.capacity.root.default-node-label-expression
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34451/#review84443
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/configuration/capacity-scheduler.xml
<https://reviews.apache.org/r/34451/#comment135707>
Actually, this will need a config item in the Upgrade Pack for 2.2 -> 2.3
- Alejandro Fernandez
On May 20, 2015, 12:31 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34451/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 12:31 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-11257
> https://issues.apache.org/jira/browse/AMBARI-11257
>
>
> Repository: ambari
>
>
> Description
> -------
>
> STR:
> Create a cluster with Ambari 2.1.0, at build 1638
> Install HDP 2.3.0.0 at build 2030, with HDFS, MR, YARN, ZK, OOZIE
> Run Yarn Service Check
> Run Oozie Service Check
>
> The Resource Manager UI shows the apps as waiting for AM to schedule them, and eventually they get killed.
>
> For Oozie, it is able to launch the MR jobs, but of course, they timeout.
>
> The problem is that the yarn.scheduler.capacity.root.default-node-label-expression property is set to " " instead of "", and this behavior changed in HDP 2.3, but works in HDP 2.2 as well.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/configuration/capacity-scheduler.xml f1d4b48
> ambari-server/src/main/resources/stacks/HDP/2.3.GlusterFS/services/YARN/configuration/capacity-scheduler.xml 63d56b0
> ambari-server/src/main/resources/stacks/HDPWIN/2.2/services/YARN/configuration/capacity-scheduler.xml 63d56b0
> ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_1.3_to_2.2.json f1d08b9
> ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_2.0_to_2.2.2.json 86521d1
> ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_2.0_to_2.2.4.json 7721c67
> ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_2.1_to_2.2.2.json a040268
> ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_2.1_to_2.2.4.json ba2b3d7
> ambari-server/src/test/python/stacks/2.0.6/configs/client-upgrade.json aeecb79
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase-2.2.json fa48c6d
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase-check-2.2.json 88017b5
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase-rs-2.2.json 3109b7a
> ambari-server/src/test/python/stacks/2.1/configs/client-upgrade.json 1534619
> ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade-hdfs-secure.json 3b6a2fa
> ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade.json 9d8cb39
> ambari-server/src/test/python/stacks/2.2/configs/pig-service-check-secure.json a7a6121
> ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json 3e6307d
> ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json caec8ca
>
> Diff: https://reviews.apache.org/r/34451/diff/
>
>
> Testing
> -------
>
> Installed ambari-server
> Patched the file capacity-scheduler.xml
> Deployed a cluster with HDP 2.3 and HDFS, YARN, MR, ZK
> Verified the property change in the configs, and that the YARN Service Check passed.
>
> ----------------------------------------------------------------------
> Total run:741
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 34451: YARN Service Check job and Oozie job don't
run because of whitespace in
yarn.scheduler.capacity.root.default-node-label-expression
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34451/#review84432
-----------------------------------------------------------
Ship it!
Ship It!
- Sumit Mohanty
On May 20, 2015, 12:31 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34451/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 12:31 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-11257
> https://issues.apache.org/jira/browse/AMBARI-11257
>
>
> Repository: ambari
>
>
> Description
> -------
>
> STR:
> Create a cluster with Ambari 2.1.0, at build 1638
> Install HDP 2.3.0.0 at build 2030, with HDFS, MR, YARN, ZK, OOZIE
> Run Yarn Service Check
> Run Oozie Service Check
>
> The Resource Manager UI shows the apps as waiting for AM to schedule them, and eventually they get killed.
>
> For Oozie, it is able to launch the MR jobs, but of course, they timeout.
>
> The problem is that the yarn.scheduler.capacity.root.default-node-label-expression property is set to " " instead of "", and this behavior changed in HDP 2.3, but works in HDP 2.2 as well.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/configuration/capacity-scheduler.xml f1d4b48
> ambari-server/src/main/resources/stacks/HDP/2.3.GlusterFS/services/YARN/configuration/capacity-scheduler.xml 63d56b0
> ambari-server/src/main/resources/stacks/HDPWIN/2.2/services/YARN/configuration/capacity-scheduler.xml 63d56b0
> ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_1.3_to_2.2.json f1d08b9
> ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_2.0_to_2.2.2.json 86521d1
> ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_2.0_to_2.2.4.json 7721c67
> ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_2.1_to_2.2.2.json a040268
> ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_2.1_to_2.2.4.json ba2b3d7
> ambari-server/src/test/python/stacks/2.0.6/configs/client-upgrade.json aeecb79
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase-2.2.json fa48c6d
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase-check-2.2.json 88017b5
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase-rs-2.2.json 3109b7a
> ambari-server/src/test/python/stacks/2.1/configs/client-upgrade.json 1534619
> ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade-hdfs-secure.json 3b6a2fa
> ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade.json 9d8cb39
> ambari-server/src/test/python/stacks/2.2/configs/pig-service-check-secure.json a7a6121
> ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json 3e6307d
> ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json caec8ca
>
> Diff: https://reviews.apache.org/r/34451/diff/
>
>
> Testing
> -------
>
> Installed ambari-server
> Patched the file capacity-scheduler.xml
> Deployed a cluster with HDP 2.3 and HDFS, YARN, MR, ZK
> Verified the property change in the configs, and that the YARN Service Check passed.
>
> ----------------------------------------------------------------------
> Total run:741
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 34451: YARN Service Check job and Oozie job don't
run because of whitespace in
yarn.scheduler.capacity.root.default-node-label-expression
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34451/#review84451
-----------------------------------------------------------
Ship it!
Ship It!
- Sumit Mohanty
On May 20, 2015, 3:10 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34451/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 3:10 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-11257
> https://issues.apache.org/jira/browse/AMBARI-11257
>
>
> Repository: ambari
>
>
> Description
> -------
>
> STR:
> Create a cluster with Ambari 2.1.0, at build 1638
> Install HDP 2.3.0.0 at build 2030, with HDFS, MR, YARN, ZK, OOZIE
> Run Yarn Service Check
> Run Oozie Service Check
>
> The Resource Manager UI shows the apps as waiting for AM to schedule them, and eventually they get killed.
>
> For Oozie, it is able to launch the MR jobs, but of course, they timeout.
>
> The problem is that the yarn.scheduler.capacity.root.default-node-label-expression property is set to " " instead of "", and this behavior changed in HDP 2.3, but works in HDP 2.2 as well.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml e954af2
> ambari-server/src/main/resources/stacks/HDP/2.3.GlusterFS/services/YARN/configuration/capacity-scheduler.xml 63d56b0
> ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/configuration/capacity-scheduler.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/34451/diff/
>
>
> Testing
> -------
>
> Installed ambari-server
> Patched the file capacity-scheduler.xml
> Deployed a cluster with HDP 2.3 and HDFS, YARN, MR, ZK
> Verified the property change in the configs, and that the YARN Service Check passed.
>
> ----------------------------------------------------------------------
> Total run:741
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 34451: YARN Service Check job and Oozie job don't
run because of whitespace in
yarn.scheduler.capacity.root.default-node-label-expression
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On May 20, 2015, 8:13 a.m., Andrew Onischuk wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.3.GlusterFS/services/YARN/configuration/capacity-scheduler.xml, line 41
> > <https://reviews.apache.org/r/34451/diff/4/?file=964876#file964876line41>
> >
> > Seems like this is caused by not striping the properties patch.
> > In that case this is not Yarn 2.3 specific and will reproduce with Yarn 2.2
> >
> > Alejandro can you please check that. And if so change it in HDP-2.2 rather than overriding it in HDP-2.3
>
> Jonathan Hurley wrote:
> I agree; we should correct it in HDP 2.2, but still need trimming logic for existing clusters, no?
I'm going to drop this issue since this is new to HDP 2.3.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34451/#review84481
-----------------------------------------------------------
On May 20, 2015, 3:10 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34451/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 3:10 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-11257
> https://issues.apache.org/jira/browse/AMBARI-11257
>
>
> Repository: ambari
>
>
> Description
> -------
>
> STR:
> Create a cluster with Ambari 2.1.0, at build 1638
> Install HDP 2.3.0.0 at build 2030, with HDFS, MR, YARN, ZK, OOZIE
> Run Yarn Service Check
> Run Oozie Service Check
>
> The Resource Manager UI shows the apps as waiting for AM to schedule them, and eventually they get killed.
>
> For Oozie, it is able to launch the MR jobs, but of course, they timeout.
>
> The problem is that the yarn.scheduler.capacity.root.default-node-label-expression property is set to " " instead of "", and this behavior changed in HDP 2.3, but works in HDP 2.2 as well.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml e954af2
> ambari-server/src/main/resources/stacks/HDP/2.3.GlusterFS/services/YARN/configuration/capacity-scheduler.xml 63d56b0
> ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/configuration/capacity-scheduler.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/34451/diff/
>
>
> Testing
> -------
>
> Installed ambari-server
> Patched the file capacity-scheduler.xml
> Deployed a cluster with HDP 2.3 and HDFS, YARN, MR, ZK
> Verified the property change in the configs, and that the YARN Service Check passed.
>
> ----------------------------------------------------------------------
> Total run:741
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 34451: YARN Service Check job and Oozie job don't
run because of whitespace in
yarn.scheduler.capacity.root.default-node-label-expression
Posted by Andrew Onischuk <ao...@hortonworks.com>.
> On May 20, 2015, 8:13 a.m., Andrew Onischuk wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.3.GlusterFS/services/YARN/configuration/capacity-scheduler.xml, line 41
> > <https://reviews.apache.org/r/34451/diff/4/?file=964876#file964876line41>
> >
> > Seems like this is caused by not striping the properties patch.
> > In that case this is not Yarn 2.3 specific and will reproduce with Yarn 2.2
> >
> > Alejandro can you please check that. And if so change it in HDP-2.2 rather than overriding it in HDP-2.3
>
> Jonathan Hurley wrote:
> I agree; we should correct it in HDP 2.2, but still need trimming logic for existing clusters, no?
>
> Alejandro Fernandez wrote:
> I'm going to drop this issue since this is new to HDP 2.3.
Alejandro, before in Ambari spaces from properties was stripped. And there was a patch not long ago which disabled that logic. That's why it's now reproducible. Due to that I think that might be an issue for HDP-2.2.
- Andrew
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34451/#review84481
-----------------------------------------------------------
On May 20, 2015, 3:10 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34451/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 3:10 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-11257
> https://issues.apache.org/jira/browse/AMBARI-11257
>
>
> Repository: ambari
>
>
> Description
> -------
>
> STR:
> Create a cluster with Ambari 2.1.0, at build 1638
> Install HDP 2.3.0.0 at build 2030, with HDFS, MR, YARN, ZK, OOZIE
> Run Yarn Service Check
> Run Oozie Service Check
>
> The Resource Manager UI shows the apps as waiting for AM to schedule them, and eventually they get killed.
>
> For Oozie, it is able to launch the MR jobs, but of course, they timeout.
>
> The problem is that the yarn.scheduler.capacity.root.default-node-label-expression property is set to " " instead of "", and this behavior changed in HDP 2.3, but works in HDP 2.2 as well.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml e954af2
> ambari-server/src/main/resources/stacks/HDP/2.3.GlusterFS/services/YARN/configuration/capacity-scheduler.xml 63d56b0
> ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/configuration/capacity-scheduler.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/34451/diff/
>
>
> Testing
> -------
>
> Installed ambari-server
> Patched the file capacity-scheduler.xml
> Deployed a cluster with HDP 2.3 and HDFS, YARN, MR, ZK
> Verified the property change in the configs, and that the YARN Service Check passed.
>
> ----------------------------------------------------------------------
> Total run:741
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 34451: YARN Service Check job and Oozie job don't
run because of whitespace in
yarn.scheduler.capacity.root.default-node-label-expression
Posted by Jonathan Hurley <jh...@hortonworks.com>.
> On May 20, 2015, 4:13 a.m., Andrew Onischuk wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.3.GlusterFS/services/YARN/configuration/capacity-scheduler.xml, line 41
> > <https://reviews.apache.org/r/34451/diff/4/?file=964876#file964876line41>
> >
> > Seems like this is caused by not striping the properties patch.
> > In that case this is not Yarn 2.3 specific and will reproduce with Yarn 2.2
> >
> > Alejandro can you please check that. And if so change it in HDP-2.2 rather than overriding it in HDP-2.3
I agree; we should correct it in HDP 2.2, but still need trimming logic for existing clusters, no?
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34451/#review84481
-----------------------------------------------------------
On May 19, 2015, 11:10 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34451/
> -----------------------------------------------------------
>
> (Updated May 19, 2015, 11:10 p.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-11257
> https://issues.apache.org/jira/browse/AMBARI-11257
>
>
> Repository: ambari
>
>
> Description
> -------
>
> STR:
> Create a cluster with Ambari 2.1.0, at build 1638
> Install HDP 2.3.0.0 at build 2030, with HDFS, MR, YARN, ZK, OOZIE
> Run Yarn Service Check
> Run Oozie Service Check
>
> The Resource Manager UI shows the apps as waiting for AM to schedule them, and eventually they get killed.
>
> For Oozie, it is able to launch the MR jobs, but of course, they timeout.
>
> The problem is that the yarn.scheduler.capacity.root.default-node-label-expression property is set to " " instead of "", and this behavior changed in HDP 2.3, but works in HDP 2.2 as well.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml e954af2
> ambari-server/src/main/resources/stacks/HDP/2.3.GlusterFS/services/YARN/configuration/capacity-scheduler.xml 63d56b0
> ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/configuration/capacity-scheduler.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/34451/diff/
>
>
> Testing
> -------
>
> Installed ambari-server
> Patched the file capacity-scheduler.xml
> Deployed a cluster with HDP 2.3 and HDFS, YARN, MR, ZK
> Verified the property change in the configs, and that the YARN Service Check passed.
>
> ----------------------------------------------------------------------
> Total run:741
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 34451: YARN Service Check job and Oozie job don't
run because of whitespace in
yarn.scheduler.capacity.root.default-node-label-expression
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On May 20, 2015, 8:13 a.m., Andrew Onischuk wrote:
> >
As far as I can tell, " " did work in Yarn HDP 2.2. Because that stack is already deployed, it must be addressed in upgrades from 2.2 -> 2.3, which is done via the Upgrade Pack (we should probably doc it as well in case users perform a manual upgrade).
So I don't believe we need to change the property in HDP 2.2.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34451/#review84481
-----------------------------------------------------------
On May 20, 2015, 3:10 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34451/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 3:10 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-11257
> https://issues.apache.org/jira/browse/AMBARI-11257
>
>
> Repository: ambari
>
>
> Description
> -------
>
> STR:
> Create a cluster with Ambari 2.1.0, at build 1638
> Install HDP 2.3.0.0 at build 2030, with HDFS, MR, YARN, ZK, OOZIE
> Run Yarn Service Check
> Run Oozie Service Check
>
> The Resource Manager UI shows the apps as waiting for AM to schedule them, and eventually they get killed.
>
> For Oozie, it is able to launch the MR jobs, but of course, they timeout.
>
> The problem is that the yarn.scheduler.capacity.root.default-node-label-expression property is set to " " instead of "", and this behavior changed in HDP 2.3, but works in HDP 2.2 as well.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml e954af2
> ambari-server/src/main/resources/stacks/HDP/2.3.GlusterFS/services/YARN/configuration/capacity-scheduler.xml 63d56b0
> ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/configuration/capacity-scheduler.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/34451/diff/
>
>
> Testing
> -------
>
> Installed ambari-server
> Patched the file capacity-scheduler.xml
> Deployed a cluster with HDP 2.3 and HDFS, YARN, MR, ZK
> Verified the property change in the configs, and that the YARN Service Check passed.
>
> ----------------------------------------------------------------------
> Total run:741
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 34451: YARN Service Check job and Oozie job don't
run because of whitespace in
yarn.scheduler.capacity.root.default-node-label-expression
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34451/#review84481
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/HDP/2.3.GlusterFS/services/YARN/configuration/capacity-scheduler.xml
<https://reviews.apache.org/r/34451/#comment135789>
Seems like this is caused by not striping the properties patch.
In that case this is not Yarn 2.3 specific and will reproduce with Yarn 2.2
Alejandro can you please check that. And if so change it in HDP-2.2 rather than overriding it in HDP-2.3
- Andrew Onischuk
On May 20, 2015, 3:10 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34451/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 3:10 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-11257
> https://issues.apache.org/jira/browse/AMBARI-11257
>
>
> Repository: ambari
>
>
> Description
> -------
>
> STR:
> Create a cluster with Ambari 2.1.0, at build 1638
> Install HDP 2.3.0.0 at build 2030, with HDFS, MR, YARN, ZK, OOZIE
> Run Yarn Service Check
> Run Oozie Service Check
>
> The Resource Manager UI shows the apps as waiting for AM to schedule them, and eventually they get killed.
>
> For Oozie, it is able to launch the MR jobs, but of course, they timeout.
>
> The problem is that the yarn.scheduler.capacity.root.default-node-label-expression property is set to " " instead of "", and this behavior changed in HDP 2.3, but works in HDP 2.2 as well.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml e954af2
> ambari-server/src/main/resources/stacks/HDP/2.3.GlusterFS/services/YARN/configuration/capacity-scheduler.xml 63d56b0
> ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/configuration/capacity-scheduler.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/34451/diff/
>
>
> Testing
> -------
>
> Installed ambari-server
> Patched the file capacity-scheduler.xml
> Deployed a cluster with HDP 2.3 and HDFS, YARN, MR, ZK
> Verified the property change in the configs, and that the YARN Service Check passed.
>
> ----------------------------------------------------------------------
> Total run:741
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 34451: YARN Service Check job and Oozie job don't
run because of whitespace in
yarn.scheduler.capacity.root.default-node-label-expression
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34451/
-----------------------------------------------------------
(Updated May 20, 2015, 3:10 a.m.)
Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
Bugs: AMBARI-11257
https://issues.apache.org/jira/browse/AMBARI-11257
Repository: ambari
Description
-------
STR:
Create a cluster with Ambari 2.1.0, at build 1638
Install HDP 2.3.0.0 at build 2030, with HDFS, MR, YARN, ZK, OOZIE
Run Yarn Service Check
Run Oozie Service Check
The Resource Manager UI shows the apps as waiting for AM to schedule them, and eventually they get killed.
For Oozie, it is able to launch the MR jobs, but of course, they timeout.
The problem is that the yarn.scheduler.capacity.root.default-node-label-expression property is set to " " instead of "", and this behavior changed in HDP 2.3, but works in HDP 2.2 as well.
Diffs (updated)
-----
ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml e954af2
ambari-server/src/main/resources/stacks/HDP/2.3.GlusterFS/services/YARN/configuration/capacity-scheduler.xml 63d56b0
ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/configuration/capacity-scheduler.xml PRE-CREATION
Diff: https://reviews.apache.org/r/34451/diff/
Testing
-------
Installed ambari-server
Patched the file capacity-scheduler.xml
Deployed a cluster with HDP 2.3 and HDFS, YARN, MR, ZK
Verified the property change in the configs, and that the YARN Service Check passed.
----------------------------------------------------------------------
Total run:741
Total errors:0
Total failures:0
OK
Thanks,
Alejandro Fernandez
Re: Review Request 34451: YARN Service Check job and Oozie job don't
run because of whitespace in
yarn.scheduler.capacity.root.default-node-label-expression
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34451/
-----------------------------------------------------------
(Updated May 20, 2015, 2:54 a.m.)
Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
Bugs: AMBARI-11257
https://issues.apache.org/jira/browse/AMBARI-11257
Repository: ambari
Description
-------
STR:
Create a cluster with Ambari 2.1.0, at build 1638
Install HDP 2.3.0.0 at build 2030, with HDFS, MR, YARN, ZK, OOZIE
Run Yarn Service Check
Run Oozie Service Check
The Resource Manager UI shows the apps as waiting for AM to schedule them, and eventually they get killed.
For Oozie, it is able to launch the MR jobs, but of course, they timeout.
The problem is that the yarn.scheduler.capacity.root.default-node-label-expression property is set to " " instead of "", and this behavior changed in HDP 2.3, but works in HDP 2.2 as well.
Diffs (updated)
-----
ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/configuration/capacity-scheduler.xml f1d4b48
ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml e954af2
ambari-server/src/main/resources/stacks/HDP/2.3.GlusterFS/services/YARN/configuration/capacity-scheduler.xml 63d56b0
ambari-server/src/main/resources/stacks/HDPWIN/2.2/services/YARN/configuration/capacity-scheduler.xml 63d56b0
Diff: https://reviews.apache.org/r/34451/diff/
Testing
-------
Installed ambari-server
Patched the file capacity-scheduler.xml
Deployed a cluster with HDP 2.3 and HDFS, YARN, MR, ZK
Verified the property change in the configs, and that the YARN Service Check passed.
----------------------------------------------------------------------
Total run:741
Total errors:0
Total failures:0
OK
Thanks,
Alejandro Fernandez
Re: Review Request 34451: YARN Service Check job and Oozie job don't
run because of whitespace in
yarn.scheduler.capacity.root.default-node-label-expression
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34451/#review84448
-----------------------------------------------------------
ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_1.3_to_2.2.json
<https://reviews.apache.org/r/34451/#comment135720>
Isn't this a change only for 2.3?
ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_2.1_to_2.2.2.json
<https://reviews.apache.org/r/34451/#comment135721>
Isn't this a change only for 2.3?
- Sumit Mohanty
On May 20, 2015, 2:29 a.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34451/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 2:29 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-11257
> https://issues.apache.org/jira/browse/AMBARI-11257
>
>
> Repository: ambari
>
>
> Description
> -------
>
> STR:
> Create a cluster with Ambari 2.1.0, at build 1638
> Install HDP 2.3.0.0 at build 2030, with HDFS, MR, YARN, ZK, OOZIE
> Run Yarn Service Check
> Run Oozie Service Check
>
> The Resource Manager UI shows the apps as waiting for AM to schedule them, and eventually they get killed.
>
> For Oozie, it is able to launch the MR jobs, but of course, they timeout.
>
> The problem is that the yarn.scheduler.capacity.root.default-node-label-expression property is set to " " instead of "", and this behavior changed in HDP 2.3, but works in HDP 2.2 as well.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/configuration/capacity-scheduler.xml f1d4b48
> ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml e954af2
> ambari-server/src/main/resources/stacks/HDP/2.3.GlusterFS/services/YARN/configuration/capacity-scheduler.xml 63d56b0
> ambari-server/src/main/resources/stacks/HDPWIN/2.2/services/YARN/configuration/capacity-scheduler.xml 63d56b0
> ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_1.3_to_2.2.json f1d08b9
> ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_2.0_to_2.2.2.json 86521d1
> ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_2.0_to_2.2.4.json 7721c67
> ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_2.1_to_2.2.2.json a040268
> ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_2.1_to_2.2.4.json ba2b3d7
> ambari-server/src/test/python/stacks/2.0.6/configs/client-upgrade.json aeecb79
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase-2.2.json fa48c6d
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase-check-2.2.json 88017b5
> ambari-server/src/test/python/stacks/2.0.6/configs/hbase-rs-2.2.json 3109b7a
> ambari-server/src/test/python/stacks/2.1/configs/client-upgrade.json 1534619
> ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade-hdfs-secure.json 3b6a2fa
> ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade.json 9d8cb39
> ambari-server/src/test/python/stacks/2.2/configs/pig-service-check-secure.json a7a6121
> ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json 3e6307d
> ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json caec8ca
>
> Diff: https://reviews.apache.org/r/34451/diff/
>
>
> Testing
> -------
>
> Installed ambari-server
> Patched the file capacity-scheduler.xml
> Deployed a cluster with HDP 2.3 and HDFS, YARN, MR, ZK
> Verified the property change in the configs, and that the YARN Service Check passed.
>
> ----------------------------------------------------------------------
> Total run:741
> Total errors:0
> Total failures:0
> OK
>
>
> Thanks,
>
> Alejandro Fernandez
>
>
Re: Review Request 34451: YARN Service Check job and Oozie job don't
run because of whitespace in
yarn.scheduler.capacity.root.default-node-label-expression
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34451/
-----------------------------------------------------------
(Updated May 20, 2015, 2:29 a.m.)
Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid Wagle.
Changes
-------
Added changes for Upgrade Pack.
Bugs: AMBARI-11257
https://issues.apache.org/jira/browse/AMBARI-11257
Repository: ambari
Description
-------
STR:
Create a cluster with Ambari 2.1.0, at build 1638
Install HDP 2.3.0.0 at build 2030, with HDFS, MR, YARN, ZK, OOZIE
Run Yarn Service Check
Run Oozie Service Check
The Resource Manager UI shows the apps as waiting for AM to schedule them, and eventually they get killed.
For Oozie, it is able to launch the MR jobs, but of course, they timeout.
The problem is that the yarn.scheduler.capacity.root.default-node-label-expression property is set to " " instead of "", and this behavior changed in HDP 2.3, but works in HDP 2.2 as well.
Diffs (updated)
-----
ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/configuration/capacity-scheduler.xml f1d4b48
ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml e954af2
ambari-server/src/main/resources/stacks/HDP/2.3.GlusterFS/services/YARN/configuration/capacity-scheduler.xml 63d56b0
ambari-server/src/main/resources/stacks/HDPWIN/2.2/services/YARN/configuration/capacity-scheduler.xml 63d56b0
ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_1.3_to_2.2.json f1d08b9
ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_2.0_to_2.2.2.json 86521d1
ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_2.0_to_2.2.4.json 7721c67
ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_2.1_to_2.2.2.json a040268
ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_2.1_to_2.2.4.json ba2b3d7
ambari-server/src/test/python/stacks/2.0.6/configs/client-upgrade.json aeecb79
ambari-server/src/test/python/stacks/2.0.6/configs/hbase-2.2.json fa48c6d
ambari-server/src/test/python/stacks/2.0.6/configs/hbase-check-2.2.json 88017b5
ambari-server/src/test/python/stacks/2.0.6/configs/hbase-rs-2.2.json 3109b7a
ambari-server/src/test/python/stacks/2.1/configs/client-upgrade.json 1534619
ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade-hdfs-secure.json 3b6a2fa
ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade.json 9d8cb39
ambari-server/src/test/python/stacks/2.2/configs/pig-service-check-secure.json a7a6121
ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json 3e6307d
ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json caec8ca
Diff: https://reviews.apache.org/r/34451/diff/
Testing
-------
Installed ambari-server
Patched the file capacity-scheduler.xml
Deployed a cluster with HDP 2.3 and HDFS, YARN, MR, ZK
Verified the property change in the configs, and that the YARN Service Check passed.
----------------------------------------------------------------------
Total run:741
Total errors:0
Total failures:0
OK
Thanks,
Alejandro Fernandez
Re: Review Request 34451: YARN Service Check job and Oozie job don't
run because of whitespace in
yarn.scheduler.capacity.root.default-node-label-expression
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34451/
-----------------------------------------------------------
(Updated May 20, 2015, 12:31 a.m.)
Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, Sumit Mohanty, and Sid Wagle.
Bugs: AMBARI-11257
https://issues.apache.org/jira/browse/AMBARI-11257
Repository: ambari
Description (updated)
-------
STR:
Create a cluster with Ambari 2.1.0, at build 1638
Install HDP 2.3.0.0 at build 2030, with HDFS, MR, YARN, ZK, OOZIE
Run Yarn Service Check
Run Oozie Service Check
The Resource Manager UI shows the apps as waiting for AM to schedule them, and eventually they get killed.
For Oozie, it is able to launch the MR jobs, but of course, they timeout.
The problem is that the yarn.scheduler.capacity.root.default-node-label-expression property is set to " " instead of "", and this behavior changed in HDP 2.3, but works in HDP 2.2 as well.
Diffs
-----
ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/configuration/capacity-scheduler.xml f1d4b48
ambari-server/src/main/resources/stacks/HDP/2.3.GlusterFS/services/YARN/configuration/capacity-scheduler.xml 63d56b0
ambari-server/src/main/resources/stacks/HDPWIN/2.2/services/YARN/configuration/capacity-scheduler.xml 63d56b0
ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_1.3_to_2.2.json f1d08b9
ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_2.0_to_2.2.2.json 86521d1
ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_2.0_to_2.2.4.json 7721c67
ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_2.1_to_2.2.2.json a040268
ambari-server/src/main/resources/upgrade/catalog/UpgradeCatalog_2.1_to_2.2.4.json ba2b3d7
ambari-server/src/test/python/stacks/2.0.6/configs/client-upgrade.json aeecb79
ambari-server/src/test/python/stacks/2.0.6/configs/hbase-2.2.json fa48c6d
ambari-server/src/test/python/stacks/2.0.6/configs/hbase-check-2.2.json 88017b5
ambari-server/src/test/python/stacks/2.0.6/configs/hbase-rs-2.2.json 3109b7a
ambari-server/src/test/python/stacks/2.1/configs/client-upgrade.json 1534619
ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade-hdfs-secure.json 3b6a2fa
ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade.json 9d8cb39
ambari-server/src/test/python/stacks/2.2/configs/pig-service-check-secure.json a7a6121
ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json 3e6307d
ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json caec8ca
Diff: https://reviews.apache.org/r/34451/diff/
Testing
-------
Installed ambari-server
Patched the file capacity-scheduler.xml
Deployed a cluster with HDP 2.3 and HDFS, YARN, MR, ZK
Verified the property change in the configs, and that the YARN Service Check passed.
----------------------------------------------------------------------
Total run:741
Total errors:0
Total failures:0
OK
Thanks,
Alejandro Fernandez