You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Robert Nettleton <rn...@hortonworks.com> on 2015/05/27 23:33:28 UTC
Review Request 34735: Implement support in Yarn stack scripts to
create directory for "yarn.timeline-service.leveldb-state-store.path".
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34735/
-----------------------------------------------------------
Review request for Ambari, Jayush Luniya, John Speidel, Mahadev Konar, Robert Levas, and Sumit Mohanty.
Bugs: AMBARI-11442
https://issues.apache.org/jira/browse/AMBARI-11442
Repository: ambari
Description
-------
This patch addresses AMBARI-11442.
The Ambari stack scripts for Yarn currently have a problem regarding the startup of the Yarn Application Timeline Service.
The timeline service has two properties for DB storage:
yarn.timeline-service.leveldb-timeline-store.path
yarn.timeline-service.leveldb-state-store.path
The current Yarn stack scripts only create a directory for "yarn.timeline-service.leveldb-timeline-store.path" during a cluster startup.
If the default settings are used for these properties, then things will work fine, since both are mapped to the same root directory.
If the two properties are set to different values, then the App Timeline Service will fail to startup, since the directory associated with "yarn.timeline-service.leveldb-state-store.path" will not have been created.
The "yarn.timeline-service.leveldb-state-store.path" property was added in Ambari 2.1 in HDP 2.2. The rest of the Yarn stack code should be updated to use this variable to create a separate directory, if needed.
This patch addresses the problem by implementing the following:
1. Adds a new parameter to read in "yarn.timeline-service.leveldb-state-store.path" whenever HDP 2.2 or greater is used (this new property was introduced in the HDP 2.2 stacks).
2. Adds some code to yarn.py to create this new directory, whenever HDP 2.2 or greater is selected as the stack.
Diffs
-----
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py 67b9565
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py d77654a
Diff: https://reviews.apache.org/r/34735/diff/
Testing
-------
1. Ran the Python unit tests (all passing).
2. Deployed a single-node Yarn cluster (HDP 2.3) with the Ambari UI, configuring these two properties to point towards different directories. Verified that the App Timeline Service starts up properly (as well as the rest of Yarn), and that Ambari creates the two separate directories.
3. Deployed a single-node Yarn cluster (HDP 2.3) with the Ambari UI, using all the defaults provided by Ambari UI for Yarn config, and verified that the App Timeline Service starts up properly (as well as the rest of Yarn).
4. Deployed a single-node Yarn cluster with HDP 2.1 with the UI defaults. This Yarn deployment failed, but upon further investigation I found that this stack deployment fails without my patch applied, and so is a separate issue in the Ambari Yarn stacks.
Thanks,
Robert Nettleton
Re: Review Request 34735: Implement support in Yarn stack scripts to
create directory for "yarn.timeline-service.leveldb-state-store.path".
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34735/#review85490
-----------------------------------------------------------
Ship it!
Ship It!
- Sumit Mohanty
On May 27, 2015, 9:33 p.m., Robert Nettleton wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34735/
> -----------------------------------------------------------
>
> (Updated May 27, 2015, 9:33 p.m.)
>
>
> Review request for Ambari, Jayush Luniya, John Speidel, Mahadev Konar, Robert Levas, and Sumit Mohanty.
>
>
> Bugs: AMBARI-11442
> https://issues.apache.org/jira/browse/AMBARI-11442
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This patch addresses AMBARI-11442.
>
> The Ambari stack scripts for Yarn currently have a problem regarding the startup of the Yarn Application Timeline Service.
>
> The timeline service has two properties for DB storage:
>
> yarn.timeline-service.leveldb-timeline-store.path
> yarn.timeline-service.leveldb-state-store.path
>
> The current Yarn stack scripts only create a directory for "yarn.timeline-service.leveldb-timeline-store.path" during a cluster startup.
>
> If the default settings are used for these properties, then things will work fine, since both are mapped to the same root directory.
>
> If the two properties are set to different values, then the App Timeline Service will fail to startup, since the directory associated with "yarn.timeline-service.leveldb-state-store.path" will not have been created.
>
> The "yarn.timeline-service.leveldb-state-store.path" property was added in Ambari 2.1 in HDP 2.2. The rest of the Yarn stack code should be updated to use this variable to create a separate directory, if needed.
>
> This patch addresses the problem by implementing the following:
>
> 1. Adds a new parameter to read in "yarn.timeline-service.leveldb-state-store.path" whenever HDP 2.2 or greater is used (this new property was introduced in the HDP 2.2 stacks).
> 2. Adds some code to yarn.py to create this new directory, whenever HDP 2.2 or greater is selected as the stack.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py 67b9565
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py d77654a
>
> Diff: https://reviews.apache.org/r/34735/diff/
>
>
> Testing
> -------
>
> 1. Ran the Python unit tests (all passing).
> 2. Deployed a single-node Yarn cluster (HDP 2.3) with the Ambari UI, configuring these two properties to point towards different directories. Verified that the App Timeline Service starts up properly (as well as the rest of Yarn), and that Ambari creates the two separate directories.
> 3. Deployed a single-node Yarn cluster (HDP 2.3) with the Ambari UI, using all the defaults provided by Ambari UI for Yarn config, and verified that the App Timeline Service starts up properly (as well as the rest of Yarn).
> 4. Deployed a single-node Yarn cluster with HDP 2.1 with the UI defaults. This Yarn deployment failed, but upon further investigation I found that this stack deployment fails without my patch applied, and so is a separate issue in the Ambari Yarn stacks.
>
>
> Thanks,
>
> Robert Nettleton
>
>
Re: Review Request 34735: Implement support in Yarn stack scripts to
create directory for "yarn.timeline-service.leveldb-state-store.path".
Posted by Jayush Luniya <jl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34735/#review85461
-----------------------------------------------------------
Ship it!
Ship It!
- Jayush Luniya
On May 27, 2015, 9:33 p.m., Robert Nettleton wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34735/
> -----------------------------------------------------------
>
> (Updated May 27, 2015, 9:33 p.m.)
>
>
> Review request for Ambari, Jayush Luniya, John Speidel, Mahadev Konar, Robert Levas, and Sumit Mohanty.
>
>
> Bugs: AMBARI-11442
> https://issues.apache.org/jira/browse/AMBARI-11442
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This patch addresses AMBARI-11442.
>
> The Ambari stack scripts for Yarn currently have a problem regarding the startup of the Yarn Application Timeline Service.
>
> The timeline service has two properties for DB storage:
>
> yarn.timeline-service.leveldb-timeline-store.path
> yarn.timeline-service.leveldb-state-store.path
>
> The current Yarn stack scripts only create a directory for "yarn.timeline-service.leveldb-timeline-store.path" during a cluster startup.
>
> If the default settings are used for these properties, then things will work fine, since both are mapped to the same root directory.
>
> If the two properties are set to different values, then the App Timeline Service will fail to startup, since the directory associated with "yarn.timeline-service.leveldb-state-store.path" will not have been created.
>
> The "yarn.timeline-service.leveldb-state-store.path" property was added in Ambari 2.1 in HDP 2.2. The rest of the Yarn stack code should be updated to use this variable to create a separate directory, if needed.
>
> This patch addresses the problem by implementing the following:
>
> 1. Adds a new parameter to read in "yarn.timeline-service.leveldb-state-store.path" whenever HDP 2.2 or greater is used (this new property was introduced in the HDP 2.2 stacks).
> 2. Adds some code to yarn.py to create this new directory, whenever HDP 2.2 or greater is selected as the stack.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py 67b9565
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py d77654a
>
> Diff: https://reviews.apache.org/r/34735/diff/
>
>
> Testing
> -------
>
> 1. Ran the Python unit tests (all passing).
> 2. Deployed a single-node Yarn cluster (HDP 2.3) with the Ambari UI, configuring these two properties to point towards different directories. Verified that the App Timeline Service starts up properly (as well as the rest of Yarn), and that Ambari creates the two separate directories.
> 3. Deployed a single-node Yarn cluster (HDP 2.3) with the Ambari UI, using all the defaults provided by Ambari UI for Yarn config, and verified that the App Timeline Service starts up properly (as well as the rest of Yarn).
> 4. Deployed a single-node Yarn cluster with HDP 2.1 with the UI defaults. This Yarn deployment failed, but upon further investigation I found that this stack deployment fails without my patch applied, and so is a separate issue in the Ambari Yarn stacks.
>
>
> Thanks,
>
> Robert Nettleton
>
>
Re: Review Request 34735: Implement support in Yarn stack scripts to
create directory for "yarn.timeline-service.leveldb-state-store.path".
Posted by Robert Levas <rl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34735/#review85493
-----------------------------------------------------------
Ship it!
Ship It!
- Robert Levas
On May 27, 2015, 5:33 p.m., Robert Nettleton wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34735/
> -----------------------------------------------------------
>
> (Updated May 27, 2015, 5:33 p.m.)
>
>
> Review request for Ambari, Jayush Luniya, John Speidel, Mahadev Konar, Robert Levas, and Sumit Mohanty.
>
>
> Bugs: AMBARI-11442
> https://issues.apache.org/jira/browse/AMBARI-11442
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This patch addresses AMBARI-11442.
>
> The Ambari stack scripts for Yarn currently have a problem regarding the startup of the Yarn Application Timeline Service.
>
> The timeline service has two properties for DB storage:
>
> yarn.timeline-service.leveldb-timeline-store.path
> yarn.timeline-service.leveldb-state-store.path
>
> The current Yarn stack scripts only create a directory for "yarn.timeline-service.leveldb-timeline-store.path" during a cluster startup.
>
> If the default settings are used for these properties, then things will work fine, since both are mapped to the same root directory.
>
> If the two properties are set to different values, then the App Timeline Service will fail to startup, since the directory associated with "yarn.timeline-service.leveldb-state-store.path" will not have been created.
>
> The "yarn.timeline-service.leveldb-state-store.path" property was added in Ambari 2.1 in HDP 2.2. The rest of the Yarn stack code should be updated to use this variable to create a separate directory, if needed.
>
> This patch addresses the problem by implementing the following:
>
> 1. Adds a new parameter to read in "yarn.timeline-service.leveldb-state-store.path" whenever HDP 2.2 or greater is used (this new property was introduced in the HDP 2.2 stacks).
> 2. Adds some code to yarn.py to create this new directory, whenever HDP 2.2 or greater is selected as the stack.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py 67b9565
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py d77654a
>
> Diff: https://reviews.apache.org/r/34735/diff/
>
>
> Testing
> -------
>
> 1. Ran the Python unit tests (all passing).
> 2. Deployed a single-node Yarn cluster (HDP 2.3) with the Ambari UI, configuring these two properties to point towards different directories. Verified that the App Timeline Service starts up properly (as well as the rest of Yarn), and that Ambari creates the two separate directories.
> 3. Deployed a single-node Yarn cluster (HDP 2.3) with the Ambari UI, using all the defaults provided by Ambari UI for Yarn config, and verified that the App Timeline Service starts up properly (as well as the rest of Yarn).
> 4. Deployed a single-node Yarn cluster with HDP 2.1 with the UI defaults. This Yarn deployment failed, but upon further investigation I found that this stack deployment fails without my patch applied, and so is a separate issue in the Ambari Yarn stacks.
>
>
> Thanks,
>
> Robert Nettleton
>
>