You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Tom Beerbower <tb...@hortonworks.com> on 2015/03/23 21:35:37 UTC
Review Request 32414: Ability to set rack awareness
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32414/
-----------------------------------------------------------
Review request for Ambari, Jonathan Hurley, John Speidel, and Nate Cole.
Bugs: AMBARI-6466
https://issues.apache.org/jira/browse/AMBARI-6466
Repository: ambari
Description
-------
1. Generate topology script used by Hadoop to determine the rack location of nodes.
2. Update core-site.xml to point to script.
3. Provide mechanism for marking services that need restart when host rack info changes.
4. React to host resource rack changes to ...
a. mark services needing restart.
b. update host/rack data for topology script
Diffs
-----
ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py 8c21503
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 982f10f
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java 6d35a36
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 98390fd
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java 3a359e5
ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java ab1ede5
ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 939a496
ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java de84f35
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/core-site.xml c0cd6b3
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 916d9b0
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/files/topology_script.py PRE-CREATION
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py 615dd54
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params.py d20e8e0
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/templates/topology_mappings.data.j2 PRE-CREATION
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/metainfo.xml 974165c
ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java 8f7c199
ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerImplTest.java 208218c
ambari-server/src/test/java/org/apache/ambari/server/state/ServiceInfoTest.java PRE-CREATION
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py 1919958
ambari-server/src/test/python/stacks/2.0.6/configs/default.json d1701bd
ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_active_node.json b3e61bc
ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_standby_node.json 4bbb0f5
ambari-server/src/test/python/stacks/2.0.6/configs/ha_default.json 652ca7b
ambari-server/src/test/python/stacks/2.0.6/configs/ha_secured.json 6c06f35
ambari-server/src/test/python/stacks/2.0.6/configs/secured.json b0b0c04
ambari-server/src/test/resources/stacks/HDP/2.0.7/services/HDFS/metainfo.xml 69f13e9
Diff: https://reviews.apache.org/r/32414/diff/
Testing
-------
Manual testing to verify that topology script is configured and available, services (HDFS, MAPRED2) are marked for restart when a change is made to host rack information, and topology script can be invoked to get updated rack information.
New unit tests added.
All unit tests pass.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 33:36 min
[INFO] Finished at: 2015-03-23T15:47:10-04:00
[INFO] Final Memory: 43M/600M
[INFO] ------------------------------------------------------------------------
Thanks,
Tom Beerbower
Re: Review Request 32414: Ability to set rack awareness
Posted by Tom Beerbower <tb...@hortonworks.com>.
> On March 24, 2015, 5:52 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py, line 288
> > <https://reviews.apache.org/r/32414/diff/1/?file=903411#file903411line288>
> >
> > Should these be added to Script.py or another class, in case other services also want to inherit it?
Thanks for the review!
This makes sense. It really shouldn't be tied to a single service.
Maybe one issue with moving it... The current code accesses some params defined in the HDFS params.py.
def create_topology_mapping():
import params
# placing the mappings file in the same folder where the topology script is located
File(params.net_topology_mapping_data_file_path,
content=Template("topology_mappings.data.j2"),
owner=params.hdfs_user,
group=params.user_group
)
If this code is moved to a common script, where do the params come from?
- Tom
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32414/#review77604
-----------------------------------------------------------
On March 23, 2015, 8:35 p.m., Tom Beerbower wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32414/
> -----------------------------------------------------------
>
> (Updated March 23, 2015, 8:35 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, John Speidel, and Nate Cole.
>
>
> Bugs: AMBARI-6466
> https://issues.apache.org/jira/browse/AMBARI-6466
>
>
> Repository: ambari
>
>
> Description
> -------
>
> 1. Generate topology script used by Hadoop to determine the rack location of nodes.
> 2. Update core-site.xml to point to script.
> 3. Provide mechanism for marking services that need restart when host rack info changes.
> 4. React to host resource rack changes to ...
> a. mark services needing restart.
> b. update host/rack data for topology script
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py 8c21503
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 982f10f
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java 6d35a36
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 98390fd
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java 3a359e5
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java ab1ede5
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 939a496
> ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java de84f35
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/core-site.xml c0cd6b3
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 916d9b0
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/files/topology_script.py PRE-CREATION
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py 615dd54
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params.py d20e8e0
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/templates/topology_mappings.data.j2 PRE-CREATION
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/metainfo.xml 974165c
> ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java 8f7c199
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerImplTest.java 208218c
> ambari-server/src/test/java/org/apache/ambari/server/state/ServiceInfoTest.java PRE-CREATION
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py 1919958
> ambari-server/src/test/python/stacks/2.0.6/configs/default.json d1701bd
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_active_node.json b3e61bc
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_standby_node.json 4bbb0f5
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_default.json 652ca7b
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_secured.json 6c06f35
> ambari-server/src/test/python/stacks/2.0.6/configs/secured.json b0b0c04
> ambari-server/src/test/resources/stacks/HDP/2.0.7/services/HDFS/metainfo.xml 69f13e9
>
> Diff: https://reviews.apache.org/r/32414/diff/
>
>
> Testing
> -------
>
> Manual testing to verify that topology script is configured and available, services (HDFS, MAPRED2) are marked for restart when a change is made to host rack information, and topology script can be invoked to get updated rack information.
>
> New unit tests added.
>
> All unit tests pass.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 33:36 min
> [INFO] Finished at: 2015-03-23T15:47:10-04:00
> [INFO] Final Memory: 43M/600M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 32414: Ability to set rack awareness
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On March 24, 2015, 5:52 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py, line 288
> > <https://reviews.apache.org/r/32414/diff/1/?file=903411#file903411line288>
> >
> > Should these be added to Script.py or another class, in case other services also want to inherit it?
>
> Tom Beerbower wrote:
> Thanks for the review!
>
> This makes sense. It really shouldn't be tied to a single service.
>
> Maybe one issue with moving it... The current code accesses some params defined in the HDFS params.py.
>
>
> def create_topology_mapping():
> import params
>
> # placing the mappings file in the same folder where the topology script is located
> File(params.net_topology_mapping_data_file_path,
> content=Template("topology_mappings.data.j2"),
> owner=params.hdfs_user,
> group=params.user_group
> )
>
> If this code is moved to a common script, where do the params come from?
I'm ok with it as it, as long as a future patch moves it to a shared class. I know Script.py has some issues with circular dependencies that are kinda ugly right now.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32414/#review77604
-----------------------------------------------------------
On March 23, 2015, 8:35 p.m., Tom Beerbower wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32414/
> -----------------------------------------------------------
>
> (Updated March 23, 2015, 8:35 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, John Speidel, and Nate Cole.
>
>
> Bugs: AMBARI-6466
> https://issues.apache.org/jira/browse/AMBARI-6466
>
>
> Repository: ambari
>
>
> Description
> -------
>
> 1. Generate topology script used by Hadoop to determine the rack location of nodes.
> 2. Update core-site.xml to point to script.
> 3. Provide mechanism for marking services that need restart when host rack info changes.
> 4. React to host resource rack changes to ...
> a. mark services needing restart.
> b. update host/rack data for topology script
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py 8c21503
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 982f10f
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java 6d35a36
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 98390fd
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java 3a359e5
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java ab1ede5
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 939a496
> ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java de84f35
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/core-site.xml c0cd6b3
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 916d9b0
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/files/topology_script.py PRE-CREATION
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py 615dd54
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params.py d20e8e0
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/templates/topology_mappings.data.j2 PRE-CREATION
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/metainfo.xml 974165c
> ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java 8f7c199
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerImplTest.java 208218c
> ambari-server/src/test/java/org/apache/ambari/server/state/ServiceInfoTest.java PRE-CREATION
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py 1919958
> ambari-server/src/test/python/stacks/2.0.6/configs/default.json d1701bd
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_active_node.json b3e61bc
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_standby_node.json 4bbb0f5
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_default.json 652ca7b
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_secured.json 6c06f35
> ambari-server/src/test/python/stacks/2.0.6/configs/secured.json b0b0c04
> ambari-server/src/test/resources/stacks/HDP/2.0.7/services/HDFS/metainfo.xml 69f13e9
>
> Diff: https://reviews.apache.org/r/32414/diff/
>
>
> Testing
> -------
>
> Manual testing to verify that topology script is configured and available, services (HDFS, MAPRED2) are marked for restart when a change is made to host rack information, and topology script can be invoked to get updated rack information.
>
> New unit tests added.
>
> All unit tests pass.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 33:36 min
> [INFO] Finished at: 2015-03-23T15:47:10-04:00
> [INFO] Final Memory: 43M/600M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 32414: Ability to set rack awareness
Posted by Tom Beerbower <tb...@hortonworks.com>.
> On March 24, 2015, 5:52 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py, line 288
> > <https://reviews.apache.org/r/32414/diff/1/?file=903411#file903411line288>
> >
> > Should these be added to Script.py or another class, in case other services also want to inherit it?
>
> Tom Beerbower wrote:
> Thanks for the review!
>
> This makes sense. It really shouldn't be tied to a single service.
>
> Maybe one issue with moving it... The current code accesses some params defined in the HDFS params.py.
>
>
> def create_topology_mapping():
> import params
>
> # placing the mappings file in the same folder where the topology script is located
> File(params.net_topology_mapping_data_file_path,
> content=Template("topology_mappings.data.j2"),
> owner=params.hdfs_user,
> group=params.user_group
> )
>
> If this code is moved to a common script, where do the params come from?
>
> Alejandro Fernandez wrote:
> I'm ok with it as it, as long as a future patch moves it to a shared class. I know Script.py has some issues with circular dependencies that are kinda ugly right now.
Yeah, I'm looking at that now. Unfortunately I can't leave it as is. As Nate pointed out, this patch will only update the topology mapping on the host that is running NN. It needs to be done on any host running a service that is rack sensitive. So, I need to move it to a common place and have it run on start/restart.
Thanks.
- Tom
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32414/#review77604
-----------------------------------------------------------
On March 23, 2015, 8:35 p.m., Tom Beerbower wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32414/
> -----------------------------------------------------------
>
> (Updated March 23, 2015, 8:35 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, John Speidel, and Nate Cole.
>
>
> Bugs: AMBARI-6466
> https://issues.apache.org/jira/browse/AMBARI-6466
>
>
> Repository: ambari
>
>
> Description
> -------
>
> 1. Generate topology script used by Hadoop to determine the rack location of nodes.
> 2. Update core-site.xml to point to script.
> 3. Provide mechanism for marking services that need restart when host rack info changes.
> 4. React to host resource rack changes to ...
> a. mark services needing restart.
> b. update host/rack data for topology script
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py 8c21503
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 982f10f
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java 6d35a36
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 98390fd
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java 3a359e5
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java ab1ede5
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 939a496
> ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java de84f35
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/core-site.xml c0cd6b3
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 916d9b0
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/files/topology_script.py PRE-CREATION
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py 615dd54
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params.py d20e8e0
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/templates/topology_mappings.data.j2 PRE-CREATION
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/metainfo.xml 974165c
> ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java 8f7c199
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerImplTest.java 208218c
> ambari-server/src/test/java/org/apache/ambari/server/state/ServiceInfoTest.java PRE-CREATION
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py 1919958
> ambari-server/src/test/python/stacks/2.0.6/configs/default.json d1701bd
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_active_node.json b3e61bc
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_standby_node.json 4bbb0f5
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_default.json 652ca7b
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_secured.json 6c06f35
> ambari-server/src/test/python/stacks/2.0.6/configs/secured.json b0b0c04
> ambari-server/src/test/resources/stacks/HDP/2.0.7/services/HDFS/metainfo.xml 69f13e9
>
> Diff: https://reviews.apache.org/r/32414/diff/
>
>
> Testing
> -------
>
> Manual testing to verify that topology script is configured and available, services (HDFS, MAPRED2) are marked for restart when a change is made to host rack information, and topology script can be invoked to get updated rack information.
>
> New unit tests added.
>
> All unit tests pass.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 33:36 min
> [INFO] Finished at: 2015-03-23T15:47:10-04:00
> [INFO] Final Memory: 43M/600M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 32414: Ability to set rack awareness
Posted by Nate Cole <nc...@hortonworks.com>.
> On March 24, 2015, 1:52 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py, line 288
> > <https://reviews.apache.org/r/32414/diff/1/?file=903411#file903411line288>
> >
> > Should these be added to Script.py or another class, in case other services also want to inherit it?
>
> Tom Beerbower wrote:
> Thanks for the review!
>
> This makes sense. It really shouldn't be tied to a single service.
>
> Maybe one issue with moving it... The current code accesses some params defined in the HDFS params.py.
>
>
> def create_topology_mapping():
> import params
>
> # placing the mappings file in the same folder where the topology script is located
> File(params.net_topology_mapping_data_file_path,
> content=Template("topology_mappings.data.j2"),
> owner=params.hdfs_user,
> group=params.user_group
> )
>
> If this code is moved to a common script, where do the params come from?
>
> Alejandro Fernandez wrote:
> I'm ok with it as it, as long as a future patch moves it to a shared class. I know Script.py has some issues with circular dependencies that are kinda ugly right now.
>
> Tom Beerbower wrote:
> Yeah, I'm looking at that now. Unfortunately I can't leave it as is. As Nate pointed out, this patch will only update the topology mapping on the host that is running NN. It needs to be done on any host running a service that is rack sensitive. So, I need to move it to a common place and have it run on start/restart.
>
> Thanks.
When you call this functionality, you would have to pass in the values from params.py. Maybe path + file + template contents or something.
- Nate
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32414/#review77604
-----------------------------------------------------------
On March 23, 2015, 4:35 p.m., Tom Beerbower wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32414/
> -----------------------------------------------------------
>
> (Updated March 23, 2015, 4:35 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, John Speidel, and Nate Cole.
>
>
> Bugs: AMBARI-6466
> https://issues.apache.org/jira/browse/AMBARI-6466
>
>
> Repository: ambari
>
>
> Description
> -------
>
> 1. Generate topology script used by Hadoop to determine the rack location of nodes.
> 2. Update core-site.xml to point to script.
> 3. Provide mechanism for marking services that need restart when host rack info changes.
> 4. React to host resource rack changes to ...
> a. mark services needing restart.
> b. update host/rack data for topology script
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py 8c21503
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 982f10f
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java 6d35a36
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 98390fd
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java 3a359e5
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java ab1ede5
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 939a496
> ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java de84f35
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/core-site.xml c0cd6b3
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 916d9b0
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/files/topology_script.py PRE-CREATION
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py 615dd54
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params.py d20e8e0
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/templates/topology_mappings.data.j2 PRE-CREATION
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/metainfo.xml 974165c
> ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java 8f7c199
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerImplTest.java 208218c
> ambari-server/src/test/java/org/apache/ambari/server/state/ServiceInfoTest.java PRE-CREATION
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py 1919958
> ambari-server/src/test/python/stacks/2.0.6/configs/default.json d1701bd
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_active_node.json b3e61bc
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_standby_node.json 4bbb0f5
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_default.json 652ca7b
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_secured.json 6c06f35
> ambari-server/src/test/python/stacks/2.0.6/configs/secured.json b0b0c04
> ambari-server/src/test/resources/stacks/HDP/2.0.7/services/HDFS/metainfo.xml 69f13e9
>
> Diff: https://reviews.apache.org/r/32414/diff/
>
>
> Testing
> -------
>
> Manual testing to verify that topology script is configured and available, services (HDFS, MAPRED2) are marked for restart when a change is made to host rack information, and topology script can be invoked to get updated rack information.
>
> New unit tests added.
>
> All unit tests pass.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 33:36 min
> [INFO] Finished at: 2015-03-23T15:47:10-04:00
> [INFO] Final Memory: 43M/600M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 32414: Ability to set rack awareness
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32414/#review77604
-----------------------------------------------------------
Ship it!
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/files/topology_script.py
<https://reviews.apache.org/r/32414/#comment125737>
If another class of exception is thrown, this function will not return a value.
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py
<https://reviews.apache.org/r/32414/#comment125738>
Should these be added to Script.py or another class, in case other services also want to inherit it?
- Alejandro Fernandez
On March 23, 2015, 8:35 p.m., Tom Beerbower wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32414/
> -----------------------------------------------------------
>
> (Updated March 23, 2015, 8:35 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, John Speidel, and Nate Cole.
>
>
> Bugs: AMBARI-6466
> https://issues.apache.org/jira/browse/AMBARI-6466
>
>
> Repository: ambari
>
>
> Description
> -------
>
> 1. Generate topology script used by Hadoop to determine the rack location of nodes.
> 2. Update core-site.xml to point to script.
> 3. Provide mechanism for marking services that need restart when host rack info changes.
> 4. React to host resource rack changes to ...
> a. mark services needing restart.
> b. update host/rack data for topology script
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py 8c21503
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 982f10f
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java 6d35a36
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 98390fd
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java 3a359e5
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java ab1ede5
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 939a496
> ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java de84f35
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/core-site.xml c0cd6b3
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 916d9b0
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/files/topology_script.py PRE-CREATION
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py 615dd54
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params.py d20e8e0
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/templates/topology_mappings.data.j2 PRE-CREATION
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/metainfo.xml 974165c
> ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java 8f7c199
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerImplTest.java 208218c
> ambari-server/src/test/java/org/apache/ambari/server/state/ServiceInfoTest.java PRE-CREATION
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py 1919958
> ambari-server/src/test/python/stacks/2.0.6/configs/default.json d1701bd
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_active_node.json b3e61bc
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_standby_node.json 4bbb0f5
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_default.json 652ca7b
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_secured.json 6c06f35
> ambari-server/src/test/python/stacks/2.0.6/configs/secured.json b0b0c04
> ambari-server/src/test/resources/stacks/HDP/2.0.7/services/HDFS/metainfo.xml 69f13e9
>
> Diff: https://reviews.apache.org/r/32414/diff/
>
>
> Testing
> -------
>
> Manual testing to verify that topology script is configured and available, services (HDFS, MAPRED2) are marked for restart when a change is made to host rack information, and topology script can be invoked to get updated rack information.
>
> New unit tests added.
>
> All unit tests pass.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 33:36 min
> [INFO] Finished at: 2015-03-23T15:47:10-04:00
> [INFO] Final Memory: 43M/600M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 32414: Ability to set rack awareness
Posted by Tom Beerbower <tb...@hortonworks.com>.
> On March 24, 2015, 1:24 p.m., Jonathan Hurley wrote:
> > Some minor nits and a few general questions just asking if the behavior is expected.
Thanks for reviewing!
> On March 24, 2015, 1:24 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java, line 529
> > <https://reviews.apache.org/r/32414/diff/1/?file=903401#file903401line529>
> >
> > Any reason this is a Boolean object? Seems like it could cause NPEs in when used by other areas of code.
To be honest, this was existing and I just left the type alone, but I think that there is a reason that the primitive type was not used. In ServiceModule, there is this code ...
if (serviceInfo.isRestartRequiredAfterChange() == null) {
serviceInfo.setRestartRequiredAfterChange(parent.isRestartRequiredAfterChange());
}
So, null indicates that the value is not set at this level and should be taken from the parent.
> On March 24, 2015, 1:24 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java, line 557
> > <https://reviews.apache.org/r/32414/diff/1/?file=903401#file903401line557>
> >
> > Same as above; any reason this can't be a primitive to avoid NPEs and null checks?
Same as above, I basically cut and pasted the code, but it looks like there is a valid reason for using the big B Boolean.
> On March 24, 2015, 1:24 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java, line 633
> > <https://reviews.apache.org/r/32414/diff/1/?file=903403#file903403line633>
> >
> > Slightly confusing; the first time I read this, I thought these were the services that need a restart. But they are just the ones that are rack sensitive. Can we rename this Set to something like `rackSenstiveServices`
Sure, that makes more sense.
> On March 24, 2015, 1:24 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py, line 21
> > <https://reviews.apache.org/r/32414/diff/1/?file=903411#file903411line21>
> >
> > Correct global imports.
Yes, I'll fix it.
> On March 24, 2015, 1:24 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py, line 69
> > <https://reviews.apache.org/r/32414/diff/1/?file=903411#file903411line69>
> >
> > If I'm reading this right, it's going to generate a new topology script/mapping every time NN is started. Is that what we want even if the environment is not rack aware?
Yes, we may needlessly regenerate the mapping and copy the script on NN restart if the rack info hasn't actually changed. I think that the cost of determining whether or not it has changed is probably more than just doing it given how infrequently a NN restart occurs.
- Tom
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32414/#review77558
-----------------------------------------------------------
On March 23, 2015, 8:35 p.m., Tom Beerbower wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32414/
> -----------------------------------------------------------
>
> (Updated March 23, 2015, 8:35 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, John Speidel, and Nate Cole.
>
>
> Bugs: AMBARI-6466
> https://issues.apache.org/jira/browse/AMBARI-6466
>
>
> Repository: ambari
>
>
> Description
> -------
>
> 1. Generate topology script used by Hadoop to determine the rack location of nodes.
> 2. Update core-site.xml to point to script.
> 3. Provide mechanism for marking services that need restart when host rack info changes.
> 4. React to host resource rack changes to ...
> a. mark services needing restart.
> b. update host/rack data for topology script
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py 8c21503
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 982f10f
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java 6d35a36
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 98390fd
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java 3a359e5
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java ab1ede5
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 939a496
> ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java de84f35
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/core-site.xml c0cd6b3
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 916d9b0
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/files/topology_script.py PRE-CREATION
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py 615dd54
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params.py d20e8e0
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/templates/topology_mappings.data.j2 PRE-CREATION
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/metainfo.xml 974165c
> ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java 8f7c199
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerImplTest.java 208218c
> ambari-server/src/test/java/org/apache/ambari/server/state/ServiceInfoTest.java PRE-CREATION
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py 1919958
> ambari-server/src/test/python/stacks/2.0.6/configs/default.json d1701bd
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_active_node.json b3e61bc
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_standby_node.json 4bbb0f5
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_default.json 652ca7b
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_secured.json 6c06f35
> ambari-server/src/test/python/stacks/2.0.6/configs/secured.json b0b0c04
> ambari-server/src/test/resources/stacks/HDP/2.0.7/services/HDFS/metainfo.xml 69f13e9
>
> Diff: https://reviews.apache.org/r/32414/diff/
>
>
> Testing
> -------
>
> Manual testing to verify that topology script is configured and available, services (HDFS, MAPRED2) are marked for restart when a change is made to host rack information, and topology script can be invoked to get updated rack information.
>
> New unit tests added.
>
> All unit tests pass.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 33:36 min
> [INFO] Finished at: 2015-03-23T15:47:10-04:00
> [INFO] Final Memory: 43M/600M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 32414: Ability to set rack awareness
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32414/#review77558
-----------------------------------------------------------
Ship it!
Some minor nits and a few general questions just asking if the behavior is expected.
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
<https://reviews.apache.org/r/32414/#comment125671>
Any reason this is a Boolean object? Seems like it could cause NPEs in when used by other areas of code.
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
<https://reviews.apache.org/r/32414/#comment125674>
Same as above; any reason this can't be a primitive to avoid NPEs and null checks?
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
<https://reviews.apache.org/r/32414/#comment125675>
Slightly confusing; the first time I read this, I thought these were the services that need a restart. But they are just the ones that are rack sensitive. Can we rename this Set to something like `rackSenstiveServices`
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py
<https://reviews.apache.org/r/32414/#comment125676>
Correct global imports.
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py
<https://reviews.apache.org/r/32414/#comment125677>
If I'm reading this right, it's going to generate a new topology script/mapping every time NN is started. Is that what we want even if the environment is not rack aware?
- Jonathan Hurley
On March 23, 2015, 4:35 p.m., Tom Beerbower wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32414/
> -----------------------------------------------------------
>
> (Updated March 23, 2015, 4:35 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, John Speidel, and Nate Cole.
>
>
> Bugs: AMBARI-6466
> https://issues.apache.org/jira/browse/AMBARI-6466
>
>
> Repository: ambari
>
>
> Description
> -------
>
> 1. Generate topology script used by Hadoop to determine the rack location of nodes.
> 2. Update core-site.xml to point to script.
> 3. Provide mechanism for marking services that need restart when host rack info changes.
> 4. React to host resource rack changes to ...
> a. mark services needing restart.
> b. update host/rack data for topology script
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py 8c21503
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 982f10f
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java 6d35a36
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 98390fd
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java 3a359e5
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java ab1ede5
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 939a496
> ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java de84f35
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/core-site.xml c0cd6b3
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 916d9b0
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/files/topology_script.py PRE-CREATION
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py 615dd54
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params.py d20e8e0
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/templates/topology_mappings.data.j2 PRE-CREATION
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/metainfo.xml 974165c
> ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java 8f7c199
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerImplTest.java 208218c
> ambari-server/src/test/java/org/apache/ambari/server/state/ServiceInfoTest.java PRE-CREATION
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py 1919958
> ambari-server/src/test/python/stacks/2.0.6/configs/default.json d1701bd
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_active_node.json b3e61bc
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_standby_node.json 4bbb0f5
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_default.json 652ca7b
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_secured.json 6c06f35
> ambari-server/src/test/python/stacks/2.0.6/configs/secured.json b0b0c04
> ambari-server/src/test/resources/stacks/HDP/2.0.7/services/HDFS/metainfo.xml 69f13e9
>
> Diff: https://reviews.apache.org/r/32414/diff/
>
>
> Testing
> -------
>
> Manual testing to verify that topology script is configured and available, services (HDFS, MAPRED2) are marked for restart when a change is made to host rack information, and topology script can be invoked to get updated rack information.
>
> New unit tests added.
>
> All unit tests pass.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 33:36 min
> [INFO] Finished at: 2015-03-23T15:47:10-04:00
> [INFO] Final Memory: 43M/600M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 32414: Ability to set rack awareness
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32414/#review77917
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py
<https://reviews.apache.org/r/32414/#comment126248>
Nice
- Alejandro Fernandez
On March 26, 2015, 2:39 p.m., Tom Beerbower wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32414/
> -----------------------------------------------------------
>
> (Updated March 26, 2015, 2:39 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, John Speidel, and Nate Cole.
>
>
> Bugs: AMBARI-6466
> https://issues.apache.org/jira/browse/AMBARI-6466
>
>
> Repository: ambari
>
>
> Description
> -------
>
> 1. Generate topology script used by Hadoop to determine the rack location of nodes.
> 2. Update core-site.xml to point to script.
> 3. Provide mechanism for marking services that need restart when host rack info changes.
> 4. React to host resource rack changes to ...
> a. mark services needing restart.
> b. update host/rack data for topology script
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py 8c21503
> ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java 5ccf1ec
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 3ac2d0d
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 5a15f64
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java 6d35a36
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 98390fd
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java 3a359e5
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java ab1ede5
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 939a496
> ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java de84f35
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/core-site.xml c0cd6b3
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 388e83b
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/metainfo.xml 974165c
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/files/topology_script.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py c90a55c
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py 7f85428
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/templates/topology_mappings.data.j2 PRE-CREATION
> ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java 8f7c199
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelperTest.java 020f8b4
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerImplTest.java 208218c
> ambari-server/src/test/java/org/apache/ambari/server/state/ServiceInfoTest.java PRE-CREATION
> ambari-server/src/test/python/stacks/2.0.6/configs/default.json d1701bd
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-START/test_before_start.py f606130
> ambari-server/src/test/resources/stacks/HDP/2.0.7/services/HDFS/metainfo.xml 69f13e9
>
> Diff: https://reviews.apache.org/r/32414/diff/
>
>
> Testing
> -------
>
> Manual testing to verify that topology script is configured and available, services (HDFS, MAPRED2) are marked for restart when a change is made to host rack information, and topology script can be invoked to get updated rack information.
>
> New unit tests added.
>
> All unit tests pass.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 33:36 min
> [INFO] Finished at: 2015-03-23T15:47:10-04:00
> [INFO] Final Memory: 43M/600M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 32414: Ability to set rack awareness
Posted by Tom Beerbower <tb...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32414/
-----------------------------------------------------------
(Updated March 26, 2015, 2:39 p.m.)
Review request for Ambari, Jonathan Hurley, John Speidel, and Nate Cole.
Changes
-------
Updated test_before_start.py tests.
Bugs: AMBARI-6466
https://issues.apache.org/jira/browse/AMBARI-6466
Repository: ambari
Description
-------
1. Generate topology script used by Hadoop to determine the rack location of nodes.
2. Update core-site.xml to point to script.
3. Provide mechanism for marking services that need restart when host rack info changes.
4. React to host resource rack changes to ...
a. mark services needing restart.
b. update host/rack data for topology script
Diffs (updated)
-----
ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py 8c21503
ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java 5ccf1ec
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 3ac2d0d
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 5a15f64
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java 6d35a36
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 98390fd
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java 3a359e5
ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java ab1ede5
ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 939a496
ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java de84f35
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/core-site.xml c0cd6b3
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 388e83b
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/metainfo.xml 974165c
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/files/topology_script.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py c90a55c
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py 7f85428
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/templates/topology_mappings.data.j2 PRE-CREATION
ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java 8f7c199
ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelperTest.java 020f8b4
ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerImplTest.java 208218c
ambari-server/src/test/java/org/apache/ambari/server/state/ServiceInfoTest.java PRE-CREATION
ambari-server/src/test/python/stacks/2.0.6/configs/default.json d1701bd
ambari-server/src/test/python/stacks/2.0.6/hooks/before-START/test_before_start.py f606130
ambari-server/src/test/resources/stacks/HDP/2.0.7/services/HDFS/metainfo.xml 69f13e9
Diff: https://reviews.apache.org/r/32414/diff/
Testing
-------
Manual testing to verify that topology script is configured and available, services (HDFS, MAPRED2) are marked for restart when a change is made to host rack information, and topology script can be invoked to get updated rack information.
New unit tests added.
All unit tests pass.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 33:36 min
[INFO] Finished at: 2015-03-23T15:47:10-04:00
[INFO] Final Memory: 43M/600M
[INFO] ------------------------------------------------------------------------
Thanks,
Tom Beerbower
Re: Review Request 32414: Ability to set rack awareness
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32414/#review77895
-----------------------------------------------------------
Ship it!
Looks great! Nice and clean using the before-START hook. Only one comment to make a python unit test for creating the rack file in the right spot.
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py
<https://reviews.apache.org/r/32414/#comment126232>
Should have python tests for 'hooks' that can verify this works properly
- Nate Cole
On March 25, 2015, 11:21 p.m., Tom Beerbower wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32414/
> -----------------------------------------------------------
>
> (Updated March 25, 2015, 11:21 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, John Speidel, and Nate Cole.
>
>
> Bugs: AMBARI-6466
> https://issues.apache.org/jira/browse/AMBARI-6466
>
>
> Repository: ambari
>
>
> Description
> -------
>
> 1. Generate topology script used by Hadoop to determine the rack location of nodes.
> 2. Update core-site.xml to point to script.
> 3. Provide mechanism for marking services that need restart when host rack info changes.
> 4. React to host resource rack changes to ...
> a. mark services needing restart.
> b. update host/rack data for topology script
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py 8c21503
> ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java 5ccf1ec
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 3ac2d0d
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 5a15f64
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java 6d35a36
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 98390fd
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java 3a359e5
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java ab1ede5
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 939a496
> ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java de84f35
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/core-site.xml c0cd6b3
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 388e83b
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/metainfo.xml 974165c
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/files/topology_script.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py c90a55c
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py 7f85428
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/templates/topology_mappings.data.j2 PRE-CREATION
> ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java 8f7c199
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelperTest.java 020f8b4
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerImplTest.java 208218c
> ambari-server/src/test/java/org/apache/ambari/server/state/ServiceInfoTest.java PRE-CREATION
> ambari-server/src/test/resources/stacks/HDP/2.0.7/services/HDFS/metainfo.xml 69f13e9
>
> Diff: https://reviews.apache.org/r/32414/diff/
>
>
> Testing
> -------
>
> Manual testing to verify that topology script is configured and available, services (HDFS, MAPRED2) are marked for restart when a change is made to host rack information, and topology script can be invoked to get updated rack information.
>
> New unit tests added.
>
> All unit tests pass.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 33:36 min
> [INFO] Finished at: 2015-03-23T15:47:10-04:00
> [INFO] Final Memory: 43M/600M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 32414: Ability to set rack awareness
Posted by Tom Beerbower <tb...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32414/
-----------------------------------------------------------
(Updated March 26, 2015, 3:21 a.m.)
Review request for Ambari, Jonathan Hurley, John Speidel, and Nate Cole.
Bugs: AMBARI-6466
https://issues.apache.org/jira/browse/AMBARI-6466
Repository: ambari
Description
-------
1. Generate topology script used by Hadoop to determine the rack location of nodes.
2. Update core-site.xml to point to script.
3. Provide mechanism for marking services that need restart when host rack info changes.
4. React to host resource rack changes to ...
a. mark services needing restart.
b. update host/rack data for topology script
Diffs (updated)
-----
ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py 8c21503
ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java 5ccf1ec
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 3ac2d0d
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 5a15f64
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java 6d35a36
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 98390fd
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java 3a359e5
ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java ab1ede5
ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 939a496
ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java de84f35
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/core-site.xml c0cd6b3
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 388e83b
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/metainfo.xml 974165c
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/files/topology_script.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py c90a55c
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py 7f85428
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/rack_awareness.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/templates/topology_mappings.data.j2 PRE-CREATION
ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java 8f7c199
ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelperTest.java 020f8b4
ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerImplTest.java 208218c
ambari-server/src/test/java/org/apache/ambari/server/state/ServiceInfoTest.java PRE-CREATION
ambari-server/src/test/resources/stacks/HDP/2.0.7/services/HDFS/metainfo.xml 69f13e9
Diff: https://reviews.apache.org/r/32414/diff/
Testing
-------
Manual testing to verify that topology script is configured and available, services (HDFS, MAPRED2) are marked for restart when a change is made to host rack information, and topology script can be invoked to get updated rack information.
New unit tests added.
All unit tests pass.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 33:36 min
[INFO] Finished at: 2015-03-23T15:47:10-04:00
[INFO] Final Memory: 43M/600M
[INFO] ------------------------------------------------------------------------
Thanks,
Tom Beerbower
Re: Review Request 32414: Ability to set rack awareness
Posted by Tom Beerbower <tb...@hortonworks.com>.
> On March 24, 2015, 1:39 p.m., Nate Cole wrote:
> > How does YARN use this information? There are no changes there related to topology/rack switching like in the HDFS service. The value in core-site would effectively be ignored since the file wouldn't be written if YARN is on separate hosts than HDFS.
Thanks for reviewing!
I think that the approach here was based on these instructions ...
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.5/bk_system-admin-guide/content/admin_configure_rack_awareness.html
... which updates the net.topology.script.file.name in core-site.xml.
Are you saying that there is something not accounted here or are the instructions outdated or something?
- Tom
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32414/#review77560
-----------------------------------------------------------
On March 23, 2015, 8:35 p.m., Tom Beerbower wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32414/
> -----------------------------------------------------------
>
> (Updated March 23, 2015, 8:35 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, John Speidel, and Nate Cole.
>
>
> Bugs: AMBARI-6466
> https://issues.apache.org/jira/browse/AMBARI-6466
>
>
> Repository: ambari
>
>
> Description
> -------
>
> 1. Generate topology script used by Hadoop to determine the rack location of nodes.
> 2. Update core-site.xml to point to script.
> 3. Provide mechanism for marking services that need restart when host rack info changes.
> 4. React to host resource rack changes to ...
> a. mark services needing restart.
> b. update host/rack data for topology script
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py 8c21503
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 982f10f
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java 6d35a36
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 98390fd
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java 3a359e5
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java ab1ede5
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 939a496
> ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java de84f35
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/core-site.xml c0cd6b3
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 916d9b0
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/files/topology_script.py PRE-CREATION
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py 615dd54
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params.py d20e8e0
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/templates/topology_mappings.data.j2 PRE-CREATION
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/metainfo.xml 974165c
> ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java 8f7c199
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerImplTest.java 208218c
> ambari-server/src/test/java/org/apache/ambari/server/state/ServiceInfoTest.java PRE-CREATION
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py 1919958
> ambari-server/src/test/python/stacks/2.0.6/configs/default.json d1701bd
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_active_node.json b3e61bc
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_standby_node.json 4bbb0f5
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_default.json 652ca7b
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_secured.json 6c06f35
> ambari-server/src/test/python/stacks/2.0.6/configs/secured.json b0b0c04
> ambari-server/src/test/resources/stacks/HDP/2.0.7/services/HDFS/metainfo.xml 69f13e9
>
> Diff: https://reviews.apache.org/r/32414/diff/
>
>
> Testing
> -------
>
> Manual testing to verify that topology script is configured and available, services (HDFS, MAPRED2) are marked for restart when a change is made to host rack information, and topology script can be invoked to get updated rack information.
>
> New unit tests added.
>
> All unit tests pass.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 33:36 min
> [INFO] Finished at: 2015-03-23T15:47:10-04:00
> [INFO] Final Memory: 43M/600M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 32414: Ability to set rack awareness
Posted by Nate Cole <nc...@hortonworks.com>.
> On March 24, 2015, 9:39 a.m., Nate Cole wrote:
> > How does YARN use this information? There are no changes there related to topology/rack switching like in the HDFS service. The value in core-site would effectively be ignored since the file wouldn't be written if YARN is on separate hosts than HDFS.
>
> Tom Beerbower wrote:
> Thanks for reviewing!
>
> I think that the approach here was based on these instructions ...
>
> http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.5/bk_system-admin-guide/content/admin_configure_rack_awareness.html
>
> ... which updates the net.topology.script.file.name in core-site.xml.
>
> Are you saying that there is something not accounted here or are the instructions outdated or something?
That document is fine, it's saying that the core-site script location is common. My point is, that the code that (unless I read it wrong) writes out the script file is only in NN. If you install YARN anywhere except the NN host, the file won't get created (even though core-site will have the topology script defined in it).
- Nate
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32414/#review77560
-----------------------------------------------------------
On March 23, 2015, 4:35 p.m., Tom Beerbower wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32414/
> -----------------------------------------------------------
>
> (Updated March 23, 2015, 4:35 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, John Speidel, and Nate Cole.
>
>
> Bugs: AMBARI-6466
> https://issues.apache.org/jira/browse/AMBARI-6466
>
>
> Repository: ambari
>
>
> Description
> -------
>
> 1. Generate topology script used by Hadoop to determine the rack location of nodes.
> 2. Update core-site.xml to point to script.
> 3. Provide mechanism for marking services that need restart when host rack info changes.
> 4. React to host resource rack changes to ...
> a. mark services needing restart.
> b. update host/rack data for topology script
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py 8c21503
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 982f10f
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java 6d35a36
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 98390fd
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java 3a359e5
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java ab1ede5
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 939a496
> ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java de84f35
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/core-site.xml c0cd6b3
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 916d9b0
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/files/topology_script.py PRE-CREATION
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py 615dd54
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params.py d20e8e0
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/templates/topology_mappings.data.j2 PRE-CREATION
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/metainfo.xml 974165c
> ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java 8f7c199
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerImplTest.java 208218c
> ambari-server/src/test/java/org/apache/ambari/server/state/ServiceInfoTest.java PRE-CREATION
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py 1919958
> ambari-server/src/test/python/stacks/2.0.6/configs/default.json d1701bd
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_active_node.json b3e61bc
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_standby_node.json 4bbb0f5
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_default.json 652ca7b
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_secured.json 6c06f35
> ambari-server/src/test/python/stacks/2.0.6/configs/secured.json b0b0c04
> ambari-server/src/test/resources/stacks/HDP/2.0.7/services/HDFS/metainfo.xml 69f13e9
>
> Diff: https://reviews.apache.org/r/32414/diff/
>
>
> Testing
> -------
>
> Manual testing to verify that topology script is configured and available, services (HDFS, MAPRED2) are marked for restart when a change is made to host rack information, and topology script can be invoked to get updated rack information.
>
> New unit tests added.
>
> All unit tests pass.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 33:36 min
> [INFO] Finished at: 2015-03-23T15:47:10-04:00
> [INFO] Final Memory: 43M/600M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 32414: Ability to set rack awareness
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32414/#review77560
-----------------------------------------------------------
How does YARN use this information? There are no changes there related to topology/rack switching like in the HDFS service. The value in core-site would effectively be ignored since the file wouldn't be written if YARN is on separate hosts than HDFS.
- Nate Cole
On March 23, 2015, 4:35 p.m., Tom Beerbower wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32414/
> -----------------------------------------------------------
>
> (Updated March 23, 2015, 4:35 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, John Speidel, and Nate Cole.
>
>
> Bugs: AMBARI-6466
> https://issues.apache.org/jira/browse/AMBARI-6466
>
>
> Repository: ambari
>
>
> Description
> -------
>
> 1. Generate topology script used by Hadoop to determine the rack location of nodes.
> 2. Update core-site.xml to point to script.
> 3. Provide mechanism for marking services that need restart when host rack info changes.
> 4. React to host resource rack changes to ...
> a. mark services needing restart.
> b. update host/rack data for topology script
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py 8c21503
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java 982f10f
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java 6d35a36
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 98390fd
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java 3a359e5
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java ab1ede5
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 939a496
> ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java de84f35
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/core-site.xml c0cd6b3
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 916d9b0
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/files/topology_script.py PRE-CREATION
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py 615dd54
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params.py d20e8e0
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/templates/topology_mappings.data.j2 PRE-CREATION
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/metainfo.xml 974165c
> ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java 8f7c199
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerImplTest.java 208218c
> ambari-server/src/test/java/org/apache/ambari/server/state/ServiceInfoTest.java PRE-CREATION
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py 1919958
> ambari-server/src/test/python/stacks/2.0.6/configs/default.json d1701bd
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_active_node.json b3e61bc
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_standby_node.json 4bbb0f5
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_default.json 652ca7b
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_secured.json 6c06f35
> ambari-server/src/test/python/stacks/2.0.6/configs/secured.json b0b0c04
> ambari-server/src/test/resources/stacks/HDP/2.0.7/services/HDFS/metainfo.xml 69f13e9
>
> Diff: https://reviews.apache.org/r/32414/diff/
>
>
> Testing
> -------
>
> Manual testing to verify that topology script is configured and available, services (HDFS, MAPRED2) are marked for restart when a change is made to host rack information, and topology script can be invoked to get updated rack information.
>
> New unit tests added.
>
> All unit tests pass.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 33:36 min
> [INFO] Finished at: 2015-03-23T15:47:10-04:00
> [INFO] Final Memory: 43M/600M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Tom Beerbower
>
>