You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Robert Nettleton <rn...@hortonworks.com> on 2015/06/18 16:49:06 UTC
Review Request 35607: Filter out Kerberos hostname properties from
exported Blueprint
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35607/
-----------------------------------------------------------
Review request for Ambari, John Speidel, Mahadev Konar, and Robert Levas.
Bugs: AMBARI-12000
https://issues.apache.org/jira/browse/AMBARI-12000
Repository: ambari
Description
-------
This patch resolves AMBARI-12000.
An export of a Blueprint from a running, Kerberized cluster would include the following properties with hostnames:
"admin_server_host" in "kerberos-env"
"kdc_host" in "kerberos-env"
"hadoop.proxyuser.yarn.hosts" in "core-site"
Exported Blueprints should not contain hostnames in general, since this makes the Blueprint less portable over different cluster configurations.
This patch implements the following changes:
1. Adds a filter to remove the "admin_server_host" and "kdc_host" properties from an exported Blueprint. These hostname properties refer to the KDC server of a Kerberized cluster, and so are not useful to an exported Blueprint at this time. Since these properties generally refer to an external KDC server, it is likely that a new cluster would point to a new KDC install, which is typically an external server.
2. Adds a property updaters to mask the hostname information in the "hadoop.proxyuser.yarn.hosts" property in "core-site". This property includes a comma-separated list of hostnames that include Yarn ResourceManager instances. While not immediately useful outside of Kerberos, having this property include the %HOSTGROUP% syntax for hostname substitution is in keeping with the handling of other "*proxyuser*" properties, and could eventually be helpful in creating Kerberized clusters in future versions of Ambari.
3. Adds a unit test to verify the changes to the handling for these three properties.
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java af3b331
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java abf5675
Diff: https://reviews.apache.org/r/35607/diff/
Testing
-------
1. Ran the ambari-server unit test suite. Results:
Java Tests:
"Results :
Tests run: 3088, Failures: 0, Errors: 0, Skipped: 28"
Python Tests:
"Failed tests:
FAIL: test_recommendTezConfigurations (test_stack_advisor.TestHDP23StackAdvisor)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/rnettleton/dev/workspaces/ambari/ambari-common/src/test/python/mock/mock.py", line 1199, in patched
return func(*args, **keywargs)
File "/Users/rnettleton/dev/workspaces/ambari/ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py", line 661, in test_recommendTezConfigurations
self.assertEquals(configurations, expected)
AssertionError: {'capacity-scheduler': {'properties': {'yarn.scheduler.capacity.root.queues': 'q [truncated]... != {'capacity-scheduler': {'properties': {'yarn.scheduler.capacity.root.queues': 'q [truncated]...
{'capacity-scheduler': {'properties': {'yarn.scheduler.capacity.root.queues': 'queue1,queue2'}},
'tez-site': {'properties': {'tez.am.launch.cmd-opts': '-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC',
'tez.am.resource.memory.mb': '4000',
'tez.runtime.io.sort.mb': '307',
'tez.runtime.unordered.output.buffer.size-mb': '57',
'tez.session.am.dag.submit.timeout.secs': '600',
'tez.task.launch.cmd-opts': '-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC',
'tez.task.resource.memory.mb': '768',
- 'tez.tez-ui.history-url.base': 'http://HW10578.local:8080/#/main/views/TEZ/0.7.0.2.3.0.0-2155/TEZ_CLUSTER_INSTANCE'}},
? ^^ ^ -- ^ ^^^
+ 'tez.tez-ui.history-url.base': 'http://hw11465.home:8080/#/main/views/TEZ/0.7.0.2.3.0.0-2155/TEZ_CLUSTER_INSTANCE'}},
? ^^ ^^^ ^ ^^
'yarn-site': {'properties': {'yarn.scheduler.maximum-allocation-mb': '8192',
'yarn.scheduler.minimum-allocation-mb': '256'}}}
----------------------------------------------------------------------
Total run:762
Total errors:0
Total failures:1"
The failure seen here also occurs in trunk without my patch applied, and so is an un-related failure.
2. Deployed a single-node HDFS/Yarn/Zookeeper/Metrics cluster with the Ambari UI, with my patch applied. Exported a Blueprint from this running cluster, and verified that these properties are no longer include in the exported Blueprint, or in the case of the proxyuser property are properly masked with %HOSTGROUP% tokens. Also verified that no hostname information is included in the exported Blueprint for this cluster.
Thanks,
Robert Nettleton
Re: Review Request 35607: Filter out Kerberos hostname properties from
exported Blueprint
Posted by Robert Levas <rl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35607/#review88380
-----------------------------------------------------------
Ship it!
Ship It!
- Robert Levas
On June 18, 2015, 10:49 a.m., Robert Nettleton wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35607/
> -----------------------------------------------------------
>
> (Updated June 18, 2015, 10:49 a.m.)
>
>
> Review request for Ambari, John Speidel, Mahadev Konar, and Robert Levas.
>
>
> Bugs: AMBARI-12000
> https://issues.apache.org/jira/browse/AMBARI-12000
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This patch resolves AMBARI-12000.
>
> An export of a Blueprint from a running, Kerberized cluster would include the following properties with hostnames:
>
> "admin_server_host" in "kerberos-env"
> "kdc_host" in "kerberos-env"
> "hadoop.proxyuser.yarn.hosts" in "core-site"
>
> Exported Blueprints should not contain hostnames in general, since this makes the Blueprint less portable over different cluster configurations.
>
> This patch implements the following changes:
>
> 1. Adds a filter to remove the "admin_server_host" and "kdc_host" properties from an exported Blueprint. These hostname properties refer to the KDC server of a Kerberized cluster, and so are not useful to an exported Blueprint at this time. Since these properties generally refer to an external KDC server, it is likely that a new cluster would point to a new KDC install, which is typically an external server.
> 2. Adds a property updaters to mask the hostname information in the "hadoop.proxyuser.yarn.hosts" property in "core-site". This property includes a comma-separated list of hostnames that include Yarn ResourceManager instances. While not immediately useful outside of Kerberos, having this property include the %HOSTGROUP% syntax for hostname substitution is in keeping with the handling of other "*proxyuser*" properties, and could eventually be helpful in creating Kerberized clusters in future versions of Ambari.
> 3. Adds a unit test to verify the changes to the handling for these three properties.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java af3b331
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java abf5675
>
> Diff: https://reviews.apache.org/r/35607/diff/
>
>
> Testing
> -------
>
> 1. Ran the ambari-server unit test suite. Results:
>
> Java Tests:
>
> "Results :
>
> Tests run: 3088, Failures: 0, Errors: 0, Skipped: 28"
>
> Python Tests:
>
> "Failed tests:
> FAIL: test_recommendTezConfigurations (test_stack_advisor.TestHDP23StackAdvisor)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/Users/rnettleton/dev/workspaces/ambari/ambari-common/src/test/python/mock/mock.py", line 1199, in patched
> return func(*args, **keywargs)
> File "/Users/rnettleton/dev/workspaces/ambari/ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py", line 661, in test_recommendTezConfigurations
> self.assertEquals(configurations, expected)
> AssertionError: {'capacity-scheduler': {'properties': {'yarn.scheduler.capacity.root.queues': 'q [truncated]... != {'capacity-scheduler': {'properties': {'yarn.scheduler.capacity.root.queues': 'q [truncated]...
> {'capacity-scheduler': {'properties': {'yarn.scheduler.capacity.root.queues': 'queue1,queue2'}},
> 'tez-site': {'properties': {'tez.am.launch.cmd-opts': '-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC',
> 'tez.am.resource.memory.mb': '4000',
> 'tez.runtime.io.sort.mb': '307',
> 'tez.runtime.unordered.output.buffer.size-mb': '57',
> 'tez.session.am.dag.submit.timeout.secs': '600',
> 'tez.task.launch.cmd-opts': '-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC',
> 'tez.task.resource.memory.mb': '768',
> - 'tez.tez-ui.history-url.base': 'http://HW10578.local:8080/#/main/views/TEZ/0.7.0.2.3.0.0-2155/TEZ_CLUSTER_INSTANCE'}},
> ? ^^ ^ -- ^ ^^^
>
> + 'tez.tez-ui.history-url.base': 'http://hw11465.home:8080/#/main/views/TEZ/0.7.0.2.3.0.0-2155/TEZ_CLUSTER_INSTANCE'}},
> ? ^^ ^^^ ^ ^^
>
> 'yarn-site': {'properties': {'yarn.scheduler.maximum-allocation-mb': '8192',
> 'yarn.scheduler.minimum-allocation-mb': '256'}}}
>
> ----------------------------------------------------------------------
> Total run:762
> Total errors:0
> Total failures:1"
>
>
> The failure seen here also occurs in trunk without my patch applied, and so is an un-related failure.
>
> 2. Deployed a single-node HDFS/Yarn/Zookeeper/Metrics cluster with the Ambari UI, with my patch applied. Exported a Blueprint from this running cluster, and verified that these properties are no longer include in the exported Blueprint, or in the case of the proxyuser property are properly masked with %HOSTGROUP% tokens. Also verified that no hostname information is included in the exported Blueprint for this cluster.
>
>
> Thanks,
>
> Robert Nettleton
>
>
Re: Review Request 35607: Filter out Kerberos hostname properties from
exported Blueprint
Posted by John Speidel <js...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35607/#review88379
-----------------------------------------------------------
Ship it!
Ship It!
- John Speidel
On June 18, 2015, 2:49 p.m., Robert Nettleton wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35607/
> -----------------------------------------------------------
>
> (Updated June 18, 2015, 2:49 p.m.)
>
>
> Review request for Ambari, John Speidel, Mahadev Konar, and Robert Levas.
>
>
> Bugs: AMBARI-12000
> https://issues.apache.org/jira/browse/AMBARI-12000
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This patch resolves AMBARI-12000.
>
> An export of a Blueprint from a running, Kerberized cluster would include the following properties with hostnames:
>
> "admin_server_host" in "kerberos-env"
> "kdc_host" in "kerberos-env"
> "hadoop.proxyuser.yarn.hosts" in "core-site"
>
> Exported Blueprints should not contain hostnames in general, since this makes the Blueprint less portable over different cluster configurations.
>
> This patch implements the following changes:
>
> 1. Adds a filter to remove the "admin_server_host" and "kdc_host" properties from an exported Blueprint. These hostname properties refer to the KDC server of a Kerberized cluster, and so are not useful to an exported Blueprint at this time. Since these properties generally refer to an external KDC server, it is likely that a new cluster would point to a new KDC install, which is typically an external server.
> 2. Adds a property updaters to mask the hostname information in the "hadoop.proxyuser.yarn.hosts" property in "core-site". This property includes a comma-separated list of hostnames that include Yarn ResourceManager instances. While not immediately useful outside of Kerberos, having this property include the %HOSTGROUP% syntax for hostname substitution is in keeping with the handling of other "*proxyuser*" properties, and could eventually be helpful in creating Kerberized clusters in future versions of Ambari.
> 3. Adds a unit test to verify the changes to the handling for these three properties.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java af3b331
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java abf5675
>
> Diff: https://reviews.apache.org/r/35607/diff/
>
>
> Testing
> -------
>
> 1. Ran the ambari-server unit test suite. Results:
>
> Java Tests:
>
> "Results :
>
> Tests run: 3088, Failures: 0, Errors: 0, Skipped: 28"
>
> Python Tests:
>
> "Failed tests:
> FAIL: test_recommendTezConfigurations (test_stack_advisor.TestHDP23StackAdvisor)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/Users/rnettleton/dev/workspaces/ambari/ambari-common/src/test/python/mock/mock.py", line 1199, in patched
> return func(*args, **keywargs)
> File "/Users/rnettleton/dev/workspaces/ambari/ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py", line 661, in test_recommendTezConfigurations
> self.assertEquals(configurations, expected)
> AssertionError: {'capacity-scheduler': {'properties': {'yarn.scheduler.capacity.root.queues': 'q [truncated]... != {'capacity-scheduler': {'properties': {'yarn.scheduler.capacity.root.queues': 'q [truncated]...
> {'capacity-scheduler': {'properties': {'yarn.scheduler.capacity.root.queues': 'queue1,queue2'}},
> 'tez-site': {'properties': {'tez.am.launch.cmd-opts': '-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC',
> 'tez.am.resource.memory.mb': '4000',
> 'tez.runtime.io.sort.mb': '307',
> 'tez.runtime.unordered.output.buffer.size-mb': '57',
> 'tez.session.am.dag.submit.timeout.secs': '600',
> 'tez.task.launch.cmd-opts': '-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC',
> 'tez.task.resource.memory.mb': '768',
> - 'tez.tez-ui.history-url.base': 'http://HW10578.local:8080/#/main/views/TEZ/0.7.0.2.3.0.0-2155/TEZ_CLUSTER_INSTANCE'}},
> ? ^^ ^ -- ^ ^^^
>
> + 'tez.tez-ui.history-url.base': 'http://hw11465.home:8080/#/main/views/TEZ/0.7.0.2.3.0.0-2155/TEZ_CLUSTER_INSTANCE'}},
> ? ^^ ^^^ ^ ^^
>
> 'yarn-site': {'properties': {'yarn.scheduler.maximum-allocation-mb': '8192',
> 'yarn.scheduler.minimum-allocation-mb': '256'}}}
>
> ----------------------------------------------------------------------
> Total run:762
> Total errors:0
> Total failures:1"
>
>
> The failure seen here also occurs in trunk without my patch applied, and so is an un-related failure.
>
> 2. Deployed a single-node HDFS/Yarn/Zookeeper/Metrics cluster with the Ambari UI, with my patch applied. Exported a Blueprint from this running cluster, and verified that these properties are no longer include in the exported Blueprint, or in the case of the proxyuser property are properly masked with %HOSTGROUP% tokens. Also verified that no hostname information is included in the exported Blueprint for this cluster.
>
>
> Thanks,
>
> Robert Nettleton
>
>