You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Robert Nettleton <rn...@hortonworks.com> on 2014/10/27 22:58:42 UTC
Review Request 27250: Proper handling exporting of 0.0.0.0 addresses
with Blueprints
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27250/
-----------------------------------------------------------
Review request for Ambari, John Speidel and Nate Cole.
Repository: ambari
Description
-------
This patch implements a fix for AMBARI-7988.
The Blueprints config processor does not currently allow service
config properties with the IP address of "0.0.0.0" to be
included in an exported Blueprint.
The following properties:
yarn.timeline-service.address,
yarn.timeline-service.webapp.address,
yarn.timeline-service.webapp.https.address
are not included in an exported Blueprint after a successful
cluster deployment the Yarn Application Timeline Service
enabled.
The problem occurs because the default values for these
properties is typically "0.0.0.0:portNumber". This is
a special IP address that the current Blueprints processor
does no recognize as a valid service address. Currently,
the processor will remove any properties that require
host topology information, but don't have any valid hostname
included in the property.
This patch resolves the problem by adding some handling
code to the Blueprint export processing. If a property, such
as the ones listed above, uses the default "0.0.0.0" address, then
this property is now allowed in the exported blueprint, as this
is a valid server address, but is not directly tied to the specific
server topology of a given cluster. If the user overrides these
properties, then the hostname substitution process/export process
will work as before.
This patch also implements a new unit test to verify this change.
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java 4183b9d
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java 04c75a1
Diff: https://reviews.apache.org/r/27250/diff/
Testing
-------
1. Ran the ambari-server unit tests with this patch applied (both trunk and 1.7.0), and all unit tests are passing with this change.
2. Manually verified the fix on 1.7.0.
3. Manually verified the fix on trunk.
Thanks,
Robert Nettleton
Re: Review Request 27250: Proper handling exporting of 0.0.0.0
addresses with Blueprints
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27250/#review58706
-----------------------------------------------------------
Ship it!
Ship It!
- Nate Cole
On Oct. 27, 2014, 5:58 p.m., Robert Nettleton wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27250/
> -----------------------------------------------------------
>
> (Updated Oct. 27, 2014, 5:58 p.m.)
>
>
> Review request for Ambari, John Speidel and Nate Cole.
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This patch implements a fix for AMBARI-7988.
>
> The Blueprints config processor does not currently allow service
> config properties with the IP address of "0.0.0.0" to be
> included in an exported Blueprint.
>
> The following properties:
>
> yarn.timeline-service.address,
> yarn.timeline-service.webapp.address,
> yarn.timeline-service.webapp.https.address
>
> are not included in an exported Blueprint after a successful
> cluster deployment the Yarn Application Timeline Service
> enabled.
>
> The problem occurs because the default values for these
> properties is typically "0.0.0.0:portNumber". This is
> a special IP address that the current Blueprints processor
> does no recognize as a valid service address. Currently,
> the processor will remove any properties that require
> host topology information, but don't have any valid hostname
> included in the property.
>
> This patch resolves the problem by adding some handling
> code to the Blueprint export processing. If a property, such
> as the ones listed above, uses the default "0.0.0.0" address, then
> this property is now allowed in the exported blueprint, as this
> is a valid server address, but is not directly tied to the specific
> server topology of a given cluster. If the user overrides these
> properties, then the hostname substitution process/export process
> will work as before.
>
> This patch also implements a new unit test to verify this change.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java 4183b9d
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java 04c75a1
>
> Diff: https://reviews.apache.org/r/27250/diff/
>
>
> Testing
> -------
>
> 1. Ran the ambari-server unit tests with this patch applied (both trunk and 1.7.0), and all unit tests are passing with this change.
> 2. Manually verified the fix on 1.7.0.
> 3. Manually verified the fix on trunk.
>
>
> Thanks,
>
> Robert Nettleton
>
>
Re: Review Request 27250: Proper handling exporting of 0.0.0.0
addresses with Blueprints
Posted by John Speidel <js...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27250/#review58702
-----------------------------------------------------------
Ship it!
Ship It!
- John Speidel
On Oct. 27, 2014, 9:58 p.m., Robert Nettleton wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27250/
> -----------------------------------------------------------
>
> (Updated Oct. 27, 2014, 9:58 p.m.)
>
>
> Review request for Ambari, John Speidel and Nate Cole.
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This patch implements a fix for AMBARI-7988.
>
> The Blueprints config processor does not currently allow service
> config properties with the IP address of "0.0.0.0" to be
> included in an exported Blueprint.
>
> The following properties:
>
> yarn.timeline-service.address,
> yarn.timeline-service.webapp.address,
> yarn.timeline-service.webapp.https.address
>
> are not included in an exported Blueprint after a successful
> cluster deployment the Yarn Application Timeline Service
> enabled.
>
> The problem occurs because the default values for these
> properties is typically "0.0.0.0:portNumber". This is
> a special IP address that the current Blueprints processor
> does no recognize as a valid service address. Currently,
> the processor will remove any properties that require
> host topology information, but don't have any valid hostname
> included in the property.
>
> This patch resolves the problem by adding some handling
> code to the Blueprint export processing. If a property, such
> as the ones listed above, uses the default "0.0.0.0" address, then
> this property is now allowed in the exported blueprint, as this
> is a valid server address, but is not directly tied to the specific
> server topology of a given cluster. If the user overrides these
> properties, then the hostname substitution process/export process
> will work as before.
>
> This patch also implements a new unit test to verify this change.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java 4183b9d
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java 04c75a1
>
> Diff: https://reviews.apache.org/r/27250/diff/
>
>
> Testing
> -------
>
> 1. Ran the ambari-server unit tests with this patch applied (both trunk and 1.7.0), and all unit tests are passing with this change.
> 2. Manually verified the fix on 1.7.0.
> 3. Manually verified the fix on trunk.
>
>
> Thanks,
>
> Robert Nettleton
>
>
Re: Review Request 27250: Proper handling exporting of 0.0.0.0
addresses with Blueprints
Posted by John Speidel <js...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27250/#review58800
-----------------------------------------------------------
Merged to 1.7.0 and trunk.
All unit tests pass.
Results :
Tests run: 2123, Failures: 0, Errors: 0, Skipped: 15
...
Total run:679
Total errors:0
Total failures:0
- John Speidel
On Oct. 27, 2014, 9:58 p.m., Robert Nettleton wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27250/
> -----------------------------------------------------------
>
> (Updated Oct. 27, 2014, 9:58 p.m.)
>
>
> Review request for Ambari, John Speidel and Nate Cole.
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This patch implements a fix for AMBARI-7988.
>
> The Blueprints config processor does not currently allow service
> config properties with the IP address of "0.0.0.0" to be
> included in an exported Blueprint.
>
> The following properties:
>
> yarn.timeline-service.address,
> yarn.timeline-service.webapp.address,
> yarn.timeline-service.webapp.https.address
>
> are not included in an exported Blueprint after a successful
> cluster deployment the Yarn Application Timeline Service
> enabled.
>
> The problem occurs because the default values for these
> properties is typically "0.0.0.0:portNumber". This is
> a special IP address that the current Blueprints processor
> does no recognize as a valid service address. Currently,
> the processor will remove any properties that require
> host topology information, but don't have any valid hostname
> included in the property.
>
> This patch resolves the problem by adding some handling
> code to the Blueprint export processing. If a property, such
> as the ones listed above, uses the default "0.0.0.0" address, then
> this property is now allowed in the exported blueprint, as this
> is a valid server address, but is not directly tied to the specific
> server topology of a given cluster. If the user overrides these
> properties, then the hostname substitution process/export process
> will work as before.
>
> This patch also implements a new unit test to verify this change.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java 4183b9d
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java 04c75a1
>
> Diff: https://reviews.apache.org/r/27250/diff/
>
>
> Testing
> -------
>
> 1. Ran the ambari-server unit tests with this patch applied (both trunk and 1.7.0), and all unit tests are passing with this change.
> 2. Manually verified the fix on 1.7.0.
> 3. Manually verified the fix on trunk.
>
>
> Thanks,
>
> Robert Nettleton
>
>
Re: Review Request 27250: Proper handling exporting of 0.0.0.0
addresses with Blueprints
Posted by John Speidel <js...@hortonworks.com>.
> On Oct. 27, 2014, 10:58 p.m., Michael Harp wrote:
> > There are several other properties that we typically set to 0.0.0.0, should they be added?
> > "hbase.master.ipc.address": "0.0.0.0",
> > "hbase.regionserver.ipc.address": "0.0.0.0"
> > "dfs.namenode.rpc-bind-host": "0.0.0.0",
> > "dfs.namenode.http-bind-host": "0.0.0.0",
> > "dfs.namenode.servicerpc-bind-host": "0.0.0.0",
> > "dfs.namenode.secondary.http-address": "0.0.0.0:50090"
> > "yarn.resourcemanager.webapp.address": "0.0.0.0:8088",
> > "yarn.nodemanager.webapp.bind-host": "0.0.0.0",
> > "yarn.nodemanager.webapp.https.bind-host": "0.0.0.0",
> > "yarn.resourcemanager.admin.bind-host": "0.0.0.0",
> > "yarn.resourcemanager.bind-host": "0.0.0.0",
> > "yarn.resourcemanager.resource-tracker.bind-host": "0.0.0.0",
> > "yarn.resourcemanager.scheduler.bind-host": "0.0.0.0",
> > "yarn.timeline-service.webapp.bind-host": "0.0.0.0",
> > "yarn.timeline-service.webapp.https.bind-host": "0.0.0.0",
> > "yarn.timeline-service.bind-host": "0.0.0.0",
> > "yarn.timeline-service.address": "0.0.0.0:10200"
> > "mapreduce.jobhistory.admin.bind-host": "0.0.0.0",
> > "mapreduce.jobhistory.webapp.bind-host": "0.0.0.0",
> > "mapreduce.jobhistory.webapp.https.bind-host": "0.0.0.0",
> > "mapreduce.jobhistory.bind-host": "0.0.0.0"
Michael,
For this issue, some service properties were not being exported to blueprint because they were known to contain topology related information (hostname/ip) and there was no host that matched the host/ip in the cluster. Because of this, the blueprint processor removed these properties from the exported set as it thought that they were pointing to external non-managed hosts. This patch simply ensures that any of these topology related properties which have a value of "0.0.0.0" will be exported to blueprint.
>From the set of properties that you have included above, only a few are included in any Ambari HDP stack. For these properties that are included in an Ambari HDP stack, this patch will ensure that they are exported to blueprint. For the others in the set, the vast majority, they would not have been affected by this issue since they are not "known" to be topology related properties so they wouldn't have be checked to see if they pointed to a managed host.
On a related matter, because of limitations in the currently available stack definition syntax, the blueprint processor has explicit knowledge of which properties are topology related. These properties are automatically updated to reflect the concrete cluster topology during cluster provisioning via a blueprint. It is only these properties which are affected by this issue. The special knowledge baked into the blueprint processor for these properites will be moved to the stack definitions where it belongs in a future release. This will make it a lot easier to add custom stack definitions.
- John
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27250/#review58711
-----------------------------------------------------------
On Oct. 27, 2014, 9:58 p.m., Robert Nettleton wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27250/
> -----------------------------------------------------------
>
> (Updated Oct. 27, 2014, 9:58 p.m.)
>
>
> Review request for Ambari, John Speidel and Nate Cole.
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This patch implements a fix for AMBARI-7988.
>
> The Blueprints config processor does not currently allow service
> config properties with the IP address of "0.0.0.0" to be
> included in an exported Blueprint.
>
> The following properties:
>
> yarn.timeline-service.address,
> yarn.timeline-service.webapp.address,
> yarn.timeline-service.webapp.https.address
>
> are not included in an exported Blueprint after a successful
> cluster deployment the Yarn Application Timeline Service
> enabled.
>
> The problem occurs because the default values for these
> properties is typically "0.0.0.0:portNumber". This is
> a special IP address that the current Blueprints processor
> does no recognize as a valid service address. Currently,
> the processor will remove any properties that require
> host topology information, but don't have any valid hostname
> included in the property.
>
> This patch resolves the problem by adding some handling
> code to the Blueprint export processing. If a property, such
> as the ones listed above, uses the default "0.0.0.0" address, then
> this property is now allowed in the exported blueprint, as this
> is a valid server address, but is not directly tied to the specific
> server topology of a given cluster. If the user overrides these
> properties, then the hostname substitution process/export process
> will work as before.
>
> This patch also implements a new unit test to verify this change.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java 4183b9d
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java 04c75a1
>
> Diff: https://reviews.apache.org/r/27250/diff/
>
>
> Testing
> -------
>
> 1. Ran the ambari-server unit tests with this patch applied (both trunk and 1.7.0), and all unit tests are passing with this change.
> 2. Manually verified the fix on 1.7.0.
> 3. Manually verified the fix on trunk.
>
>
> Thanks,
>
> Robert Nettleton
>
>
Re: Review Request 27250: Proper handling exporting of 0.0.0.0
addresses with Blueprints
Posted by Michael Harp <mi...@teradata.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27250/#review58711
-----------------------------------------------------------
There are several other properties that we typically set to 0.0.0.0, should they be added?
"hbase.master.ipc.address": "0.0.0.0",
"hbase.regionserver.ipc.address": "0.0.0.0"
"dfs.namenode.rpc-bind-host": "0.0.0.0",
"dfs.namenode.http-bind-host": "0.0.0.0",
"dfs.namenode.servicerpc-bind-host": "0.0.0.0",
"dfs.namenode.secondary.http-address": "0.0.0.0:50090"
"yarn.resourcemanager.webapp.address": "0.0.0.0:8088",
"yarn.nodemanager.webapp.bind-host": "0.0.0.0",
"yarn.nodemanager.webapp.https.bind-host": "0.0.0.0",
"yarn.resourcemanager.admin.bind-host": "0.0.0.0",
"yarn.resourcemanager.bind-host": "0.0.0.0",
"yarn.resourcemanager.resource-tracker.bind-host": "0.0.0.0",
"yarn.resourcemanager.scheduler.bind-host": "0.0.0.0",
"yarn.timeline-service.webapp.bind-host": "0.0.0.0",
"yarn.timeline-service.webapp.https.bind-host": "0.0.0.0",
"yarn.timeline-service.bind-host": "0.0.0.0",
"yarn.timeline-service.address": "0.0.0.0:10200"
"mapreduce.jobhistory.admin.bind-host": "0.0.0.0",
"mapreduce.jobhistory.webapp.bind-host": "0.0.0.0",
"mapreduce.jobhistory.webapp.https.bind-host": "0.0.0.0",
"mapreduce.jobhistory.bind-host": "0.0.0.0"
- Michael Harp
On Oct. 27, 2014, 9:58 p.m., Robert Nettleton wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27250/
> -----------------------------------------------------------
>
> (Updated Oct. 27, 2014, 9:58 p.m.)
>
>
> Review request for Ambari, John Speidel and Nate Cole.
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This patch implements a fix for AMBARI-7988.
>
> The Blueprints config processor does not currently allow service
> config properties with the IP address of "0.0.0.0" to be
> included in an exported Blueprint.
>
> The following properties:
>
> yarn.timeline-service.address,
> yarn.timeline-service.webapp.address,
> yarn.timeline-service.webapp.https.address
>
> are not included in an exported Blueprint after a successful
> cluster deployment the Yarn Application Timeline Service
> enabled.
>
> The problem occurs because the default values for these
> properties is typically "0.0.0.0:portNumber". This is
> a special IP address that the current Blueprints processor
> does no recognize as a valid service address. Currently,
> the processor will remove any properties that require
> host topology information, but don't have any valid hostname
> included in the property.
>
> This patch resolves the problem by adding some handling
> code to the Blueprint export processing. If a property, such
> as the ones listed above, uses the default "0.0.0.0" address, then
> this property is now allowed in the exported blueprint, as this
> is a valid server address, but is not directly tied to the specific
> server topology of a given cluster. If the user overrides these
> properties, then the hostname substitution process/export process
> will work as before.
>
> This patch also implements a new unit test to verify this change.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java 4183b9d
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java 04c75a1
>
> Diff: https://reviews.apache.org/r/27250/diff/
>
>
> Testing
> -------
>
> 1. Ran the ambari-server unit tests with this patch applied (both trunk and 1.7.0), and all unit tests are passing with this change.
> 2. Manually verified the fix on 1.7.0.
> 3. Manually verified the fix on trunk.
>
>
> Thanks,
>
> Robert Nettleton
>
>