You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by John Speidel <js...@hortonworks.com> on 2015/05/18 22:34:34 UTC
Review Request 34370: Cluster blueprint export specifies wrong host
groups for components
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34370/
-----------------------------------------------------------
Review request for Ambari and Robert Nettleton.
Bugs: AMBARI-11226
https://issues.apache.org/jira/browse/AMBARI-11226
Repository: ambari
Description
-------
Steps to reproduce:
1. Setup a 3-node cluster (using vagrant or other option)
2. Using the UI, startup a cluster that includes: HDFS, Yarn/MR, Tez, Zookeeper, Ambari Metrics. There is no need to enable HA to reproduce this problem.
3. When this cluster deploys successfully, export a Blueprint from this cluster using the REST API:
api/v1/clusters/clusterone?format=blueprint
The resulting Blueprint will include many invalid mappings. I discovered this problem yesterday while attempting to run a "round-trip" test (start cluster, export Blueprint, re-create cluster with exported Blueprint). I noticed that many services, including the NameNode and any services that depend upon it, were not starting up properly.
I will attach the Blueprint I'm seeing when I export from a running 3-node cluster with the components listed above.
The relationship between the components in various host groups seems to no longer match up properly with the exported properties.
Here are a few examples I've found:
1. In the attached Blueprint, the "NAMENODE" component is included in "host_group_2". The problem is that the exported configuration refers to the NameNode being in a separate host group:
in "core-site": "fs.defaultFS" : "hdfs://%HOSTGROUP::host_group_1%:8020",
in "hdfs-site: "dfs.namenode.http-address" : "%HOSTGROUP::host_group_1%:50070",
"dfs.namenode.https-address" : "%HOSTGROUP::host_group_1%:50470",
It's important to note that the properties above should refer to the host group that contains the NameNode, but they actually refer to a host group "host_group_1" that does not include a "NAMENODE".
2. The "SECONDARY_NAMENODE" component is included in "host_group_3", but the config refers to a different host group:
in "hdfs-site" : "dfs.namenode.secondary.http-address" : "%HOSTGROUP::host_group_2%:50090",
Again, the problem is the same: the properties refer to services on a given host group, but the services are not there.
There are probably more examples of this in the Blueprint, but these are the most prominent.
This causes any attempt to deploy with this Blueprint to fail, since the NameNode cannot start with invalid configuration. Subsequently, any services that are clients of the NameNode fail,since they attempt to connect to a service that is not deployed on the configured port.
This will cause any attempt to "round-trip" deploy an exported Blueprint to fail.
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ExportBlueprintRequest.java 9318db9
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ExportBlueprintRequestTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/34370/diff/
Testing
-------
Functional Testing:
- Exported a blueprint from a cluster created via UI and blueprint and confirmed correct component -> hostgroup mapping
Unit Testing:
- Created new unit test for this issue
- Currently running entire unit test suite and will update review with unit test output summary
Thanks,
John Speidel
Re: Review Request 34370: Cluster blueprint export specifies wrong
host groups for components
Posted by John Speidel <js...@hortonworks.com>.
> On May 18, 2015, 8:46 p.m., Robert Nettleton wrote:
> > Ship It!
Results :
Tests run: 2991, Failures: 0, Errors: 0, Skipped: 75
...
Total run:741
Total errors:0
Total failures:0
- John
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34370/#review84214
-----------------------------------------------------------
On May 18, 2015, 8:34 p.m., John Speidel wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34370/
> -----------------------------------------------------------
>
> (Updated May 18, 2015, 8:34 p.m.)
>
>
> Review request for Ambari and Robert Nettleton.
>
>
> Bugs: AMBARI-11226
> https://issues.apache.org/jira/browse/AMBARI-11226
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Steps to reproduce:
> 1. Setup a 3-node cluster (using vagrant or other option)
> 2. Using the UI, startup a cluster that includes: HDFS, Yarn/MR, Tez, Zookeeper, Ambari Metrics. There is no need to enable HA to reproduce this problem.
> 3. When this cluster deploys successfully, export a Blueprint from this cluster using the REST API:
> api/v1/clusters/clusterone?format=blueprint
> The resulting Blueprint will include many invalid mappings. I discovered this problem yesterday while attempting to run a "round-trip" test (start cluster, export Blueprint, re-create cluster with exported Blueprint). I noticed that many services, including the NameNode and any services that depend upon it, were not starting up properly.
> I will attach the Blueprint I'm seeing when I export from a running 3-node cluster with the components listed above.
> The relationship between the components in various host groups seems to no longer match up properly with the exported properties.
> Here are a few examples I've found:
> 1. In the attached Blueprint, the "NAMENODE" component is included in "host_group_2". The problem is that the exported configuration refers to the NameNode being in a separate host group:
> in "core-site": "fs.defaultFS" : "hdfs://%HOSTGROUP::host_group_1%:8020",
> in "hdfs-site: "dfs.namenode.http-address" : "%HOSTGROUP::host_group_1%:50070",
> "dfs.namenode.https-address" : "%HOSTGROUP::host_group_1%:50470",
> It's important to note that the properties above should refer to the host group that contains the NameNode, but they actually refer to a host group "host_group_1" that does not include a "NAMENODE".
> 2. The "SECONDARY_NAMENODE" component is included in "host_group_3", but the config refers to a different host group:
> in "hdfs-site" : "dfs.namenode.secondary.http-address" : "%HOSTGROUP::host_group_2%:50090",
> Again, the problem is the same: the properties refer to services on a given host group, but the services are not there.
> There are probably more examples of this in the Blueprint, but these are the most prominent.
> This causes any attempt to deploy with this Blueprint to fail, since the NameNode cannot start with invalid configuration. Subsequently, any services that are clients of the NameNode fail,since they attempt to connect to a service that is not deployed on the configured port.
> This will cause any attempt to "round-trip" deploy an exported Blueprint to fail.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ExportBlueprintRequest.java 9318db9
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ExportBlueprintRequestTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/34370/diff/
>
>
> Testing
> -------
>
> Functional Testing:
> - Exported a blueprint from a cluster created via UI and blueprint and confirmed correct component -> hostgroup mapping
>
> Unit Testing:
> - Created new unit test for this issue
> - Currently running entire unit test suite and will update review with unit test output summary
>
>
> Thanks,
>
> John Speidel
>
>
Re: Review Request 34370: Cluster blueprint export specifies wrong
host groups for components
Posted by Robert Nettleton <rn...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34370/#review84214
-----------------------------------------------------------
Ship it!
Ship It!
- Robert Nettleton
On May 18, 2015, 8:34 p.m., John Speidel wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34370/
> -----------------------------------------------------------
>
> (Updated May 18, 2015, 8:34 p.m.)
>
>
> Review request for Ambari and Robert Nettleton.
>
>
> Bugs: AMBARI-11226
> https://issues.apache.org/jira/browse/AMBARI-11226
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Steps to reproduce:
> 1. Setup a 3-node cluster (using vagrant or other option)
> 2. Using the UI, startup a cluster that includes: HDFS, Yarn/MR, Tez, Zookeeper, Ambari Metrics. There is no need to enable HA to reproduce this problem.
> 3. When this cluster deploys successfully, export a Blueprint from this cluster using the REST API:
> api/v1/clusters/clusterone?format=blueprint
> The resulting Blueprint will include many invalid mappings. I discovered this problem yesterday while attempting to run a "round-trip" test (start cluster, export Blueprint, re-create cluster with exported Blueprint). I noticed that many services, including the NameNode and any services that depend upon it, were not starting up properly.
> I will attach the Blueprint I'm seeing when I export from a running 3-node cluster with the components listed above.
> The relationship between the components in various host groups seems to no longer match up properly with the exported properties.
> Here are a few examples I've found:
> 1. In the attached Blueprint, the "NAMENODE" component is included in "host_group_2". The problem is that the exported configuration refers to the NameNode being in a separate host group:
> in "core-site": "fs.defaultFS" : "hdfs://%HOSTGROUP::host_group_1%:8020",
> in "hdfs-site: "dfs.namenode.http-address" : "%HOSTGROUP::host_group_1%:50070",
> "dfs.namenode.https-address" : "%HOSTGROUP::host_group_1%:50470",
> It's important to note that the properties above should refer to the host group that contains the NameNode, but they actually refer to a host group "host_group_1" that does not include a "NAMENODE".
> 2. The "SECONDARY_NAMENODE" component is included in "host_group_3", but the config refers to a different host group:
> in "hdfs-site" : "dfs.namenode.secondary.http-address" : "%HOSTGROUP::host_group_2%:50090",
> Again, the problem is the same: the properties refer to services on a given host group, but the services are not there.
> There are probably more examples of this in the Blueprint, but these are the most prominent.
> This causes any attempt to deploy with this Blueprint to fail, since the NameNode cannot start with invalid configuration. Subsequently, any services that are clients of the NameNode fail,since they attempt to connect to a service that is not deployed on the configured port.
> This will cause any attempt to "round-trip" deploy an exported Blueprint to fail.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ExportBlueprintRequest.java 9318db9
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ExportBlueprintRequestTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/34370/diff/
>
>
> Testing
> -------
>
> Functional Testing:
> - Exported a blueprint from a cluster created via UI and blueprint and confirmed correct component -> hostgroup mapping
>
> Unit Testing:
> - Created new unit test for this issue
> - Currently running entire unit test suite and will update review with unit test output summary
>
>
> Thanks,
>
> John Speidel
>
>