You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Dmytro Sen <ds...@hortonworks.com> on 2016/04/18 15:07:22 UTC
Review Request 46327: Blueprint based installations should
automatically add dfs.internal.nameservices during cluster creation
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46327/
-----------------------------------------------------------
Review request for Ambari, John Speidel and Sumit Mohanty.
Bugs: AMBARI-15791
https://issues.apache.org/jira/browse/AMBARI-15791
Repository: ambari
Description
-------
Blueprint config processing engine should automatically populate dfs.internal.nameservices when it is deploying HA clusters. The work-around till then is to explicitly add the property when submitting the blueprint.
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java 1433a1b
ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterTopologyImpl.java 4f2f02d
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java 834953b
Diff: https://reviews.apache.org/r/46327/diff/
Testing
-------
Unit tests passed
Thanks,
Dmytro Sen
Re: Review Request 46327: Blueprint based installations should
automatically add dfs.internal.nameservices during cluster creation
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46327/#review129329
-----------------------------------------------------------
Ship it!
Ship It!
- Sumit Mohanty
On April 18, 2016, 1:07 p.m., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46327/
> -----------------------------------------------------------
>
> (Updated April 18, 2016, 1:07 p.m.)
>
>
> Review request for Ambari, John Speidel and Sumit Mohanty.
>
>
> Bugs: AMBARI-15791
> https://issues.apache.org/jira/browse/AMBARI-15791
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Blueprint config processing engine should automatically populate dfs.internal.nameservices when it is deploying HA clusters. The work-around till then is to explicitly add the property when submitting the blueprint.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java 1433a1b
> ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterTopologyImpl.java 4f2f02d
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java 834953b
>
> Diff: https://reviews.apache.org/r/46327/diff/
>
>
> Testing
> -------
>
> Unit tests passed
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 46327: Blueprint based installations should
automatically add dfs.internal.nameservices during cluster creation
Posted by Robert Nettleton <rn...@hortonworks.com>.
> On April 18, 2016, 1:39 p.m., Robert Nettleton wrote:
> > Could we get some more background context as to why this change is necessary? I've checked the associated Apache JIRA, and there isn't much information there.
> >
> > In particular, can the submitter please explain why this internal property needs to be added, especially considering that the HDFS documentation seems to indicate that the public and internal forms of this property should generally have the same value?
> >
> > Thanks.
>
> Dmytro Sen wrote:
> HDFS-6376 allows distcp to copy data between HA clusters. Users can use a new configuration property "dfs.internal.nameservices" to explicitly specify the name services belonging to the local cluster, while continue using the configuration property "dfs.nameservices" to specify all the name services in the local and remote clusters.
>
> All ambari users have single namesevice value in "dfs.nameservices" on existing clusters, that's the name service belonging to the local cluster. After upgrade to 2.4.0 or restoring ambari cluster from a blueprint created from any older version, we add "dfs.internal.nameservices" pointing to the local cluster and allow user to add to "dfs.nameservices" any external name services.
Ok, thanks for the clarification.
- Robert
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46327/#review129331
-----------------------------------------------------------
On April 18, 2016, 1:39 p.m., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46327/
> -----------------------------------------------------------
>
> (Updated April 18, 2016, 1:39 p.m.)
>
>
> Review request for Ambari, John Speidel and Sumit Mohanty.
>
>
> Bugs: AMBARI-15791
> https://issues.apache.org/jira/browse/AMBARI-15791
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Blueprint config processing engine should automatically populate dfs.internal.nameservices when it is deploying HA clusters. The work-around till then is to explicitly add the property when submitting the blueprint.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java 1433a1b
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java 834953b
>
> Diff: https://reviews.apache.org/r/46327/diff/
>
>
> Testing
> -------
>
> Unit tests passed
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 46327: Blueprint based installations should
automatically add dfs.internal.nameservices during cluster creation
Posted by Dmytro Sen <ds...@hortonworks.com>.
> On Апрель 18, 2016, 1:39 п.п., Robert Nettleton wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java, line 858
> > <https://reviews.apache.org/r/46327/diff/1/?file=1348454#file1348454line858>
> >
> > Why is this internal property being checked here?
> >
> > The update change above seems to indicate that these properties should always be set to the same values. If that is the case, then why is this method being modified?
We only add "dfs.internal.nameservices" if it's not specified in user's blueprint and always rely on "dfs.internal.nameservices" instead of "dfs.nameservices". "dfs.nameservices" might contain external name services, not managed by Ambari
if (clusterTopology.isNameNodeHAEnabled()) {
// add "dfs.internal.nameservices" if it's not specified
Map<String, String> hdfsSiteConfig = clusterConfig.getFullProperties().get("hdfs-site");
String nameservices = hdfsSiteConfig.get("dfs.nameservices");
String int_nameservices = hdfsSiteConfig.get("dfs.internal.nameservices");
if(int_nameservices == null && nameservices != null) {
clusterConfig.setProperty("hdfs-site", "dfs.internal.nameservices", nameservices);
}
- Dmytro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46327/#review129331
-----------------------------------------------------------
On Апрель 18, 2016, 1:39 п.п., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46327/
> -----------------------------------------------------------
>
> (Updated Апрель 18, 2016, 1:39 п.п.)
>
>
> Review request for Ambari, John Speidel and Sumit Mohanty.
>
>
> Bugs: AMBARI-15791
> https://issues.apache.org/jira/browse/AMBARI-15791
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Blueprint config processing engine should automatically populate dfs.internal.nameservices when it is deploying HA clusters. The work-around till then is to explicitly add the property when submitting the blueprint.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java 1433a1b
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java 834953b
>
> Diff: https://reviews.apache.org/r/46327/diff/
>
>
> Testing
> -------
>
> Unit tests passed
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 46327: Blueprint based installations should
automatically add dfs.internal.nameservices during cluster creation
Posted by Dmytro Sen <ds...@hortonworks.com>.
> On Апрель 18, 2016, 1:39 п.п., Robert Nettleton wrote:
> > Could we get some more background context as to why this change is necessary? I've checked the associated Apache JIRA, and there isn't much information there.
> >
> > In particular, can the submitter please explain why this internal property needs to be added, especially considering that the HDFS documentation seems to indicate that the public and internal forms of this property should generally have the same value?
> >
> > Thanks.
HDFS-6376 allows distcp to copy data between HA clusters. Users can use a new configuration property "dfs.internal.nameservices" to explicitly specify the name services belonging to the local cluster, while continue using the configuration property "dfs.nameservices" to specify all the name services in the local and remote clusters.
All ambari users have single namesevice value in "dfs.nameservices" on existing clusters, that's the name service belonging to the local cluster. After upgrade to 2.4.0 or restoring ambari cluster from a blueprint created from any older version, we add "dfs.internal.nameservices" pointing to the local cluster and allow user to add to "dfs.nameservices" any external name services.
- Dmytro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46327/#review129331
-----------------------------------------------------------
On Апрель 18, 2016, 1:39 п.п., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46327/
> -----------------------------------------------------------
>
> (Updated Апрель 18, 2016, 1:39 п.п.)
>
>
> Review request for Ambari, John Speidel and Sumit Mohanty.
>
>
> Bugs: AMBARI-15791
> https://issues.apache.org/jira/browse/AMBARI-15791
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Blueprint config processing engine should automatically populate dfs.internal.nameservices when it is deploying HA clusters. The work-around till then is to explicitly add the property when submitting the blueprint.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java 1433a1b
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java 834953b
>
> Diff: https://reviews.apache.org/r/46327/diff/
>
>
> Testing
> -------
>
> Unit tests passed
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 46327: Blueprint based installations should
automatically add dfs.internal.nameservices during cluster creation
Posted by Dmytro Sen <ds...@hortonworks.com>.
> On Апрель 18, 2016, 1:39 п.п., Robert Nettleton wrote:
> > ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java, line 4900
> > <https://reviews.apache.org/r/46327/diff/1/?file=1348456#file1348456line4900>
> >
> > I'm not certain that the tests using the public property should be modified. I'd recommend adding new unit tests to verify that the internal property being added here is handled properly.
We rely on "dfs.internal.nameservices" in 2.4.0, "dfs.nameservices" might contain name services, not managed by Ambari.
- Dmytro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46327/#review129331
-----------------------------------------------------------
On Апрель 18, 2016, 1:39 п.п., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46327/
> -----------------------------------------------------------
>
> (Updated Апрель 18, 2016, 1:39 п.п.)
>
>
> Review request for Ambari, John Speidel and Sumit Mohanty.
>
>
> Bugs: AMBARI-15791
> https://issues.apache.org/jira/browse/AMBARI-15791
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Blueprint config processing engine should automatically populate dfs.internal.nameservices when it is deploying HA clusters. The work-around till then is to explicitly add the property when submitting the blueprint.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java 1433a1b
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java 834953b
>
> Diff: https://reviews.apache.org/r/46327/diff/
>
>
> Testing
> -------
>
> Unit tests passed
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 46327: Blueprint based installations should
automatically add dfs.internal.nameservices during cluster creation
Posted by Robert Nettleton <rn...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46327/#review129331
-----------------------------------------------------------
Could we get some more background context as to why this change is necessary? I've checked the associated Apache JIRA, and there isn't much information there.
In particular, can the submitter please explain why this internal property needs to be added, especially considering that the HDFS documentation seems to indicate that the public and internal forms of this property should generally have the same value?
Thanks.
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java (line 858)
<https://reviews.apache.org/r/46327/#comment192783>
Why is this internal property being checked here?
The update change above seems to indicate that these properties should always be set to the same values. If that is the case, then why is this method being modified?
ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterTopologyImpl.java (line 193)
<https://reviews.apache.org/r/46327/#comment192785>
+1 on Sumit's comment. This change does not appear necessary.
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java (line 4900)
<https://reviews.apache.org/r/46327/#comment192786>
I'm not certain that the tests using the public property should be modified. I'd recommend adding new unit tests to verify that the internal property being added here is handled properly.
- Robert Nettleton
On April 18, 2016, 1:07 p.m., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46327/
> -----------------------------------------------------------
>
> (Updated April 18, 2016, 1:07 p.m.)
>
>
> Review request for Ambari, John Speidel and Sumit Mohanty.
>
>
> Bugs: AMBARI-15791
> https://issues.apache.org/jira/browse/AMBARI-15791
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Blueprint config processing engine should automatically populate dfs.internal.nameservices when it is deploying HA clusters. The work-around till then is to explicitly add the property when submitting the blueprint.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java 1433a1b
> ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterTopologyImpl.java 4f2f02d
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java 834953b
>
> Diff: https://reviews.apache.org/r/46327/diff/
>
>
> Testing
> -------
>
> Unit tests passed
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 46327: Blueprint based installations should
automatically add dfs.internal.nameservices during cluster creation
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46327/#review129328
-----------------------------------------------------------
ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterTopologyImpl.java (line 193)
<https://reviews.apache.org/r/46327/#comment192782>
This change should not be needed. Existence of dfs.nameservices should be enough.
- Sumit Mohanty
On April 18, 2016, 1:07 p.m., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46327/
> -----------------------------------------------------------
>
> (Updated April 18, 2016, 1:07 p.m.)
>
>
> Review request for Ambari, John Speidel and Sumit Mohanty.
>
>
> Bugs: AMBARI-15791
> https://issues.apache.org/jira/browse/AMBARI-15791
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Blueprint config processing engine should automatically populate dfs.internal.nameservices when it is deploying HA clusters. The work-around till then is to explicitly add the property when submitting the blueprint.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java 1433a1b
> ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterTopologyImpl.java 4f2f02d
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java 834953b
>
> Diff: https://reviews.apache.org/r/46327/diff/
>
>
> Testing
> -------
>
> Unit tests passed
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 46327: Blueprint based installations should
automatically add dfs.internal.nameservices during cluster creation
Posted by Robert Nettleton <rn...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46327/#review129340
-----------------------------------------------------------
Ship it!
Ship It!
- Robert Nettleton
On April 18, 2016, 1:39 p.m., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46327/
> -----------------------------------------------------------
>
> (Updated April 18, 2016, 1:39 p.m.)
>
>
> Review request for Ambari, John Speidel and Sumit Mohanty.
>
>
> Bugs: AMBARI-15791
> https://issues.apache.org/jira/browse/AMBARI-15791
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Blueprint config processing engine should automatically populate dfs.internal.nameservices when it is deploying HA clusters. The work-around till then is to explicitly add the property when submitting the blueprint.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java 1433a1b
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java 834953b
>
> Diff: https://reviews.apache.org/r/46327/diff/
>
>
> Testing
> -------
>
> Unit tests passed
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 46327: Blueprint based installations should
automatically add dfs.internal.nameservices during cluster creation
Posted by Dmytro Sen <ds...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46327/
-----------------------------------------------------------
(Updated Апрель 18, 2016, 1:39 п.п.)
Review request for Ambari, John Speidel and Sumit Mohanty.
Bugs: AMBARI-15791
https://issues.apache.org/jira/browse/AMBARI-15791
Repository: ambari
Description
-------
Blueprint config processing engine should automatically populate dfs.internal.nameservices when it is deploying HA clusters. The work-around till then is to explicitly add the property when submitting the blueprint.
Diffs (updated)
-----
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java 1433a1b
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java 834953b
Diff: https://reviews.apache.org/r/46327/diff/
Testing
-------
Unit tests passed
Thanks,
Dmytro Sen