You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/04/01 19:26:13 UTC
[GitHub] [hadoop-ozone] hanishakoneru opened a new pull request #749:
HDDS-3322. StandAlone Pipelines are created in an infinite loop
hanishakoneru opened a new pull request #749: HDDS-3322. StandAlone Pipelines are created in an infinite loop
URL: https://github.com/apache/hadoop-ozone/pull/749
## What changes were proposed in this pull request?
BackgroundPipelineCreator keeps creating pipelines of configured Replication type and all available Replication factors until some exception occurs while creating the pipeline such as no more available nodes.
When Replication Type is set to STAND_ALONE, we do not check if a DN has already been used to create a pipeline of same factor or not and keep reusing the same DNs to create new pipelines. This causes the pipeline creation to happen in an infinite loop.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-3322
## How was this patch tested?
Tested on a real cluster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] vivekratnavel commented on issue #749: HDDS-3322.
StandAlone Pipelines are created in an infinite loop
Posted by GitBox <gi...@apache.org>.
vivekratnavel commented on issue #749: HDDS-3322. StandAlone Pipelines are created in an infinite loop
URL: https://github.com/apache/hadoop-ozone/pull/749#issuecomment-614412497
@hanishakoneru Thanks for working on this! Can you please fix checkstyle issues and address review comments?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on issue #749: HDDS-3322.
StandAlone Pipelines are created in an infinite loop
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on issue #749: HDDS-3322. StandAlone Pipelines are created in an infinite loop
URL: https://github.com/apache/hadoop-ozone/pull/749#issuecomment-615032101
Thank You @hanishakoneru for the contribution and @vivekratnavel for the review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 merged pull request #749: HDDS-3322.
StandAlone Pipelines are created in an infinite loop
Posted by GitBox <gi...@apache.org>.
bharatviswa504 merged pull request #749: HDDS-3322. StandAlone Pipelines are created in an infinite loop
URL: https://github.com/apache/hadoop-ozone/pull/749
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request
#749: HDDS-3322. StandAlone Pipelines are created in an infinite loop
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #749: HDDS-3322. StandAlone Pipelines are created in an infinite loop
URL: https://github.com/apache/hadoop-ozone/pull/749#discussion_r405255429
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SimplePipelineProvider.java
##########
@@ -32,18 +31,18 @@
/**
* Implements Api for creating stand alone pipelines.
*/
-public class SimplePipelineProvider implements PipelineProvider {
+public class SimplePipelineProvider extends PipelineProvider {
- private final NodeManager nodeManager;
-
- public SimplePipelineProvider(NodeManager nodeManager) {
- this.nodeManager = nodeManager;
+ public SimplePipelineProvider(NodeManager nodeManager,
+ PipelineStateManager stateManager) {
+ super(nodeManager, stateManager);
}
@Override
public Pipeline create(ReplicationFactor factor) throws IOException {
- List<DatanodeDetails> dns =
- nodeManager.getNodes(NodeState.HEALTHY);
+ List<DatanodeDetails> dns = pickNodesNeverUsed(ReplicationType.STAND_ALONE,
Review comment:
And also for Simple Pipeline Provider, we should not create pipeline with factor 3.
This is an existing issue, but when looking into this patch found it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] hanishakoneru commented on issue #749: HDDS-3322.
StandAlone Pipelines are created in an infinite loop
Posted by GitBox <gi...@apache.org>.
hanishakoneru commented on issue #749: HDDS-3322. StandAlone Pipelines are created in an infinite loop
URL: https://github.com/apache/hadoop-ozone/pull/749#issuecomment-614744200
Thanks @bharatviswa504 and @vivekratnavel for the reviews.
> One question, so you have tested the scenario by changing Replication Type (ozone.replication.type) to STAND_ALONE.
Yup.
> And also now we have support for Ratis with 1 and 3 factor. I think now we can remove SimplePipelineProvider, Any thoughts?
Agree that we can remove it eventually.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] hanishakoneru commented on a change in pull request
#749: HDDS-3322. StandAlone Pipelines are created in an infinite loop
Posted by GitBox <gi...@apache.org>.
hanishakoneru commented on a change in pull request #749: HDDS-3322. StandAlone Pipelines are created in an infinite loop
URL: https://github.com/apache/hadoop-ozone/pull/749#discussion_r409672521
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SimplePipelineProvider.java
##########
@@ -32,18 +31,18 @@
/**
* Implements Api for creating stand alone pipelines.
*/
-public class SimplePipelineProvider implements PipelineProvider {
+public class SimplePipelineProvider extends PipelineProvider {
- private final NodeManager nodeManager;
-
- public SimplePipelineProvider(NodeManager nodeManager) {
- this.nodeManager = nodeManager;
+ public SimplePipelineProvider(NodeManager nodeManager,
+ PipelineStateManager stateManager) {
+ super(nodeManager, stateManager);
}
@Override
public Pipeline create(ReplicationFactor factor) throws IOException {
- List<DatanodeDetails> dns =
- nodeManager.getNodes(NodeState.HEALTHY);
+ List<DatanodeDetails> dns = pickNodesNeverUsed(ReplicationType.STAND_ALONE,
Review comment:
Updated to create only factor ONE pipelines for StandAlone type.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org