You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "sodonnel (via GitHub)" <gi...@apache.org> on 2023/06/05 12:08:50 UTC
[GitHub] [ozone] sodonnel opened a new pull request, #4829: HDDS-8698. EC: Avoid unbounded pipeline creation if all existing pipelines don't meet criteria
sodonnel opened a new pull request, #4829:
URL: https://github.com/apache/ozone/pull/4829
## What changes were proposed in this pull request?
Currently, if the pipeline limit is not reached, when a new block is requested, a new pipeline is allocated and returned to the client.
If the pipeline limit is reached, we attempt to return a pipeline from the existing set.
If none of those pipelines meet the criteria, then we fall back to creating another pipeline. This could result in unbounded pipeline growth if a client is having problems.
We should therefore limit the pipelines that can be created via this exceptional case.
For example, if we close some pipelines when searching the existing list (due to them being full), we know we have freed up some capacity to create a new pipeline.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8698
## How was this patch tested?
Modified existing tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sodonnel commented on a diff in pull request #4829: HDDS-8698. EC: Avoid unbounded pipeline creation if all existing pipelines don't meet criteria
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #4829:
URL: https://github.com/apache/ozone/pull/4829#discussion_r1218278044
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/WritableECContainerProvider.java:
##########
@@ -143,14 +145,20 @@ public ContainerInfo getContainer(final long size,
+ "container", e);
existingPipelines.remove(pipelineIndex);
pipelineManager.closePipeline(pipeline, true);
+ openPipelineCount--;
}
}
}
// If we get here, all the pipelines we tried were no good. So try to
// allocate a new one.
try {
synchronized (this) {
- return allocateContainer(repConfig, size, owner, excludeList);
+ if (openPipelineCount < minimumPipelines) {
+ return allocateContainer(repConfig, size, owner, excludeList);
+ }
+ throw new IOException("Unable to allocate a pipeline for "
+ + repConfig + " after trying all existing pipelines as the max "
+ + "limit has been reached and no pipelines where closed");
Review Comment:
yea it is a bit confusing now the meaning has changed. Should be an easy change (hopefully).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai merged pull request #4829: HDDS-8698. EC: Avoid unbounded pipeline creation if all existing pipelines don't meet criteria
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #4829:
URL: https://github.com/apache/ozone/pull/4829
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #4829: HDDS-8698. EC: Avoid unbounded pipeline creation if all existing pipelines don't meet criteria
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #4829:
URL: https://github.com/apache/ozone/pull/4829#discussion_r1218260791
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/WritableECContainerProvider.java:
##########
@@ -143,14 +145,20 @@ public ContainerInfo getContainer(final long size,
+ "container", e);
existingPipelines.remove(pipelineIndex);
pipelineManager.closePipeline(pipeline, true);
+ openPipelineCount--;
}
}
}
// If we get here, all the pipelines we tried were no good. So try to
// allocate a new one.
try {
synchronized (this) {
- return allocateContainer(repConfig, size, owner, excludeList);
+ if (openPipelineCount < minimumPipelines) {
+ return allocateContainer(repConfig, size, owner, excludeList);
+ }
+ throw new IOException("Unable to allocate a pipeline for "
+ + repConfig + " after trying all existing pipelines as the max "
+ + "limit has been reached and no pipelines where closed");
Review Comment:
Nit: I wonder if we should rename `minimumPipelines` (and `getMinimumPipelines`) to `maximum`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org