You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "slfan1989 (via GitHub)" <gi...@apache.org> on 2026/05/15 06:03:00 UTC
[PR] HDDS-15280. Fix EC pipeline factor calculation. [ozone]
slfan1989 opened a new pull request, #10272:
URL: https://github.com/apache/ozone/pull/10272
## What changes were proposed in this pull request?
This PR fixes the EC pipeline limit calculation in WritableECContainerProvider.
`ozone.scm.ec.pipeline.per.volume.factor` is a double configuration, but the previous calculation cast the factor to int before multiplying it by the healthy volume count.
As a result, fractional values such as `0.5` were truncated to `0`, causing the volume-based EC pipeline limit to be calculated incorrectly.
This PR updates the calculation to apply the factor first and cast only the final result to int.
It also adds unit test coverage for fractional factor values.
## What is the link to the Apache JIRA
JIRA: HDDS-15280. Fix EC pipeline factor calculation.
## How was this patch tested?
Add Junit Test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15280. Fix EC pipeline factor calculation. [ozone]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #10272:
URL: https://github.com/apache/ozone/pull/10272#issuecomment-4466132191
> Thanks @slfan1989 for the patch, @yandrey321 for the review.
@adoroszlai Thanks for the review and merge! @yandrey321 Thanks for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15280. Fix EC pipeline factor calculation. [ozone]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #10272:
URL: https://github.com/apache/ozone/pull/10272#discussion_r3249285894
##########
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestWritableECContainerProvider.java:
##########
@@ -194,6 +194,23 @@ void testPipelinesCreatedBasedOnTotalDiskCountWithFactor(
assertReusesExisting(allocated, pipelineLimit);
}
+ @ParameterizedTest
+ @MethodSource("policies")
+ void testPipelinesCreatedBasedOnTotalDiskCountWithFractionalFactor(
+ PipelineChoosePolicy policy) throws IOException {
+ provider = createSubject(policy);
+ double factor = 0.5;
+ providerConf.setMinimumPipelines(1);
+ providerConf.setPipelinePerVolumeFactor(factor);
+ nodeManager.setNumHealthyVolumes(20);
+
+ int volumeCount = nodeManager.totalHealthyVolumeCount();
+ int pipelineLimit =
+ (int) (factor * volumeCount / repConfig.getRequiredNodes());
+ Set<ContainerInfo> allocated = assertDistinctContainers(pipelineLimit);
+ assertReusesExisting(allocated, pipelineLimit);
+ }
+
Review Comment:
Updated the existing factor test to use a fractional value and removed the duplicate test case.
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/WritableECContainerProvider.java:
##########
@@ -187,7 +187,8 @@ private int getMaximumPipelines(ECReplicationConfig repConfig) {
int volumeBasedCount = 0;
if (factor > 0) {
int volumes = nodeManager.totalHealthyVolumeCount();
- volumeBasedCount = (int) factor * volumes / repConfig.getRequiredNodes();
+ volumeBasedCount =
+ (int) (factor * volumes / repConfig.getRequiredNodes());
Review Comment:
Fixed, moved it back to a single line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15280. Fix EC pipeline factor calculation. [ozone]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #10272:
URL: https://github.com/apache/ozone/pull/10272#discussion_r3249284645
##########
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestWritableECContainerProvider.java:
##########
@@ -194,6 +194,23 @@ void testPipelinesCreatedBasedOnTotalDiskCountWithFactor(
assertReusesExisting(allocated, pipelineLimit);
}
+ @ParameterizedTest
+ @MethodSource("policies")
+ void testPipelinesCreatedBasedOnTotalDiskCountWithFractionalFactor(
+ PipelineChoosePolicy policy) throws IOException {
+ provider = createSubject(policy);
+ double factor = 0.5;
+ providerConf.setMinimumPipelines(1);
+ providerConf.setPipelinePerVolumeFactor(factor);
+ nodeManager.setNumHealthyVolumes(20);
+
+ int volumeCount = nodeManager.totalHealthyVolumeCount();
+ int pipelineLimit =
+ (int) (factor * volumeCount / repConfig.getRequiredNodes());
+ Set<ContainerInfo> allocated = assertDistinctContainers(pipelineLimit);
+ assertReusesExisting(allocated, pipelineLimit);
+ }
+
Review Comment:
Updated the existing factor test to use a fractional value and removed the duplicate test case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15280. Fix EC pipeline factor calculation. [ozone]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #10272:
URL: https://github.com/apache/ozone/pull/10272#discussion_r3249285894
##########
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestWritableECContainerProvider.java:
##########
@@ -194,6 +194,23 @@ void testPipelinesCreatedBasedOnTotalDiskCountWithFactor(
assertReusesExisting(allocated, pipelineLimit);
}
+ @ParameterizedTest
+ @MethodSource("policies")
+ void testPipelinesCreatedBasedOnTotalDiskCountWithFractionalFactor(
+ PipelineChoosePolicy policy) throws IOException {
+ provider = createSubject(policy);
+ double factor = 0.5;
+ providerConf.setMinimumPipelines(1);
+ providerConf.setPipelinePerVolumeFactor(factor);
+ nodeManager.setNumHealthyVolumes(20);
+
+ int volumeCount = nodeManager.totalHealthyVolumeCount();
+ int pipelineLimit =
+ (int) (factor * volumeCount / repConfig.getRequiredNodes());
+ Set<ContainerInfo> allocated = assertDistinctContainers(pipelineLimit);
+ assertReusesExisting(allocated, pipelineLimit);
+ }
+
Review Comment:
Updated the existing factor test to use a fractional value and removed the duplicate test case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15280. Fix EC pipeline factor calculation. [ozone]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #10272:
URL: https://github.com/apache/ozone/pull/10272#issuecomment-4458319141
@adoroszlai @ChenSammi Could you please take a look at this PR when you have time? Any feedback would be appreciated. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15280. Fix EC pipeline factor calculation. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #10272:
URL: https://github.com/apache/ozone/pull/10272#issuecomment-4466098210
Thanks @slfan1989 for the patch, @yandrey321 for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15280. Fix EC pipeline factor calculation. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #10272:
URL: https://github.com/apache/ozone/pull/10272
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15280. Fix EC pipeline factor calculation. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #10272:
URL: https://github.com/apache/ozone/pull/10272#discussion_r3248239797
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/WritableECContainerProvider.java:
##########
@@ -187,7 +187,8 @@ private int getMaximumPipelines(ECReplicationConfig repConfig) {
int volumeBasedCount = 0;
if (factor > 0) {
int volumes = nodeManager.totalHealthyVolumeCount();
- volumeBasedCount = (int) factor * volumes / repConfig.getRequiredNodes();
+ volumeBasedCount =
+ (int) (factor * volumes / repConfig.getRequiredNodes());
Review Comment:
nit: no need to wrap to new line
##########
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestWritableECContainerProvider.java:
##########
@@ -194,6 +194,23 @@ void testPipelinesCreatedBasedOnTotalDiskCountWithFactor(
assertReusesExisting(allocated, pipelineLimit);
}
+ @ParameterizedTest
+ @MethodSource("policies")
+ void testPipelinesCreatedBasedOnTotalDiskCountWithFractionalFactor(
+ PipelineChoosePolicy policy) throws IOException {
+ provider = createSubject(policy);
+ double factor = 0.5;
+ providerConf.setMinimumPipelines(1);
+ providerConf.setPipelinePerVolumeFactor(factor);
+ nodeManager.setNumHealthyVolumes(20);
+
+ int volumeCount = nodeManager.totalHealthyVolumeCount();
+ int pipelineLimit =
+ (int) (factor * volumeCount / repConfig.getRequiredNodes());
+ Set<ContainerInfo> allocated = assertDistinctContainers(pipelineLimit);
+ assertReusesExisting(allocated, pipelineLimit);
+ }
+
Review Comment:
I don't think we need a separate test case (with duplicated code) to test both fractional and integer factors. Integer factor can be simply replaced with fractional one in existing test case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org