You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "szetszwo (via GitHub)" <gi...@apache.org> on 2026/09/02 22:20:47 UTC
[PR] HDDS-16365. Pipeline.getReplicaIndexes() unnecessarily create a new map. [ozone]
szetszwo opened a new pull request, #11194:
URL: https://github.com/apache/ozone/pull/11194
## What changes were proposed in this pull request?
## What is the link to the Apache JIRA
HDDS-16365
## How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16365. Pipeline.getReplicaIndexes() unnecessarily create a new map. [ozone]
Posted by "rich7420 (via GitHub)" <gi...@apache.org>.
rich7420 commented on code in PR #11194:
URL: https://github.com/apache/ozone/pull/11194#discussion_r3941561195
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/client/BlockID.java:
##########
@@ -98,24 +98,24 @@ public void appendTo(StringBuilder sb) {
}
@JsonIgnore
- public ContainerProtos.DatanodeBlockID getDatanodeBlockIDProtobuf() {
- ContainerProtos.DatanodeBlockID.Builder blockID = getDatanodeBlockIDProtobufBuilder();
- if (replicaIndex != null) {
- blockID.setReplicaIndex(replicaIndex);
- }
- return blockID.build();
+ public DatanodeBlockID getDatanodeBlockIDProtobuf() {
+ return getDatanodeBlockIDProtobufBuilder(replicaIndex).build();
}
@JsonIgnore
- public ContainerProtos.DatanodeBlockID.Builder getDatanodeBlockIDProtobufBuilder() {
- return ContainerProtos.DatanodeBlockID.newBuilder().
- setContainerID(containerBlockID.getContainerID())
+ public DatanodeBlockID.Builder getDatanodeBlockIDProtobufBuilder(Integer replicaIdx) {
+ final DatanodeBlockID.Builder b = DatanodeBlockID.newBuilder()
+ .setContainerID(containerBlockID.getContainerID())
.setLocalID(containerBlockID.getLocalID())
.setBlockCommitSequenceId(blockCommitSequenceId);
+ if (replicaIdx != null && replicaIdx > 0) {
Review Comment:
This drops explicit replica index 0 on re-serialization. A base/head probe (`setReplicaIndex(0)` -> `getFromProtobuf` -> serialize) keeps presence on base but loses it here. Please preserve explicit zero and add a round-trip test.
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/pipeline/Pipeline.java:
##########
@@ -240,10 +238,27 @@ public int getReplicaIndex(DatanodeDetails dn) {
}
/**
- * Get the replicaIndex Map.
+ * @param fromIndex the replica index starting from (inclusive)
+ * @param toIndex the replica index starting to (exclusive)
+ * @return true iff this pipeline contain all replica indexes within the given range.
*/
- public Map<DatanodeDetails, Integer> getReplicaIndexes() {
- return this.getNodes().stream().collect(Collectors.toMap(Function.identity(), this::getReplicaIndex));
+ public boolean containsAllReplicaIndexes(int fromIndex, int toIndex) {
+ final boolean[] contains = new boolean[toIndex - fromIndex];
+ for (int r : replicaIndexes.values()) {
+ if (r >= fromIndex && r < toIndex) {
+ contains[r - fromIndex] = true;
+ }
+ }
+ for (boolean contain : contains) {
+ if (!contain) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ public Map<DatanodeDetails, Integer> getReplicaIndexesForTesting() {
Review Comment:
This renames the `ozone admin pipeline list --json` field from `replicaIndexes` to `replicaIndexesForTesting`, since `Pipeline` is serialized directly here. A base/head `JsonUtils` probe reproduces it. Please preserve the existing property name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16365. Pipeline.getReplicaIndexes() unnecessarily create a new map. [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #11194:
URL: https://github.com/apache/ozone/pull/11194#issuecomment-5654664311
@rich7420 , @chungen0126 , thanks for reviewing this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16365. Pipeline.getReplicaIndexes() unnecessarily create a new map. [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #11194:
URL: https://github.com/apache/ozone/pull/11194#discussion_r3994263062
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/storage/ContainerProtocolCalls.java:
##########
@@ -931,41 +920,6 @@ public static List<Validator> toValidatorList(Validator validator) {
return Collections.unmodifiableList(validators);
}
- public static HashMap<DatanodeDetails, GetBlockResponseProto>
Review Comment:
@rich7420 , ContainerProtocolCalls is just a private API although the methods are public. It has nothing to do if it is already in released Ozone versions.
Creating new JIRAs for removing unused methods is wasting our resource. Let's do it here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16365. Pipeline.getReplicaIndexes() unnecessarily create a new map. [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on code in PR #11194:
URL: https://github.com/apache/ozone/pull/11194#discussion_r3995929322
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/pipeline/Pipeline.java:
##########
@@ -240,10 +238,27 @@ public int getReplicaIndex(DatanodeDetails dn) {
}
/**
- * Get the replicaIndex Map.
+ * @param fromIndex the replica index starting from (inclusive)
+ * @param toIndex the replica index starting to (exclusive)
+ * @return true iff this pipeline contain all replica indexes within the given range.
Review Comment:
typo
```suggestion
* @return true if this pipeline contains all replica indexes within the given range.
```
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/pipeline/Pipeline.java:
##########
@@ -240,10 +238,27 @@ public int getReplicaIndex(DatanodeDetails dn) {
}
/**
- * Get the replicaIndex Map.
+ * @param fromIndex the replica index starting from (inclusive)
+ * @param toIndex the replica index starting to (exclusive)
+ * @return true iff this pipeline contain all replica indexes within the given range.
*/
+ public boolean containsAllReplicaIndexes(int fromIndex, int toIndex) {
+ final boolean[] contains = new boolean[toIndex - fromIndex];
+ for (int r : replicaIndexes.values()) {
+ if (r >= fromIndex && r < toIndex) {
+ contains[r - fromIndex] = true;
+ }
+ }
+ for (boolean contain : contains) {
+ if (!contain) {
+ return false;
+ }
+ }
+ return true;
+ }
+
public Map<DatanodeDetails, Integer> getReplicaIndexes() {
Review Comment:
If this method is only used for testing, consider adding the `@VisibleForTesting` annotation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16365. Pipeline.getReplicaIndexes() unnecessarily create a new map. [ozone]
Posted by "rich7420 (via GitHub)" <gi...@apache.org>.
rich7420 commented on code in PR #11194:
URL: https://github.com/apache/ozone/pull/11194#discussion_r3996175600
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/storage/ContainerProtocolCalls.java:
##########
@@ -931,41 +920,6 @@ public static List<Validator> toValidatorList(Validator validator) {
return Collections.unmodifiableList(validators);
}
- public static HashMap<DatanodeDetails, GetBlockResponseProto>
Review Comment:
Oh ok, thanks for the explaination
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16365. Pipeline.getReplicaIndexes() unnecessarily create a new map. [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #11194:
URL: https://github.com/apache/ozone/pull/11194#discussion_r3996690328
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/pipeline/Pipeline.java:
##########
@@ -240,10 +238,27 @@ public int getReplicaIndex(DatanodeDetails dn) {
}
/**
- * Get the replicaIndex Map.
+ * @param fromIndex the replica index starting from (inclusive)
+ * @param toIndex the replica index starting to (exclusive)
+ * @return true iff this pipeline contain all replica indexes within the given range.
*/
+ public boolean containsAllReplicaIndexes(int fromIndex, int toIndex) {
+ final boolean[] contains = new boolean[toIndex - fromIndex];
+ for (int r : replicaIndexes.values()) {
+ if (r >= fromIndex && r < toIndex) {
+ contains[r - fromIndex] = true;
+ }
+ }
+ for (boolean contain : contains) {
+ if (!contain) {
+ return false;
+ }
+ }
+ return true;
+ }
+
public Map<DatanodeDetails, Integer> getReplicaIndexes() {
Review Comment:
As @rich7420 mentioned, it is used for JSON. Otherwise, I would rename it to getReplicaIndexesForTesting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16365. Pipeline.getReplicaIndexes() unnecessarily create a new map. [ozone]
Posted by "rich7420 (via GitHub)" <gi...@apache.org>.
rich7420 commented on code in PR #11194:
URL: https://github.com/apache/ozone/pull/11194#discussion_r3941561195
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/client/BlockID.java:
##########
@@ -98,24 +98,24 @@ public void appendTo(StringBuilder sb) {
}
@JsonIgnore
- public ContainerProtos.DatanodeBlockID getDatanodeBlockIDProtobuf() {
- ContainerProtos.DatanodeBlockID.Builder blockID = getDatanodeBlockIDProtobufBuilder();
- if (replicaIndex != null) {
- blockID.setReplicaIndex(replicaIndex);
- }
- return blockID.build();
+ public DatanodeBlockID getDatanodeBlockIDProtobuf() {
+ return getDatanodeBlockIDProtobufBuilder(replicaIndex).build();
}
@JsonIgnore
- public ContainerProtos.DatanodeBlockID.Builder getDatanodeBlockIDProtobufBuilder() {
- return ContainerProtos.DatanodeBlockID.newBuilder().
- setContainerID(containerBlockID.getContainerID())
+ public DatanodeBlockID.Builder getDatanodeBlockIDProtobufBuilder(Integer replicaIdx) {
+ final DatanodeBlockID.Builder b = DatanodeBlockID.newBuilder()
+ .setContainerID(containerBlockID.getContainerID())
.setLocalID(containerBlockID.getLocalID())
.setBlockCommitSequenceId(blockCommitSequenceId);
+ if (replicaIdx != null && replicaIdx > 0) {
Review Comment:
This drops explicit replica index 0 on re-serialization. A base/head probe ( -> -> serialize) keeps presence on base but loses it here. Please preserve explicit zero and add a round-trip test.
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/pipeline/Pipeline.java:
##########
@@ -240,10 +238,27 @@ public int getReplicaIndex(DatanodeDetails dn) {
}
/**
- * Get the replicaIndex Map.
+ * @param fromIndex the replica index starting from (inclusive)
+ * @param toIndex the replica index starting to (exclusive)
+ * @return true iff this pipeline contain all replica indexes within the given range.
*/
- public Map<DatanodeDetails, Integer> getReplicaIndexes() {
- return this.getNodes().stream().collect(Collectors.toMap(Function.identity(), this::getReplicaIndex));
+ public boolean containsAllReplicaIndexes(int fromIndex, int toIndex) {
+ final boolean[] contains = new boolean[toIndex - fromIndex];
+ for (int r : replicaIndexes.values()) {
+ if (r >= fromIndex && r < toIndex) {
+ contains[r - fromIndex] = true;
+ }
+ }
+ for (boolean contain : contains) {
+ if (!contain) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ public Map<DatanodeDetails, Integer> getReplicaIndexesForTesting() {
Review Comment:
This renames the field from to , since is serialized directly here. A base/head probe reproduces it. Please preserve the existing property name.
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/storage/ContainerProtocolCalls.java:
##########
@@ -931,41 +920,6 @@ public static List<Validator> toValidatorList(Validator validator) {
return Collections.unmodifiableList(validators);
}
- public static HashMap<DatanodeDetails, GetBlockResponseProto>
Review Comment:
Please avoid removing this public method in HDDS-16365. It is unrelated, was added by HDDS-3867, and exists in released Ozone versions; please restore it or move the cleanup to a follow-up Jira.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16365. Pipeline.getReplicaIndexes() unnecessarily create a new map. [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo merged PR #11194:
URL: https://github.com/apache/ozone/pull/11194
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org