You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "devmadhuu (via GitHub)" <gi...@apache.org> on 2025/06/25 10:53:31 UTC
[PR] HDDS-13335. Ozone Recon - Implement and adopt lightweight proto for Recon listKeys API. [ozone]
devmadhuu opened a new pull request, #8699:
URL: https://github.com/apache/ozone/pull/8699
## What changes were proposed in this pull request?
Recon's listKeys API (/api/v1/keys/listKeys) uses full KeyInfo proto where lot of other key proto fields are not desired and needed. Some of them are like
`KeyLocationList, FileEncryptionInfoProto, OzoneAclInfo, FileChecksumProto etc `
In case of large keys or Multi part upload keys, key locations list can be huge and can create performance impact for some clients calling the listKeys API. So suggested and proposed light weight proto is:
```
message KeyInfoProtoLight {
required string volumeName = 1;
required string bucketName = 2;
required string keyName = 3;
required uint64 dataSize = 4;
required hadoop.hdds.ReplicationType type = 5;
optional hadoop.hdds.ReplicationFactor factor = 6;
required uint64 creationTime = 8;
required uint64 modificationTime = 9;
optional uint64 objectID = 14;
optional uint64 updateID = 15;
optional uint64 parentID = 16;
optional hadoop.hdds.ECReplicationConfig ecReplicationConfig = 17;
optional bool isFile = 19;
}
```
Above proto is light weight compared to keyinfo proto and skipped not needed fields.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-13335
## How was this patch tested?
This patch is tested with existing junit and integration test cases. And tested using docker cluster locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13335. Implement and adopt lightweight proto for Recon listKeys API [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on code in PR #8699:
URL: https://github.com/apache/ozone/pull/8699#discussion_r2199823485
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/ReconBasicOmKeyInfo.java:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api.types;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.util.Objects;
+import org.apache.hadoop.hdds.client.ReplicationConfig;
+import org.apache.hadoop.hdds.utils.db.Codec;
+import org.apache.hadoop.hdds.utils.db.DelegatedCodec;
+import org.apache.hadoop.hdds.utils.db.Proto2Codec;
+import org.apache.hadoop.ozone.om.helpers.QuotaUtil;
+import org.apache.hadoop.ozone.om.helpers.WithParentObjectId;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos;
+
+/**
+ * Lightweight OmKeyInfo class.
+ */
+@JsonIgnoreProperties({
+ "metadata", "objectID", "updateID", "parentObjectID", "objectInfo"
+})
+public final class ReconBasicOmKeyInfo extends WithParentObjectId {
+
+ private final String volumeName;
+ private final String bucketName;
+ private final String keyName;
+ private final long dataSize;
+ private final long creationTime;
+ private final long modificationTime;
+
+ /** This is key table key of rocksDB and will help UI to implement pagination
+ * where UI will use the last record key to send in API as preKeyPrefix. */
+ @JsonProperty("key")
+ private String key;
+
+ /** Path of a key/file. */
+ @JsonProperty("path")
+ private String path;
+
+ @JsonProperty("replicatedSize")
+ private final long replicatedSize;
+
+ @JsonProperty("replicationInfo")
+ private final ReplicationConfig replicationConfig;
+
+ private final boolean isFile;
+ private long parentId;
+
+ public static Codec<ReconBasicOmKeyInfo> getCodec() {
+ return new DelegatedCodec<>(
+ Proto2Codec.get(OzoneManagerProtocolProtos.KeyInfoProtoLight.getDefaultInstance()),
+ ReconBasicOmKeyInfo::getFromProtobuf,
+ ReconBasicOmKeyInfo::toProtobuf,
Review Comment:
Ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13335. Implement and adopt lightweight proto for Recon listKeys API [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl merged PR #8699:
URL: https://github.com/apache/ozone/pull/8699
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13335. Implement and adopt lightweight proto for Recon listKeys API [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on code in PR #8699:
URL: https://github.com/apache/ozone/pull/8699#discussion_r2199774036
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/ReconBasicOmKeyInfo.java:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api.types;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.util.Objects;
+import org.apache.hadoop.hdds.client.ReplicationConfig;
+import org.apache.hadoop.hdds.utils.db.Codec;
+import org.apache.hadoop.hdds.utils.db.DelegatedCodec;
+import org.apache.hadoop.hdds.utils.db.Proto2Codec;
+import org.apache.hadoop.ozone.om.helpers.QuotaUtil;
+import org.apache.hadoop.ozone.om.helpers.WithParentObjectId;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos;
+
+/**
+ * Lightweight OmKeyInfo class.
+ */
+@JsonIgnoreProperties({
+ "metadata", "objectID", "updateID", "parentObjectID", "objectInfo"
+})
+public final class ReconBasicOmKeyInfo extends WithParentObjectId {
Review Comment:
Yes, this will be used for other features as well including deleted keys space utilization feature.
##########
hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto:
##########
@@ -1172,6 +1172,22 @@ message KeyInfo {
optional uint64 expectedDataGeneration = 22;
}
+message KeyInfoProtoLight {
Review Comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13335. Implement and adopt lightweight proto for Recon listKeys API [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #8699:
URL: https://github.com/apache/ozone/pull/8699#discussion_r2198165231
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/ReconBasicOmKeyInfo.java:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api.types;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.util.Objects;
+import org.apache.hadoop.hdds.client.ReplicationConfig;
+import org.apache.hadoop.hdds.utils.db.Codec;
+import org.apache.hadoop.hdds.utils.db.DelegatedCodec;
+import org.apache.hadoop.hdds.utils.db.Proto2Codec;
+import org.apache.hadoop.ozone.om.helpers.QuotaUtil;
+import org.apache.hadoop.ozone.om.helpers.WithParentObjectId;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos;
+
+/**
+ * Lightweight OmKeyInfo class.
+ */
+@JsonIgnoreProperties({
+ "metadata", "objectID", "updateID", "parentObjectID", "objectInfo"
+})
+public final class ReconBasicOmKeyInfo extends WithParentObjectId {
Review Comment:
ReconBasicOmKeyInfo is coupled with response format defining json property. Do this class can be used anywhere in recon if required or any other usecase in recon generic be used?
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/ReconBasicOmKeyInfo.java:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api.types;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.util.Objects;
+import org.apache.hadoop.hdds.client.ReplicationConfig;
+import org.apache.hadoop.hdds.utils.db.Codec;
+import org.apache.hadoop.hdds.utils.db.DelegatedCodec;
+import org.apache.hadoop.hdds.utils.db.Proto2Codec;
+import org.apache.hadoop.ozone.om.helpers.QuotaUtil;
+import org.apache.hadoop.ozone.om.helpers.WithParentObjectId;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos;
+
+/**
+ * Lightweight OmKeyInfo class.
+ */
+@JsonIgnoreProperties({
+ "metadata", "objectID", "updateID", "parentObjectID", "objectInfo"
+})
+public final class ReconBasicOmKeyInfo extends WithParentObjectId {
+
+ private final String volumeName;
+ private final String bucketName;
+ private final String keyName;
+ private final long dataSize;
+ private final long creationTime;
+ private final long modificationTime;
+
+ /** This is key table key of rocksDB and will help UI to implement pagination
+ * where UI will use the last record key to send in API as preKeyPrefix. */
+ @JsonProperty("key")
+ private String key;
+
+ /** Path of a key/file. */
+ @JsonProperty("path")
+ private String path;
+
+ @JsonProperty("replicatedSize")
+ private final long replicatedSize;
+
+ @JsonProperty("replicationInfo")
+ private final ReplicationConfig replicationConfig;
+
+ private final boolean isFile;
+ private long parentId;
+
+ public static Codec<ReconBasicOmKeyInfo> getCodec() {
+ return new DelegatedCodec<>(
+ Proto2Codec.get(OzoneManagerProtocolProtos.KeyInfoProtoLight.getDefaultInstance()),
+ ReconBasicOmKeyInfo::getFromProtobuf,
+ ReconBasicOmKeyInfo::toProtobuf,
Review Comment:
Instead of defining method for every case, we can define as below, above can be changed as,
```
DelegateCodec.decodeOnly(
Proto2Codec.get(OzoneManagerProtocolProtos.KeyInfoProtoLight.getDefaultInstance()),
ReconBasicOmKeyInfo::getFromProtobuf
ReconBasicOmKeyInfo.class);
```
```
public static <T, DELEGATE> DelegatedCodec<T, DELEGATE> decodeOnly(
Codec<DELEGATE> delegate, CheckedFunction<DELEGATE, T, CodecException> forward, Class<T> clazz) {
return new DelegatedCodec<>(delegate, forward, unsupportedBackward(), clazz, CopyType.DEEP);
}
private static <A, B> CheckedFunction<A, B, CodecException> unsupportedBackward() {
return a -> { throw new UnsupportedOperationException("Unsupported backward conversion"); };
}
##########
hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto:
##########
@@ -1172,6 +1172,22 @@ message KeyInfo {
optional uint64 expectedDataGeneration = 22;
}
+message KeyInfoProtoLight {
Review Comment:
add comment for this type specifying its sub-part of keyInfo with selected fields only and keeping index same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org