You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/04/13 14:57:40 UTC
[GitHub] [ozone] xBis7 opened a new pull request, #3309: HDDS-5504. Support namespace summaries (du, dist & counts) for legacy FS buckets
xBis7 opened a new pull request, #3309:
URL: https://github.com/apache/ozone/pull/3309
## What changes were proposed in this pull request?
The `Recon/api/NSSummaryEndpoint.java` so far has been handling only FSO buckets.
Keeping the class as it is, makes it very difficult to make an implementation for non FSO buckets as well.
In order to allow the addition of the nonFSO bucket handling code, we refactored the Bucket specific code out of the NSSummaryEndpoint. We created an abstract BucketHandler class with a concrete FSOBucket subclass. That subclass functions exactly as the code does now.
In the process of refactoring, we noticed the large switch statements in NSSummaryEndpoint.java. They all switch on the case of the EntityType being summarized. That didn't seem very object oriented, so we refactored those switch statements into an abstract class named EntityHandler, with concrete subclasses for each of the 6 defined entities, (Root, Volume, etc.)
No functionality was changed in the process. We just re-organized the code to make it easier to maintain/more object oriented.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5504
https://issues.apache.org/jira/browse/HDDS-5505
## How was this patch tested?
All the original tests remain unchanged and continue to pass.
To make reviews easier, we also wanted to find a way to diff the old code with the new. Ordinary diff tools can't handle this amount of refactoring. So we wrote a simple python script to insert the refactored methods back into the original switch statements. That way we could use standard diff tools to confirm no unexpected changes were made in the refactoring.
So for example, [using GitHub diffs:](https://github.com/GeorgeJahad/ozone/compare/64d67668d2f8cbfa60d0b8055ee53d0c104e20d8..GeorgeJahad:reorgNSSummaryEndpoint2?w=1)
Lines 124-140 show the diffs in the Root entity case of the getBasicInfo() switch statement. In the green sections, you can see that the only diffs are that we added the getSummaryResponse() method header and replaced the "break" statement with a method return statement.
Obviously that isn't valid java, but it isn't meant to be; it is just a file of code that has been generated to allow us to be sure that the changes were as we intended.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #3309: HDDS-5504. Support namespace summaries (du, dist & counts) for legacy FS buckets
Posted by GitBox <gi...@apache.org>.
smengcl commented on code in PR #3309:
URL: https://github.com/apache/ozone/pull/3309#discussion_r943654174
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManagerUtils.java:
##########
@@ -62,7 +62,7 @@ private OzoneManagerUtils() {
* omMetadataManager().getBucketTable().get(buckKey)
*/
- private static OmBucketInfo getOmBucketInfo(OMMetadataManager metaMgr,
+ public static OmBucketInfo getOmBucketInfo(OMMetadataManager metaMgr,
Review Comment:
Got it. In the case would you also clarify this in the PR title (and jira) a bit more?
e.g.: `HDDS-5504. Refactor namespace summaries (du, dist & counts) for upcoming legacy FS buckets support`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on pull request #3309: HDDS-5504. Support namespace summaries (du, dist & counts) for legacy FS buckets
Posted by GitBox <gi...@apache.org>.
xBis7 commented on PR #3309:
URL: https://github.com/apache/ozone/pull/3309#issuecomment-1212010338
@smengcl I made the requested changes. Let me know what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #3309: HDDS-5504. Support namespace summaries (du, dist & counts) for legacy FS buckets
Posted by GitBox <gi...@apache.org>.
smengcl commented on code in PR #3309:
URL: https://github.com/apache/ozone/pull/3309#discussion_r943657525
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/BucketHandler.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+import static org.apache.hadoop.ozone.om.helpers.OzoneFSUtils.removeTrailingSlashIfNeeded;
+
+/**
+ * Abstract class for handling all bucket types.
+ * The abstract methods have different implementation for each bucket type.
+ */
+public abstract class BucketHandler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ BucketHandler.class);
+
+ private final ReconNamespaceSummaryManager reconNamespaceSummaryManager;
+
+ private final ReconOMMetadataManager omMetadataManager;
+
+ private final OzoneStorageContainerManager reconSCM;
Review Comment:
1) If this won't be used at all, I think we should remove it. We can add it back when we actually need it.
2) or, you could keep it and add a comment above the line how this is planned to be used later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on a diff in pull request #3309: HDDS-5504. Support namespace summaries (du, dist & counts) for legacy FS buckets
Posted by GitBox <gi...@apache.org>.
xBis7 commented on code in PR #3309:
URL: https://github.com/apache/ozone/pull/3309#discussion_r942917496
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/ReconRestServletModule.java:
##########
@@ -125,7 +125,7 @@ private void addFilters(String basePath, Set<String> adminSubPaths) {
if (aclEnabled) {
for (String path: adminSubPaths) {
String adminPath =
- UriBuilder.fromPath(basePath).path(path).build().toString();
+ UriBuilder.fromPath(basePath).path(path + "*").build().toString();
Review Comment:
This has been tested with the smoketest that we added for FSO NSSummaries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] avijayanhwx commented on pull request #3309: HDDS-5504. Support namespace summaries (du, dist & counts) for legacy FS buckets
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on PR #3309:
URL: https://github.com/apache/ozone/pull/3309#issuecomment-1117648747
@xBis7 Approach looks great to me, thanks for taking up this refactoring. Please finish up your patch and move this out of draft whenever possible. I can review it at greater depth.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on pull request #3309: HDDS-5504. Refactor namespace summaries (du, dist & counts) for upcoming legacy FS buckets support
Posted by GitBox <gi...@apache.org>.
xBis7 commented on PR #3309:
URL: https://github.com/apache/ozone/pull/3309#issuecomment-1212224293
@smengcl Sorry about the ticket mess. I just fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #3309: HDDS-5504. Support namespace summaries (du, dist & counts) for legacy FS buckets
Posted by GitBox <gi...@apache.org>.
GeorgeJahad commented on code in PR #3309:
URL: https://github.com/apache/ozone/pull/3309#discussion_r943638471
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/NSSummary.java:
##########
@@ -93,7 +92,7 @@ public void setChildDir(Set<Long> childDir) {
}
public void setDirName(String dirName) {
- this.dirName = dirName;
+ this.dirName = removeTrailingSlashIfNeeded(dirName);
Review Comment:
Yes, I believe it is correct and should be there for Legacy buckets. A lot of legacy logic depends on the correct placement of that slash.
Moving it to the legacy PR should be fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on a diff in pull request #3309: HDDS-5504. Support namespace summaries (du, dist & counts) for legacy FS buckets
Posted by GitBox <gi...@apache.org>.
xBis7 commented on code in PR #3309:
URL: https://github.com/apache/ozone/pull/3309#discussion_r942916932
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManagerUtils.java:
##########
@@ -62,7 +62,7 @@ private OzoneManagerUtils() {
* omMetadataManager().getBucketTable().get(buckKey)
*/
- private static OmBucketInfo getOmBucketInfo(OMMetadataManager metaMgr,
+ public static OmBucketInfo getOmBucketInfo(OMMetadataManager metaMgr,
Review Comment:
It's used for legacy buckets. This PR it's just the existing code that has been refactored. We must have missed it, I will change it back. It will make sense in the next PR where it will be used by the legacy bucket code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on pull request #3309: HDDS-5504. Support namespace summaries (du, dist & counts) for legacy FS buckets
Posted by GitBox <gi...@apache.org>.
xBis7 commented on PR #3309:
URL: https://github.com/apache/ozone/pull/3309#issuecomment-1118494730
@avijayanhwx Thanks for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on a diff in pull request #3309: HDDS-5504. Support namespace summaries (du, dist & counts) for legacy FS buckets
Posted by GitBox <gi...@apache.org>.
xBis7 commented on code in PR #3309:
URL: https://github.com/apache/ozone/pull/3309#discussion_r943662977
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/BucketHandler.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+import static org.apache.hadoop.ozone.om.helpers.OzoneFSUtils.removeTrailingSlashIfNeeded;
+
+/**
+ * Abstract class for handling all bucket types.
+ * The abstract methods have different implementation for each bucket type.
+ */
+public abstract class BucketHandler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ BucketHandler.class);
+
+ private final ReconNamespaceSummaryManager reconNamespaceSummaryManager;
+
+ private final ReconOMMetadataManager omMetadataManager;
+
+ private final OzoneStorageContainerManager reconSCM;
Review Comment:
I will remove it. It's easy to get later on if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swagle merged pull request #3309: HDDS-5504. Refactor namespace summaries (du, dist & counts) for upcoming legacy FS buckets support
Posted by GitBox <gi...@apache.org>.
swagle merged PR #3309:
URL: https://github.com/apache/ozone/pull/3309
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #3309: HDDS-5504. Support namespace summaries (du, dist & counts) for legacy FS buckets
Posted by GitBox <gi...@apache.org>.
smengcl commented on code in PR #3309:
URL: https://github.com/apache/ozone/pull/3309#discussion_r942328077
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/EntityType.java:
##########
@@ -1,4 +1,4 @@
-/*
+/**
Review Comment:
nit: unintentional change?
```suggestion
/*
```
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/NSSummary.java:
##########
@@ -36,11 +38,8 @@ public class NSSummary {
private String dirName;
public NSSummary() {
- this.numOfFiles = 0;
- this.sizeOfFiles = 0L;
- this.fileSizeBucket = new int[ReconConstants.NUM_OF_BINS];
- this.childDir = new HashSet<>();
- this.dirName = "";
+ this(0, 0, new int[ReconConstants.NUM_OF_BINS],
Review Comment:
nit. serves as a reminder that the second arg is a `long`.
```suggestion
this(0, 0L, new int[ReconConstants.NUM_OF_BINS],
```
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/NSSummary.java:
##########
@@ -93,7 +92,7 @@ public void setChildDir(Set<Long> childDir) {
}
public void setDirName(String dirName) {
- this.dirName = dirName;
+ this.dirName = removeTrailingSlashIfNeeded(dirName);
Review Comment:
Double check the usage here.
IIRC if the target is a directory (not a key), removing the trailing slash might cause issue because of the key mismatch when getting value from RocksDB.
Something might have changes since then, if this is tested unaffected I'd be fine with this change.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/BucketHandler.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+import static org.apache.hadoop.ozone.om.helpers.OzoneFSUtils.removeTrailingSlashIfNeeded;
+
+/**
+ * Abstract class for handling all bucket types.
+ * The abstract methods have different implementation for each bucket type.
+ */
+public abstract class BucketHandler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ BucketHandler.class);
+
+ private final ReconNamespaceSummaryManager reconNamespaceSummaryManager;
+
+ private final ReconOMMetadataManager omMetadataManager;
+
+ private final OzoneStorageContainerManager reconSCM;
Review Comment:
This is not used after initial assignment in constructor. Is this for future-proofing just in case we need to reach back to reconSCM?
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManagerUtils.java:
##########
@@ -62,7 +62,7 @@ private OzoneManagerUtils() {
* omMetadataManager().getBucketTable().get(buckKey)
*/
- private static OmBucketInfo getOmBucketInfo(OMMetadataManager metaMgr,
+ public static OmBucketInfo getOmBucketInfo(OMMetadataManager metaMgr,
Review Comment:
Unnecessary change here? looks like `getOmBucketInfo()` was meant to be called from `getBucketLayout()` inside the class. and there doesn't seem to have any new usages here in this PR.
```suggestion
private static OmBucketInfo getOmBucketInfo(OMMetadataManager metaMgr,
```
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestFSONSSummaryEndpoint.java:
##########
@@ -0,0 +1,1249 @@
+/*'
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.ContainerReplicaProto.State;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.container.ContainerReplica;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.handlers.BucketHandler;
+import org.apache.hadoop.ozone.recon.api.handlers.EntityHandler;
+import org.apache.hadoop.ozone.recon.api.types.NamespaceSummaryResponse;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.FileSizeDistributionResponse;
+import org.apache.hadoop.ozone.recon.api.types.ResponseStatus;
+import org.apache.hadoop.ozone.recon.api.types.QuotaUsageResponse;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.scm.ReconStorageContainerManagerFacade;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.StorageContainerServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.spi.impl.StorageContainerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.tasks.FSONSSummaryTask;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.ws.rs.core.Response;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.HashSet;
+
+import static org.apache.hadoop.hdds.protocol.MockDatanodeDetails.randomDatanodeDetails;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeDirToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProviderWithFSO;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for FSONSSummary REST APIs.
+ * We tested on a mini file system with the following setting:
+ * vol
+ * / \
+ * bucket1 bucket2
+ * / \ / \
+ * file1 dir1 file4 file5
+ * / \ \
+ * dir2 dir3 dir4
+ * / \ \
+ * file2 file3 file6
+ * ----------------------------------------
+ * vol2
+ * / \
+ * bucket3 bucket4
+ * / \ /
+ * file8 dir5 file11
+ * / \
+ * file9 file10
+ * This is a test for the Rest APIs only. We have tested NSSummaryTask before,
+ * so there is no need to test process() on DB's updates
+ */
+public class TestFSONSSummaryEndpoint {
Review Comment:
`TestNSSummaryEndpointWithFSO`
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/FSONSSummaryTask.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+import org.apache.commons.lang3.tuple.ImmutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.WithParentObjectId;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.inject.Inject;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+
+import static org.apache.hadoop.ozone.om.OmMetadataManagerImpl.DIRECTORY_TABLE;
+import static org.apache.hadoop.ozone.om.OmMetadataManagerImpl.FILE_TABLE;
+
+/**
+ * Class for handling FSO specific tasks.
+ */
+public class FSONSSummaryTask extends NSSummaryTask {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(FSONSSummaryTask.class);
+
+ @Inject
+ public FSONSSummaryTask(ReconNamespaceSummaryManager
+ reconNamespaceSummaryManager) {
+ super(reconNamespaceSummaryManager);
+ }
+
+ @Override
+ public String getTaskName() {
+ return "FSONSSummaryTask";
+ }
+
+ // We only listen to updates from FSO-enabled KeyTable(FileTable) and DirTable
+ public Collection<String> getTaskTables() {
+ return Arrays.asList(new String[]{FILE_TABLE, DIRECTORY_TABLE});
Review Comment:
nit: redundant `new String[]`
```suggestion
return Arrays.asList(FILE_TABLE, DIRECTORY_TABLE);
```
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestFSONSSummaryEndpoint.java:
##########
@@ -0,0 +1,1249 @@
+/*'
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.ContainerReplicaProto.State;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.container.ContainerReplica;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.handlers.BucketHandler;
+import org.apache.hadoop.ozone.recon.api.handlers.EntityHandler;
+import org.apache.hadoop.ozone.recon.api.types.NamespaceSummaryResponse;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.FileSizeDistributionResponse;
+import org.apache.hadoop.ozone.recon.api.types.ResponseStatus;
+import org.apache.hadoop.ozone.recon.api.types.QuotaUsageResponse;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.scm.ReconStorageContainerManagerFacade;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.StorageContainerServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.spi.impl.StorageContainerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.tasks.FSONSSummaryTask;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.ws.rs.core.Response;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.HashSet;
+
+import static org.apache.hadoop.hdds.protocol.MockDatanodeDetails.randomDatanodeDetails;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeDirToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProviderWithFSO;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for FSONSSummary REST APIs.
Review Comment:
```suggestion
* Test for NSSummary REST APIs with FSO.
```
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/FSONSSummaryTask.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+import org.apache.commons.lang3.tuple.ImmutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.WithParentObjectId;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.inject.Inject;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+
+import static org.apache.hadoop.ozone.om.OmMetadataManagerImpl.DIRECTORY_TABLE;
+import static org.apache.hadoop.ozone.om.OmMetadataManagerImpl.FILE_TABLE;
+
+/**
+ * Class for handling FSO specific tasks.
+ */
+public class FSONSSummaryTask extends NSSummaryTask {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(FSONSSummaryTask.class);
+
+ @Inject
+ public FSONSSummaryTask(ReconNamespaceSummaryManager
+ reconNamespaceSummaryManager) {
+ super(reconNamespaceSummaryManager);
+ }
+
+ @Override
+ public String getTaskName() {
+ return "FSONSSummaryTask";
Review Comment:
`NSSummaryTaskWithFSO`
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestFSONSSummaryEndpoint.java:
##########
@@ -0,0 +1,1249 @@
+/*'
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos.ContainerReplicaProto.State;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.container.ContainerReplica;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.handlers.BucketHandler;
+import org.apache.hadoop.ozone.recon.api.handlers.EntityHandler;
+import org.apache.hadoop.ozone.recon.api.types.NamespaceSummaryResponse;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.FileSizeDistributionResponse;
+import org.apache.hadoop.ozone.recon.api.types.ResponseStatus;
+import org.apache.hadoop.ozone.recon.api.types.QuotaUsageResponse;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.scm.ReconStorageContainerManagerFacade;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.StorageContainerServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.spi.impl.StorageContainerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.tasks.FSONSSummaryTask;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.ws.rs.core.Response;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.HashSet;
+
+import static org.apache.hadoop.hdds.protocol.MockDatanodeDetails.randomDatanodeDetails;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeDirToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProviderWithFSO;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for FSONSSummary REST APIs.
Review Comment:
Roughly, `%s/FSONSSummary/NSSummaryWithFSO/g`
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/ReconRestServletModule.java:
##########
@@ -125,7 +125,7 @@ private void addFilters(String basePath, Set<String> adminSubPaths) {
if (aclEnabled) {
for (String path: adminSubPaths) {
String adminPath =
- UriBuilder.fromPath(basePath).path(path).build().toString();
+ UriBuilder.fromPath(basePath).path(path + "*").build().toString();
Review Comment:
Good catch. Is this change tested (manually or with UT)?
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/FSONSSummaryTask.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+import org.apache.commons.lang3.tuple.ImmutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.WithParentObjectId;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.inject.Inject;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+
+import static org.apache.hadoop.ozone.om.OmMetadataManagerImpl.DIRECTORY_TABLE;
+import static org.apache.hadoop.ozone.om.OmMetadataManagerImpl.FILE_TABLE;
+
+/**
+ * Class for handling FSO specific tasks.
+ */
+public class FSONSSummaryTask extends NSSummaryTask {
Review Comment:
Let's rename this to `NSSummaryTaskWithFSO`, like what we did with other FSO classes.
It's hard for me to tell `FSONS` is `FSO` and `NS` at a first glance.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/BucketHandler.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+import static org.apache.hadoop.ozone.om.helpers.OzoneFSUtils.removeTrailingSlashIfNeeded;
+
+/**
+ * Abstract class for handling all bucket types.
+ * The abstract methods have different implementation for each bucket type.
+ */
+public abstract class BucketHandler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ BucketHandler.class);
+
+ private final ReconNamespaceSummaryManager reconNamespaceSummaryManager;
+
+ private final ReconOMMetadataManager omMetadataManager;
+
+ private final OzoneStorageContainerManager reconSCM;
+
+ private final ContainerManager containerManager;
+
+ public BucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM) {
+ this.reconNamespaceSummaryManager = reconNamespaceSummaryManager;
+ this.omMetadataManager = omMetadataManager;
+ this.reconSCM = reconSCM;
+ this.containerManager = reconSCM.getContainerManager();
+ }
+
+ public ReconOMMetadataManager getOmMetadataManager() {
+ return omMetadataManager;
+ }
+
+ public ReconNamespaceSummaryManager getReconNamespaceSummaryManager() {
+ return reconNamespaceSummaryManager;
+ }
+
+ public abstract EntityType determineKeyPath(String keyName)
+ throws IOException;
+
+ public abstract long calculateDUUnderObject(long parentId)
+ throws IOException;
+
+ public abstract long handleDirectKeys(long parentId,
+ boolean withReplica, boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ String normalizedPath) throws IOException;
+
+ public abstract long getDirObjectId(String[] names)
+ throws IOException;
+
+ public abstract long getDirObjectId(String[] names, int cutoff)
+ throws IOException;
+
+ public abstract BucketLayout getBucketLayout();
+
+ public abstract OmKeyInfo getKeyInfo(String[] names)
+ throws IOException;
+
+ /**
+ *
+ * @param path
+ * @param nextLevel
+ * @return
+ */
Review Comment:
javadoc is incomplete.
need javadoc for other public methods as well.
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/TestFSONSSummaryTask.java:
##########
@@ -0,0 +1,604 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.utils.db.RDBBatchOperation;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.OzoneManagerServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.ClassRule;
+import org.junit.Assert;
+import org.junit.experimental.runners.Enclosed;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProviderWithFSO;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.initializeNewOmMetadataManager;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeDirToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+
+/**
+ * Test for FSONSSummaryTask.
+ */
+@RunWith(Enclosed.class)
+public final class TestFSONSSummaryTask {
Review Comment:
same here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #3309: HDDS-5504. Support namespace summaries (du, dist & counts) for legacy FS buckets
Posted by GitBox <gi...@apache.org>.
smengcl commented on code in PR #3309:
URL: https://github.com/apache/ozone/pull/3309#discussion_r943661886
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/NSSummary.java:
##########
@@ -93,7 +92,7 @@ public void setChildDir(Set<Long> childDir) {
}
public void setDirName(String dirName) {
- this.dirName = dirName;
+ this.dirName = removeTrailingSlashIfNeeded(dirName);
Review Comment:
ah, I saw it is removed in the latest commit. In this case we can add it back in the actual legacy bucket support patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on pull request #3309: HDDS-5504. Refactor namespace summaries (du, dist & counts) for upcoming legacy FS buckets support
Posted by GitBox <gi...@apache.org>.
xBis7 commented on PR #3309:
URL: https://github.com/apache/ozone/pull/3309#issuecomment-1212199136
@smengcl I changed the PR title and pushed the change. Do you want me to create a new ticket for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on pull request #3309: HDDS-5504. Support namespace summaries (du, dist & counts) for legacy FS buckets
Posted by GitBox <gi...@apache.org>.
xBis7 commented on PR #3309:
URL: https://github.com/apache/ozone/pull/3309#issuecomment-1211293016
@smengcl Thanks for the review! I will make the changes. There are no functional changes in this PR, just the existing code refactored. We kept checking diffs making sure of that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #3309: HDDS-5504. Support namespace summaries (du, dist & counts) for legacy FS buckets
Posted by GitBox <gi...@apache.org>.
smengcl commented on code in PR #3309:
URL: https://github.com/apache/ozone/pull/3309#discussion_r943654609
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/ReconRestServletModule.java:
##########
@@ -125,7 +125,7 @@ private void addFilters(String basePath, Set<String> adminSubPaths) {
if (aclEnabled) {
for (String path: adminSubPaths) {
String adminPath =
- UriBuilder.fromPath(basePath).path(path).build().toString();
+ UriBuilder.fromPath(basePath).path(path + "*").build().toString();
Review Comment:
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #3309: HDDS-5504. Support namespace summaries (du, dist & counts) for legacy FS buckets
Posted by GitBox <gi...@apache.org>.
smengcl commented on code in PR #3309:
URL: https://github.com/apache/ozone/pull/3309#discussion_r943661886
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/NSSummary.java:
##########
@@ -93,7 +92,7 @@ public void setChildDir(Set<Long> childDir) {
}
public void setDirName(String dirName) {
- this.dirName = dirName;
+ this.dirName = removeTrailingSlashIfNeeded(dirName);
Review Comment:
ah, I see it is removed in the latest commit. In this case we can add it back in the actual legacy bucket support patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #3309: HDDS-5504. Support namespace summaries (du, dist & counts) for legacy FS buckets
Posted by GitBox <gi...@apache.org>.
smengcl commented on code in PR #3309:
URL: https://github.com/apache/ozone/pull/3309#discussion_r943657525
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/BucketHandler.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+import static org.apache.hadoop.ozone.om.helpers.OzoneFSUtils.removeTrailingSlashIfNeeded;
+
+/**
+ * Abstract class for handling all bucket types.
+ * The abstract methods have different implementation for each bucket type.
+ */
+public abstract class BucketHandler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ BucketHandler.class);
+
+ private final ReconNamespaceSummaryManager reconNamespaceSummaryManager;
+
+ private final ReconOMMetadataManager omMetadataManager;
+
+ private final OzoneStorageContainerManager reconSCM;
Review Comment:
1) If this won't be used at all, I think we should remove it. We can add it back when we actually need it.
or, 2) you could keep it and add a comment above the line how this is planned to be used later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #3309: HDDS-5504. Support namespace summaries (du, dist & counts) for legacy FS buckets
Posted by GitBox <gi...@apache.org>.
smengcl commented on code in PR #3309:
URL: https://github.com/apache/ozone/pull/3309#discussion_r943659873
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/NSSummary.java:
##########
@@ -93,7 +92,7 @@ public void setChildDir(Set<Long> childDir) {
}
public void setDirName(String dirName) {
- this.dirName = dirName;
+ this.dirName = removeTrailingSlashIfNeeded(dirName);
Review Comment:
Got it. I'm fine with this change. Thanks for the clarification!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on a diff in pull request #3309: HDDS-5504. Support namespace summaries (du, dist & counts) for legacy FS buckets
Posted by GitBox <gi...@apache.org>.
xBis7 commented on code in PR #3309:
URL: https://github.com/apache/ozone/pull/3309#discussion_r943488483
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/NSSummary.java:
##########
@@ -93,7 +92,7 @@ public void setChildDir(Set<Long> childDir) {
}
public void setDirName(String dirName) {
- this.dirName = dirName;
+ this.dirName = removeTrailingSlashIfNeeded(dirName);
Review Comment:
> IIRC if the target is a directory (not a key), removing the trailing slash might cause issue because of the key mismatch when getting value from RocksDB.
@smengcl This is the case for legacy, for FSO there is no difference. We wanted to handle legacy the same way as FSO and that's why we are removing the trailing slash. It's used by every test method, it's not making a difference. I can remove it and include it in the next PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on pull request #3309: HDDS-7121. Refactor namespace summaries (du, dist & counts) for upcoming legacy FS buckets support
Posted by GitBox <gi...@apache.org>.
smengcl commented on PR #3309:
URL: https://github.com/apache/ozone/pull/3309#issuecomment-1212210084
> @smengcl I changed the PR title and pushed the change. Do you want me to create a new ticket for this?
Hi, I see you also changed the jira ID. Although what I thought was to: 1) keep the jira ID and only change the title (as PR links are already associated under HDDS-5504); 2) then use the new jira (HDDS-7121) to implement the actual legacy bucket logic.
Well, as long as things don't break, it is up to you. :) We can manually fix the PR links in the JIRAs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] GeorgeJahad commented on pull request #3309: HDDS-5504. Support namespace summaries (du, dist & counts) for legacy FS buckets
Posted by GitBox <gi...@apache.org>.
GeorgeJahad commented on PR #3309:
URL: https://github.com/apache/ozone/pull/3309#issuecomment-1099507267
@avijayanhwx This is the refactoring draft PR we were telling you about. Please take a look when you have some time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on a diff in pull request #3309: HDDS-5504. Support namespace summaries (du, dist & counts) for legacy FS buckets
Posted by GitBox <gi...@apache.org>.
xBis7 commented on code in PR #3309:
URL: https://github.com/apache/ozone/pull/3309#discussion_r943374958
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/BucketHandler.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+import static org.apache.hadoop.ozone.om.helpers.OzoneFSUtils.removeTrailingSlashIfNeeded;
+
+/**
+ * Abstract class for handling all bucket types.
+ * The abstract methods have different implementation for each bucket type.
+ */
+public abstract class BucketHandler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ BucketHandler.class);
+
+ private final ReconNamespaceSummaryManager reconNamespaceSummaryManager;
+
+ private final ReconOMMetadataManager omMetadataManager;
+
+ private final OzoneStorageContainerManager reconSCM;
Review Comment:
@smengcl Yes, should we remove it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] swagle commented on pull request #3309: HDDS-5504. Refactor namespace summaries (du, dist & counts) for upcoming legacy FS buckets support
Posted by GitBox <gi...@apache.org>.
swagle commented on PR #3309:
URL: https://github.com/apache/ozone/pull/3309#issuecomment-1212569643
Thanks, Siyao for the review! and @xBis7 for the contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] avijayanhwx commented on pull request #3309: HDDS-5504. Support namespace summaries (du, dist & counts) for legacy FS buckets
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on PR #3309:
URL: https://github.com/apache/ozone/pull/3309#issuecomment-1117648746
@xBis7 Approach looks great to me, thanks for taking up this refactoring. Please finish up your patch and move this out of draft whenever possible. I can review it at greater depth.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org