You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ArafatKhan2198 (via GitHub)" <gi...@apache.org> on 2023/02/03 08:33:21 UTC
[GitHub] [ozone] ArafatKhan2198 opened a new pull request, #4245: HDDS-7810 Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets
ArafatKhan2198 opened a new pull request, #4245:
URL: https://github.com/apache/ozone/pull/4245
## What changes were proposed in this pull request?
Recon currently returns 500 when given a du request for an object store bucket, since there is no BucketHandler subclass for OBS buckets that can be returned from BucketHandler#getBucketHandler. This jira is to add an ObjectStoreBucketHandler for OBS buckets. It may end up being similar to the LegacyBucketHandler.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-7810?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel
## How was this patch tested?
Tested it Manually on the UI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1418470502
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling Legacy buckets.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ OBSBucketHandler.class);
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ public EntityType determineKeyPath(String keyName)
+ throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ iterator.seek(key);
+ if (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+ if (dbKey.equals(key)) {
+ return EntityType.KEY;
+ }
+ }
+ return EntityType.UNKNOWN;
+ }
+
+ /**
+ * This method handles disk usage of direct keys.
+ *
+ * @param parentId parent OBS bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * for each direct key's DU
+ * @param listFile if listFile is enabled, append key DU as a children
+ * keys
+ * @param duData the current DU data
+ * @param normalizedPath the normalized path request
+ * @return the total DU of all direct keys
+ * @throws IOException IOE
+ */
+ public long handleDirectKeys(long parentId, boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ String normalizedPath) throws IOException {
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+ long keyDataSizeWithReplica = 0L;
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ String seekPrefix = OM_KEY_PREFIX +
+ vol +
+ OM_KEY_PREFIX +
+ bucket +
+ OM_KEY_PREFIX;
+
+ NSSummary nsSummary = getReconNamespaceSummaryManager()
+ .getNSSummary(parentId);
+ // Handle the case of an empty bucket.
+ if (nsSummary == null) {
+ return 0;
+ }
+
+ iterator.seek(seekPrefix);
+
+ while (iterator.hasNext()) {
+ // KeyName : OmKeyInfo-Object
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+
+ // Exit loop if the key doesn't match the seekPrefix.
+ if (!dbKey.startsWith(seekPrefix)) {
+ break;
+ }
+
+ OmKeyInfo keyInfo = kv.getValue();
+ if (keyInfo != null) {
+ DUResponse.DiskUsage diskUsage = new DUResponse.DiskUsage();
+ String objectName = keyInfo.getKeyName();
+ diskUsage.setSubpath(objectName);
+ diskUsage.setKey(true);
+ diskUsage.setSize(keyInfo.getDataSize());
+
+ if (withReplica) {
+ long keyDU = keyInfo.getReplicatedSize();
+ keyDataSizeWithReplica += keyDU;
+ diskUsage.setSizeWithReplica(keyDU);
+ }
+ // List all the keys for the OBS bucket if requested.
+ if (listFile) {
+ duData.add(diskUsage);
+ }
+ }
+ }
+
+ return keyDataSizeWithReplica;
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ public long calculateDUUnderObject(long parentId)
+ throws IOException {
+ return Long.parseLong(null);
Review Comment:
Now since we are not proceeding with prefix filtering for OBS keys, we no longer need to use `calculateDUUnderObject` as we will be treating everything inside the bucket as keys hence `handleDirectKeys` method should be enough.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling Legacy buckets.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ OBSBucketHandler.class);
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ public EntityType determineKeyPath(String keyName)
Review Comment:
Done!
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling Legacy buckets.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ OBSBucketHandler.class);
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ public EntityType determineKeyPath(String keyName)
+ throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ iterator.seek(key);
+ if (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+ if (dbKey.equals(key)) {
+ return EntityType.KEY;
+ }
+ }
+ return EntityType.UNKNOWN;
+ }
+
+ /**
+ * This method handles disk usage of direct keys.
+ *
+ * @param parentId parent OBS bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * for each direct key's DU
+ * @param listFile if listFile is enabled, append key DU as a children
+ * keys
+ * @param duData the current DU data
+ * @param normalizedPath the normalized path request
+ * @return the total DU of all direct keys
+ * @throws IOException IOE
+ */
+ public long handleDirectKeys(long parentId, boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ String normalizedPath) throws IOException {
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+ long keyDataSizeWithReplica = 0L;
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ String seekPrefix = OM_KEY_PREFIX +
+ vol +
+ OM_KEY_PREFIX +
+ bucket +
+ OM_KEY_PREFIX;
+
+ NSSummary nsSummary = getReconNamespaceSummaryManager()
+ .getNSSummary(parentId);
+ // Handle the case of an empty bucket.
+ if (nsSummary == null) {
+ return 0;
+ }
+
+ iterator.seek(seekPrefix);
+
+ while (iterator.hasNext()) {
+ // KeyName : OmKeyInfo-Object
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+
+ // Exit loop if the key doesn't match the seekPrefix.
+ if (!dbKey.startsWith(seekPrefix)) {
+ break;
+ }
+
+ OmKeyInfo keyInfo = kv.getValue();
+ if (keyInfo != null) {
+ DUResponse.DiskUsage diskUsage = new DUResponse.DiskUsage();
+ String objectName = keyInfo.getKeyName();
+ diskUsage.setSubpath(objectName);
+ diskUsage.setKey(true);
+ diskUsage.setSize(keyInfo.getDataSize());
+
+ if (withReplica) {
+ long keyDU = keyInfo.getReplicatedSize();
+ keyDataSizeWithReplica += keyDU;
+ diskUsage.setSizeWithReplica(keyDU);
+ }
+ // List all the keys for the OBS bucket if requested.
+ if (listFile) {
+ duData.add(diskUsage);
+ }
+ }
+ }
+
+ return keyDataSizeWithReplica;
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ public long calculateDUUnderObject(long parentId)
+ throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ public long getDirObjectId(String[] names) throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ public long getDirObjectId(String[] names, int cutoff) throws IOException {
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "dombizita (via GitHub)" <gi...@apache.org>.
dombizita commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1489513481
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTaskWithOBS.java:
##########
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.WithParentObjectId;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+
+import static org.apache.hadoop.ozone.om.OmMetadataManagerImpl.KEY_TABLE;
+
+
+/**
+ * Class for handling OBS specific tasks.
+ */
+public class NSSummaryTaskWithOBS extends NSSummaryTaskDbEventHandler {
+
+ private static final BucketLayout BUCKET_LAYOUT = BucketLayout.OBJECT_STORE;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(NSSummaryTaskWithOBS.class);
+
+
+ public NSSummaryTaskWithOBS(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager reconOMMetadataManager,
+ OzoneConfiguration ozoneConfiguration) {
+ super(reconNamespaceSummaryManager,
+ reconOMMetadataManager, ozoneConfiguration);
+ }
+
+
+ public boolean reprocessWithOBS(OMMetadataManager omMetadataManager) {
+ Map<Long, NSSummary> nsSummaryMap = new HashMap<>();
+
+ try {
+ Table<String, OmKeyInfo> keyTable =
+ omMetadataManager.getKeyTable(BUCKET_LAYOUT);
+
+ try (TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ keyTableIter = keyTable.iterator()) {
+
+ while (keyTableIter.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = keyTableIter.next();
+ OmKeyInfo keyInfo = kv.getValue();
+
+ // KeyTable entries belong to both Legacy and OBS buckets.
+ // Check bucket layout and if it's anything other than OBS,
+ // continue to the next iteration.
+ String volumeName = keyInfo.getVolumeName();
+ String bucketName = keyInfo.getBucketName();
+ String bucketDBKey = omMetadataManager
+ .getBucketKey(volumeName, bucketName);
+ // Get bucket info from bucket table
+ OmBucketInfo omBucketInfo = omMetadataManager
+ .getBucketTable().getSkipCache(bucketDBKey);
+
+ if (omBucketInfo.getBucketLayout() != BucketLayout.OBJECT_STORE) {
Review Comment:
We could use the `BUCKET_LAYOUT` constant you created.
```suggestion
if (omBucketInfo.getBucketLayout() != BUCKET_LAYOUT) {
```
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/TestNSSummaryTaskWithOBS.java:
##########
@@ -0,0 +1,556 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.RDBBatchOperation;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.NSSummaryEndpoint;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Nested;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import javax.ws.rs.core.Response;
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.*;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeDirToOm;
+import static org.mockito.Mockito.mock;
+
+public class TestNSSummaryTaskWithOBS {
+ private static ReconNamespaceSummaryManager reconNamespaceSummaryManager;
+ private static OMMetadataManager omMetadataManager;
+ private static ReconOMMetadataManager reconOMMetadataManager;
+ private static NSSummaryTaskWithOBS nSSummaryTaskWithOBS;
+ private static OzoneConfiguration omConfiguration;
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String KEY_ONE = "key1";
+ private static final String KEY_TWO = "key2";
+ private static final String KEY_THREE = "dir1/dir2/key3";
+ private static final String KEY_FOUR = "key4///////////";
+ private static final String KEY_FIVE = "//////////";
+ private static final String KEY_SIX = "key6";
+ private static final String KEY_SEVEN = "key7";
+
+ private static final String TEST_USER = "TestUser";
+
+ private static final long PARENT_OBJECT_ID_ZERO = 0L;
+ private static final long VOL_OBJECT_ID = 0L;
+ private static final long BUCKET_ONE_OBJECT_ID = 1L;
+ private static final long BUCKET_TWO_OBJECT_ID = 2L;
+ private static final long KEY_ONE_OBJECT_ID = 3L;
+ private static final long KEY_TWO_OBJECT_ID = 5L;
+ private static final long KEY_FOUR_OBJECT_ID = 6L;
+ private static final long KEY_THREE_OBJECT_ID = 8L;
+ private static final long KEY_FIVE_OBJECT_ID = 9L;
+ private static final long KEY_SIX_OBJECT_ID = 10L;
+ private static final long KEY_SEVEN_OBJECT_ID = 11L;
+
+
+ private static final long KEY_ONE_SIZE = 500L;
+ private static final long KEY_TWO_OLD_SIZE = 1025L;
+ private static final long KEY_TWO_UPDATE_SIZE = 1023L;
+ private static final long KEY_THREE_SIZE =
+ ReconConstants.MAX_FILE_SIZE_UPPER_BOUND - 100L;
+ private static final long KEY_FOUR_SIZE = 2050L;
+ private static final long KEY_FIVE_SIZE = 100L;
+ private static final long KEY_SIX_SIZE = 6000L;
+ private static final long KEY_SEVEN_SIZE = 7000L;
+
+ private static Set<Long> bucketOneAns = new HashSet<>();
+ private static Set<Long> bucketTwoAns = new HashSet<>();
+ private static Set<Long> dirOneAns = new HashSet<>();
+
+ private TestNSSummaryTaskWithOBS() {
+ }
+
+ @BeforeAll
+ public static void setUp(@TempDir File tmpDir) throws Exception {
+ initializeNewOmMetadataManager(new File(tmpDir, "om"));
+ OzoneManagerServiceProviderImpl ozoneManagerServiceProvider =
+ getMockOzoneManagerServiceProvider();
+ reconOMMetadataManager = getTestReconOmMetadataManager(omMetadataManager,
+ new File(tmpDir, "recon"));
+
+ ReconTestInjector reconTestInjector =
+ new ReconTestInjector.Builder(tmpDir)
+ .withReconOm(reconOMMetadataManager)
+ .withOmServiceProvider(ozoneManagerServiceProvider)
+ .withReconSqlDb()
+ .withContainerDB()
+ .build();
+ reconNamespaceSummaryManager =
+ reconTestInjector.getInstance(ReconNamespaceSummaryManager.class);
+
+ NSSummary nonExistentSummary =
+ reconNamespaceSummaryManager.getNSSummary(BUCKET_ONE_OBJECT_ID);
+ Assertions.assertNull(nonExistentSummary);
Review Comment:
I don't see that you changed the imports for the assertions, could you check it?
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTaskWithOBS.java:
##########
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.WithParentObjectId;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+
+import static org.apache.hadoop.ozone.om.OmMetadataManagerImpl.KEY_TABLE;
+
+
+/**
+ * Class for handling OBS specific tasks.
+ */
+public class NSSummaryTaskWithOBS extends NSSummaryTaskDbEventHandler {
+
+ private static final BucketLayout BUCKET_LAYOUT = BucketLayout.OBJECT_STORE;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(NSSummaryTaskWithOBS.class);
+
+
+ public NSSummaryTaskWithOBS(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager reconOMMetadataManager,
+ OzoneConfiguration ozoneConfiguration) {
+ super(reconNamespaceSummaryManager,
+ reconOMMetadataManager, ozoneConfiguration);
+ }
+
+
+ public boolean reprocessWithOBS(OMMetadataManager omMetadataManager) {
+ Map<Long, NSSummary> nsSummaryMap = new HashMap<>();
+
+ try {
+ Table<String, OmKeyInfo> keyTable =
+ omMetadataManager.getKeyTable(BUCKET_LAYOUT);
+
+ try (TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ keyTableIter = keyTable.iterator()) {
+
+ while (keyTableIter.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = keyTableIter.next();
+ OmKeyInfo keyInfo = kv.getValue();
+
+ // KeyTable entries belong to both Legacy and OBS buckets.
+ // Check bucket layout and if it's anything other than OBS,
+ // continue to the next iteration.
+ String volumeName = keyInfo.getVolumeName();
+ String bucketName = keyInfo.getBucketName();
+ String bucketDBKey = omMetadataManager
+ .getBucketKey(volumeName, bucketName);
+ // Get bucket info from bucket table
+ OmBucketInfo omBucketInfo = omMetadataManager
+ .getBucketTable().getSkipCache(bucketDBKey);
+
+ if (omBucketInfo.getBucketLayout() != BucketLayout.OBJECT_STORE) {
+ continue;
+ }
+
+ setKeyParentID(keyInfo);
+
+ handlePutKeyEvent(keyInfo, nsSummaryMap);
+ if (!checkAndCallFlushToDB(nsSummaryMap)) {
+ return false;
+ }
+ }
+ }
+ } catch (IOException ioEx) {
+ LOG.error("Unable to reprocess Namespace Summary data in Recon DB. ",
+ ioEx);
+ return false;
+ }
+
+ // flush and commit left out entries at end
+ if (!flushAndCommitNSToDB(nsSummaryMap)) {
+ return false;
+ }
+ LOG.info("Completed a reprocess run of NSSummaryTaskWithOBS");
+ return true;
+ }
+
+ public boolean processWithOBS(OMUpdateEventBatch events) {
+ Iterator<OMDBUpdateEvent> eventIterator = events.getIterator();
+ Map<Long, NSSummary> nsSummaryMap = new HashMap<>();
+
+ while (eventIterator.hasNext()) {
+ OMDBUpdateEvent<String, ? extends WithParentObjectId> omdbUpdateEvent =
+ eventIterator.next();
+ OMDBUpdateEvent.OMDBUpdateAction action = omdbUpdateEvent.getAction();
+
+ // We only process updates on OM's KeyTable
+ String table = omdbUpdateEvent.getTable();
+ boolean updateOnKeyTable = table.equals(KEY_TABLE);
+ if (!updateOnKeyTable) {
+ continue;
+ }
+
+ String updatedKey = omdbUpdateEvent.getKey();
+
+ try {
+ OMDBUpdateEvent<String, ?> keyTableUpdateEvent = omdbUpdateEvent;
+ Object value = keyTableUpdateEvent.getValue();
+ Object oldValue = keyTableUpdateEvent.getOldValue();
+ if (!(value instanceof OmKeyInfo)) {
+ LOG.warn("Unexpected value type {} for key {}. Skipping processing.",
+ value.getClass().getName(), updatedKey);
+ continue;
+ }
+ OmKeyInfo updatedKeyInfo = (OmKeyInfo) value;
+ OmKeyInfo oldKeyInfo = (OmKeyInfo) oldValue;
+
+ // KeyTable entries belong to both OBS and Legacy buckets.
+ // Check bucket layout and if it's anything other than OBS,
+ // continue to the next iteration.
+ String volumeName = updatedKeyInfo.getVolumeName();
+ String bucketName = updatedKeyInfo.getBucketName();
+ String bucketDBKey =
+ getReconOMMetadataManager().getBucketKey(volumeName, bucketName);
+ // Get bucket info from bucket table
+ OmBucketInfo omBucketInfo = getReconOMMetadataManager().getBucketTable()
+ .getSkipCache(bucketDBKey);
+
+ if (omBucketInfo.getBucketLayout() != BucketLayout.OBJECT_STORE) {
Review Comment:
```suggestion
if (omBucketInfo.getBucketLayout() != BUCKET_LAYOUT) {
```
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTaskWithOBS.java:
##########
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.WithParentObjectId;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+
+import static org.apache.hadoop.ozone.om.OmMetadataManagerImpl.KEY_TABLE;
+
+
+/**
+ * Class for handling OBS specific tasks.
+ */
+public class NSSummaryTaskWithOBS extends NSSummaryTaskDbEventHandler {
+
+ private static final BucketLayout BUCKET_LAYOUT = BucketLayout.OBJECT_STORE;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(NSSummaryTaskWithOBS.class);
+
+
+ public NSSummaryTaskWithOBS(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager reconOMMetadataManager,
+ OzoneConfiguration ozoneConfiguration) {
+ super(reconNamespaceSummaryManager,
+ reconOMMetadataManager, ozoneConfiguration);
+ }
+
+
+ public boolean reprocessWithOBS(OMMetadataManager omMetadataManager) {
+ Map<Long, NSSummary> nsSummaryMap = new HashMap<>();
+
+ try {
+ Table<String, OmKeyInfo> keyTable =
+ omMetadataManager.getKeyTable(BUCKET_LAYOUT);
+
+ try (TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ keyTableIter = keyTable.iterator()) {
+
+ while (keyTableIter.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = keyTableIter.next();
+ OmKeyInfo keyInfo = kv.getValue();
+
+ // KeyTable entries belong to both Legacy and OBS buckets.
+ // Check bucket layout and if it's anything other than OBS,
+ // continue to the next iteration.
+ String volumeName = keyInfo.getVolumeName();
+ String bucketName = keyInfo.getBucketName();
+ String bucketDBKey = omMetadataManager
+ .getBucketKey(volumeName, bucketName);
+ // Get bucket info from bucket table
+ OmBucketInfo omBucketInfo = omMetadataManager
+ .getBucketTable().getSkipCache(bucketDBKey);
+
+ if (omBucketInfo.getBucketLayout() != BucketLayout.OBJECT_STORE) {
+ continue;
+ }
+
+ setKeyParentID(keyInfo);
+
+ handlePutKeyEvent(keyInfo, nsSummaryMap);
+ if (!checkAndCallFlushToDB(nsSummaryMap)) {
+ return false;
+ }
+ }
+ }
+ } catch (IOException ioEx) {
+ LOG.error("Unable to reprocess Namespace Summary data in Recon DB. ",
+ ioEx);
+ return false;
+ }
+
+ // flush and commit left out entries at end
+ if (!flushAndCommitNSToDB(nsSummaryMap)) {
+ return false;
+ }
+ LOG.info("Completed a reprocess run of NSSummaryTaskWithOBS");
+ return true;
+ }
+
+ public boolean processWithOBS(OMUpdateEventBatch events) {
+ Iterator<OMDBUpdateEvent> eventIterator = events.getIterator();
+ Map<Long, NSSummary> nsSummaryMap = new HashMap<>();
+
+ while (eventIterator.hasNext()) {
+ OMDBUpdateEvent<String, ? extends WithParentObjectId> omdbUpdateEvent =
+ eventIterator.next();
+ OMDBUpdateEvent.OMDBUpdateAction action = omdbUpdateEvent.getAction();
+
+ // We only process updates on OM's KeyTable
+ String table = omdbUpdateEvent.getTable();
+ boolean updateOnKeyTable = table.equals(KEY_TABLE);
+ if (!updateOnKeyTable) {
+ continue;
+ }
+
+ String updatedKey = omdbUpdateEvent.getKey();
+
+ try {
+ OMDBUpdateEvent<String, ?> keyTableUpdateEvent = omdbUpdateEvent;
+ Object value = keyTableUpdateEvent.getValue();
+ Object oldValue = keyTableUpdateEvent.getOldValue();
+ if (!(value instanceof OmKeyInfo)) {
+ LOG.warn("Unexpected value type {} for key {}. Skipping processing.",
+ value.getClass().getName(), updatedKey);
+ continue;
+ }
+ OmKeyInfo updatedKeyInfo = (OmKeyInfo) value;
+ OmKeyInfo oldKeyInfo = (OmKeyInfo) oldValue;
+
+ // KeyTable entries belong to both OBS and Legacy buckets.
+ // Check bucket layout and if it's anything other than OBS,
+ // continue to the next iteration.
+ String volumeName = updatedKeyInfo.getVolumeName();
+ String bucketName = updatedKeyInfo.getBucketName();
+ String bucketDBKey =
+ getReconOMMetadataManager().getBucketKey(volumeName, bucketName);
+ // Get bucket info from bucket table
+ OmBucketInfo omBucketInfo = getReconOMMetadataManager().getBucketTable()
+ .getSkipCache(bucketDBKey);
+
+ if (omBucketInfo.getBucketLayout() != BucketLayout.OBJECT_STORE) {
+ continue;
+ }
+
+ setKeyParentID(updatedKeyInfo);
+
+ switch (action) {
+ case PUT:
+ handlePutKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+ case DELETE:
+ handleDeleteKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+ case UPDATE:
+ if (oldKeyInfo != null) {
+ // delete first, then put
+ setKeyParentID(oldKeyInfo);
+ handleDeleteKeyEvent(oldKeyInfo, nsSummaryMap);
+ } else {
+ LOG.warn("Update event does not have the old keyInfo for {}.",
+ updatedKey);
+ }
+ handlePutKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+ default:
+ LOG.debug("Skipping DB update event: {}", action);
+ }
+
+ if (!checkAndCallFlushToDB(nsSummaryMap)) {
+ return false;
+ }
+ } catch (IOException ioEx) {
+ LOG.error("Unable to process Namespace Summary data in Recon DB. ",
+ ioEx);
+ return false;
+ }
+ if (!checkAndCallFlushToDB(nsSummaryMap)) {
+ return false;
+ }
+ }
+
+ // Flush and commit left-out entries at the end
+ if (!flushAndCommitNSToDB(nsSummaryMap)) {
+ return false;
+ }
+
+ LOG.info("Completed a process run of NSSummaryTaskWithOBS");
+ return true;
+ }
+
+
+ /**
+ * KeyTable entries don't have the parentId set.
+ * In order to reuse the existing methods that rely on
+ * the parentId, we have to set it explicitly.
+ * Note: For an OBS key, the parentId will always correspond to the ID of the
+ * OBS bucket in which it is located.
+ *
+ * @param keyInfo
+ * @throws IOException
+ */
+ private void setKeyParentID(OmKeyInfo keyInfo)
+ throws IOException {
+ String bucketKey = getReconOMMetadataManager()
+ .getBucketKey(keyInfo.getVolumeName(), keyInfo.getBucketName());
+ OmBucketInfo parentBucketInfo =
+ getReconOMMetadataManager().getBucketTable().getSkipCache(bucketKey);
+
+ if (parentBucketInfo != null) {
+ keyInfo.setParentObjectID(parentBucketInfo.getObjectID());
+ } else {
+ throw new IOException("ParentKeyInfo for " +
+ "NSSummaryTaskWithLegacy is null");
Review Comment:
nit
```suggestion
"NSSummaryTaskWithOBS is null");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] GeorgeJahad commented on pull request #4245: HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets.
Posted by "GeorgeJahad (via GitHub)" <gi...@apache.org>.
GeorgeJahad commented on PR #4245:
URL: https://github.com/apache/ozone/pull/4245#issuecomment-1428857167
tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1469211469
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling OBS buckets NameSpaceSummaries.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ @Override
+ public EntityType determineKeyPath(String keyName) throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ try (
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator()) {
+ iterator.seek(key);
+ if (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+ if (dbKey.equals(key)) {
+ return EntityType.KEY;
+ }
+ }
+ }
+ return EntityType.UNKNOWN;
+ }
+
+ /**
+ * This method handles disk usage of direct keys.
+ *
+ * @param parentId parent OBS bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * for each direct key's DU
+ * @param listFile if listFile is enabled, append key DU as a children
+ * keys
+ * @param duData the current DU data
+ * @param normalizedPath the normalized path request
+ * @return the total DU of all direct keys
+ * @throws IOException IOE
+ */
+ @Override
+ public long handleDirectKeys(long parentId, boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ String normalizedPath) throws IOException {
+
+ NSSummary nsSummary = getReconNamespaceSummaryManager()
+ .getNSSummary(parentId);
+ // Handle the case of an empty bucket.
+ if (nsSummary == null) {
+ return 0;
+ }
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+ long keyDataSizeWithReplica = 0L;
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
Review Comment:
This one needs to be closed, too.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling OBS buckets NameSpaceSummaries.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ @Override
+ public EntityType determineKeyPath(String keyName) throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ try (
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator()) {
+ iterator.seek(key);
+ if (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+ if (dbKey.equals(key)) {
+ return EntityType.KEY;
+ }
+ }
+ }
+ return EntityType.UNKNOWN;
+ }
+
+ /**
+ * This method handles disk usage of direct keys.
+ *
+ * @param parentId parent OBS bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * for each direct key's DU
+ * @param listFile if listFile is enabled, append key DU as a children
+ * keys
+ * @param duData the current DU data
+ * @param normalizedPath the normalized path request
+ * @return the total DU of all direct keys
+ * @throws IOException IOE
+ */
+ @Override
+ public long handleDirectKeys(long parentId, boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ String normalizedPath) throws IOException {
+
+ NSSummary nsSummary = getReconNamespaceSummaryManager()
+ .getNSSummary(parentId);
+ // Handle the case of an empty bucket.
+ if (nsSummary == null) {
+ return 0;
+ }
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+ long keyDataSizeWithReplica = 0L;
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ String seekPrefix = OM_KEY_PREFIX +
+ vol +
+ OM_KEY_PREFIX +
+ bucket +
+ OM_KEY_PREFIX;
+
+ iterator.seek(seekPrefix);
+
+ while (iterator.hasNext()) {
+ // KeyName : OmKeyInfo-Object
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+
+ // Exit loop if the key doesn't match the seekPrefix.
+ if (!dbKey.startsWith(seekPrefix)) {
+ break;
+ }
+
+ OmKeyInfo keyInfo = kv.getValue();
+ if (keyInfo != null) {
+ DUResponse.DiskUsage diskUsage = new DUResponse.DiskUsage();
+ String objectName = keyInfo.getKeyName();
+ diskUsage.setSubpath(objectName);
+ diskUsage.setKey(true);
+ diskUsage.setSize(keyInfo.getDataSize());
+
+ if (withReplica) {
+ long keyDU = keyInfo.getReplicatedSize();
+ keyDataSizeWithReplica += keyDU;
+ diskUsage.setSizeWithReplica(keyDU);
+ }
+ // List all the keys for the OBS bucket if requested.
+ if (listFile) {
+ duData.add(diskUsage);
+ }
+ }
+ }
+
+ return keyDataSizeWithReplica;
+ }
+
+ /**
+ * Object stores do not support directories.
+ *
+ * @return UnsupportedOperationException
Review Comment:
nit: exception is thrown here, not returned
```suggestion
* @throws UnsupportedOperationException always
```
(Also for other similar methods.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1469224019
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling OBS buckets NameSpaceSummaries.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ @Override
+ public EntityType determineKeyPath(String keyName) throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ try (
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator()) {
+ iterator.seek(key);
+ if (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+ if (dbKey.equals(key)) {
+ return EntityType.KEY;
+ }
+ }
+ }
+ return EntityType.UNKNOWN;
+ }
+
+ /**
+ * This method handles disk usage of direct keys.
+ *
+ * @param parentId parent OBS bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * for each direct key's DU
+ * @param listFile if listFile is enabled, append key DU as a children
+ * keys
+ * @param duData the current DU data
+ * @param normalizedPath the normalized path request
+ * @return the total DU of all direct keys
+ * @throws IOException IOE
+ */
+ @Override
+ public long handleDirectKeys(long parentId, boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ String normalizedPath) throws IOException {
+
+ NSSummary nsSummary = getReconNamespaceSummaryManager()
+ .getNSSummary(parentId);
+ // Handle the case of an empty bucket.
+ if (nsSummary == null) {
+ return 0;
+ }
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+ long keyDataSizeWithReplica = 0L;
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
Review Comment:
Thanks for pointing it out.
I have made the changes.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling OBS buckets NameSpaceSummaries.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ @Override
+ public EntityType determineKeyPath(String keyName) throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ try (
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator()) {
+ iterator.seek(key);
+ if (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+ if (dbKey.equals(key)) {
+ return EntityType.KEY;
+ }
+ }
+ }
+ return EntityType.UNKNOWN;
+ }
+
+ /**
+ * This method handles disk usage of direct keys.
+ *
+ * @param parentId parent OBS bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * for each direct key's DU
+ * @param listFile if listFile is enabled, append key DU as a children
+ * keys
+ * @param duData the current DU data
+ * @param normalizedPath the normalized path request
+ * @return the total DU of all direct keys
+ * @throws IOException IOE
+ */
+ @Override
+ public long handleDirectKeys(long parentId, boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ String normalizedPath) throws IOException {
+
+ NSSummary nsSummary = getReconNamespaceSummaryManager()
+ .getNSSummary(parentId);
+ // Handle the case of an empty bucket.
+ if (nsSummary == null) {
+ return 0;
+ }
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+ long keyDataSizeWithReplica = 0L;
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ String seekPrefix = OM_KEY_PREFIX +
+ vol +
+ OM_KEY_PREFIX +
+ bucket +
+ OM_KEY_PREFIX;
+
+ iterator.seek(seekPrefix);
+
+ while (iterator.hasNext()) {
+ // KeyName : OmKeyInfo-Object
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+
+ // Exit loop if the key doesn't match the seekPrefix.
+ if (!dbKey.startsWith(seekPrefix)) {
+ break;
+ }
+
+ OmKeyInfo keyInfo = kv.getValue();
+ if (keyInfo != null) {
+ DUResponse.DiskUsage diskUsage = new DUResponse.DiskUsage();
+ String objectName = keyInfo.getKeyName();
+ diskUsage.setSubpath(objectName);
+ diskUsage.setKey(true);
+ diskUsage.setSize(keyInfo.getDataSize());
+
+ if (withReplica) {
+ long keyDU = keyInfo.getReplicatedSize();
+ keyDataSizeWithReplica += keyDU;
+ diskUsage.setSizeWithReplica(keyDU);
+ }
+ // List all the keys for the OBS bucket if requested.
+ if (listFile) {
+ duData.add(diskUsage);
+ }
+ }
+ }
+
+ return keyDataSizeWithReplica;
+ }
+
+ /**
+ * Object stores do not support directories.
+ *
+ * @return UnsupportedOperationException
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1468513986
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling OBS buckets NameSpaceSummaries.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ @Override
+ public EntityType determineKeyPath(String keyName)
+ throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ iterator.seek(key);
+ if (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+ if (dbKey.equals(key)) {
+ return EntityType.KEY;
+ }
+ }
+ return EntityType.UNKNOWN;
+ }
+
+ /**
+ * This method handles disk usage of direct keys.
+ *
+ * @param parentId parent OBS bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * for each direct key's DU
+ * @param listFile if listFile is enabled, append key DU as a children
+ * keys
+ * @param duData the current DU data
+ * @param normalizedPath the normalized path request
+ * @return the total DU of all direct keys
+ * @throws IOException IOE
+ */
+ @Override
+ public long handleDirectKeys(long parentId, boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ String normalizedPath) throws IOException {
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+ long keyDataSizeWithReplica = 0L;
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ String seekPrefix = OM_KEY_PREFIX +
+ vol +
+ OM_KEY_PREFIX +
+ bucket +
+ OM_KEY_PREFIX;
+
+ NSSummary nsSummary = getReconNamespaceSummaryManager()
+ .getNSSummary(parentId);
+ // Handle the case of an empty bucket.
+ if (nsSummary == null) {
+ return 0;
+ }
Review Comment:
Check at the start of the method.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling OBS buckets NameSpaceSummaries.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ @Override
+ public EntityType determineKeyPath(String keyName)
+ throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ iterator.seek(key);
+ if (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+ if (dbKey.equals(key)) {
+ return EntityType.KEY;
+ }
+ }
+ return EntityType.UNKNOWN;
+ }
+
+ /**
+ * This method handles disk usage of direct keys.
+ *
+ * @param parentId parent OBS bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * for each direct key's DU
+ * @param listFile if listFile is enabled, append key DU as a children
+ * keys
+ * @param duData the current DU data
+ * @param normalizedPath the normalized path request
+ * @return the total DU of all direct keys
+ * @throws IOException IOE
+ */
+ @Override
+ public long handleDirectKeys(long parentId, boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ String normalizedPath) throws IOException {
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+ long keyDataSizeWithReplica = 0L;
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ String seekPrefix = OM_KEY_PREFIX +
+ vol +
+ OM_KEY_PREFIX +
+ bucket +
+ OM_KEY_PREFIX;
+
+ NSSummary nsSummary = getReconNamespaceSummaryManager()
+ .getNSSummary(parentId);
+ // Handle the case of an empty bucket.
+ if (nsSummary == null) {
+ return 0;
+ }
+
+ iterator.seek(seekPrefix);
+
+ while (iterator.hasNext()) {
+ // KeyName : OmKeyInfo-Object
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+
+ // Exit loop if the key doesn't match the seekPrefix.
+ if (!dbKey.startsWith(seekPrefix)) {
+ break;
+ }
+
+ OmKeyInfo keyInfo = kv.getValue();
+ if (keyInfo != null) {
+ DUResponse.DiskUsage diskUsage = new DUResponse.DiskUsage();
+ String objectName = keyInfo.getKeyName();
+ diskUsage.setSubpath(objectName);
+ diskUsage.setKey(true);
+ diskUsage.setSize(keyInfo.getDataSize());
+
+ if (withReplica) {
+ long keyDU = keyInfo.getReplicatedSize();
+ keyDataSizeWithReplica += keyDU;
+ diskUsage.setSizeWithReplica(keyDU);
+ }
+ // List all the keys for the OBS bucket if requested.
+ if (listFile) {
+ duData.add(diskUsage);
+ }
+ }
+ }
+
+ return keyDataSizeWithReplica;
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public long calculateDUUnderObject(long parentId)
+ throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public long getDirObjectId(String[] names) throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public long getDirObjectId(String[] names, int cutoff) throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Returns the keyInfo object from the KEY table.
+ * @return OmKeyInfo
+ */
+ @Override
+ public OmKeyInfo getKeyInfo(String[] names) throws IOException {
+ String ozoneKey = OM_KEY_PREFIX;
+ ozoneKey += String.join(OM_KEY_PREFIX, names);
+
+ OmKeyInfo keyInfo = getKeyTable().getSkipCache(ozoneKey);
+ return keyInfo;
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public OmDirectoryInfo getDirInfo(String[] names) throws IOException {
+ return null;
+ }
+
+ public Table<String, OmKeyInfo> getKeyTable() {
+ Table keyTable =
+ getOmMetadataManager().getKeyTable(getBucketLayout());
+ return keyTable;
Review Comment:
nit:
```suggestion
return getOmMetadataManager().getKeyTable(getBucketLayout());
```
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTask.java:
##########
@@ -137,17 +151,23 @@ public Pair<String, Boolean> reprocess(OMMetadataManager omMetadataManager) {
}
}
} catch (InterruptedException ex) {
- LOG.error("Error while reprocessing NSSummary " +
- "table in Recon DB. ", ex);
+ LOG.error("Error while reprocessing NSSummary table in Recon DB.", ex);
return new ImmutablePair<>(getTaskName(), false);
} catch (ExecutionException ex2) {
- LOG.error("Error while reprocessing NSSummary " +
- "table in Recon DB. ", ex2);
+ LOG.error("Error while reprocessing NSSummary table in Recon DB.", ex2);
return new ImmutablePair<>(getTaskName(), false);
} finally {
executorService.shutdown();
+
+ long endTime = System.nanoTime(); // Record end time
+ long durationInMillis = (endTime - startTime) / 1_000_000; // Convert to milliseconds
Review Comment:
Use `TimeUnit.NANOSECONDS.toMillis()` for conversion.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling OBS buckets NameSpaceSummaries.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ @Override
+ public EntityType determineKeyPath(String keyName)
+ throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
Review Comment:
`TableIterator` needs to be closed. Please use in `try-with-resources` to ensure that.
Also in all other similar code.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling OBS buckets NameSpaceSummaries.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ @Override
+ public EntityType determineKeyPath(String keyName)
+ throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ iterator.seek(key);
+ if (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+ if (dbKey.equals(key)) {
+ return EntityType.KEY;
+ }
+ }
+ return EntityType.UNKNOWN;
+ }
+
+ /**
+ * This method handles disk usage of direct keys.
+ *
+ * @param parentId parent OBS bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * for each direct key's DU
+ * @param listFile if listFile is enabled, append key DU as a children
+ * keys
+ * @param duData the current DU data
+ * @param normalizedPath the normalized path request
+ * @return the total DU of all direct keys
+ * @throws IOException IOE
+ */
+ @Override
+ public long handleDirectKeys(long parentId, boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ String normalizedPath) throws IOException {
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+ long keyDataSizeWithReplica = 0L;
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ String seekPrefix = OM_KEY_PREFIX +
+ vol +
+ OM_KEY_PREFIX +
+ bucket +
+ OM_KEY_PREFIX;
+
+ NSSummary nsSummary = getReconNamespaceSummaryManager()
+ .getNSSummary(parentId);
+ // Handle the case of an empty bucket.
+ if (nsSummary == null) {
+ return 0;
+ }
+
+ iterator.seek(seekPrefix);
+
+ while (iterator.hasNext()) {
+ // KeyName : OmKeyInfo-Object
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+
+ // Exit loop if the key doesn't match the seekPrefix.
+ if (!dbKey.startsWith(seekPrefix)) {
+ break;
+ }
+
+ OmKeyInfo keyInfo = kv.getValue();
+ if (keyInfo != null) {
+ DUResponse.DiskUsage diskUsage = new DUResponse.DiskUsage();
+ String objectName = keyInfo.getKeyName();
+ diskUsage.setSubpath(objectName);
+ diskUsage.setKey(true);
+ diskUsage.setSize(keyInfo.getDataSize());
+
+ if (withReplica) {
+ long keyDU = keyInfo.getReplicatedSize();
+ keyDataSizeWithReplica += keyDU;
+ diskUsage.setSizeWithReplica(keyDU);
+ }
+ // List all the keys for the OBS bucket if requested.
+ if (listFile) {
+ duData.add(diskUsage);
+ }
+ }
+ }
+
+ return keyDataSizeWithReplica;
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public long calculateDUUnderObject(long parentId)
+ throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public long getDirObjectId(String[] names) throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public long getDirObjectId(String[] names, int cutoff) throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Returns the keyInfo object from the KEY table.
+ * @return OmKeyInfo
+ */
+ @Override
+ public OmKeyInfo getKeyInfo(String[] names) throws IOException {
+ String ozoneKey = OM_KEY_PREFIX;
+ ozoneKey += String.join(OM_KEY_PREFIX, names);
+
+ OmKeyInfo keyInfo = getKeyTable().getSkipCache(ozoneKey);
+ return keyInfo;
Review Comment:
nit: inline the variable `keyInfo`
```suggestion
return getKeyTable().getSkipCache(ozoneKey);
```
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling OBS buckets NameSpaceSummaries.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ @Override
+ public EntityType determineKeyPath(String keyName)
+ throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ iterator.seek(key);
+ if (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+ if (dbKey.equals(key)) {
+ return EntityType.KEY;
+ }
+ }
+ return EntityType.UNKNOWN;
+ }
+
+ /**
+ * This method handles disk usage of direct keys.
+ *
+ * @param parentId parent OBS bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * for each direct key's DU
+ * @param listFile if listFile is enabled, append key DU as a children
+ * keys
+ * @param duData the current DU data
+ * @param normalizedPath the normalized path request
+ * @return the total DU of all direct keys
+ * @throws IOException IOE
+ */
+ @Override
+ public long handleDirectKeys(long parentId, boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ String normalizedPath) throws IOException {
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+ long keyDataSizeWithReplica = 0L;
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ String seekPrefix = OM_KEY_PREFIX +
+ vol +
+ OM_KEY_PREFIX +
+ bucket +
+ OM_KEY_PREFIX;
+
+ NSSummary nsSummary = getReconNamespaceSummaryManager()
+ .getNSSummary(parentId);
+ // Handle the case of an empty bucket.
+ if (nsSummary == null) {
+ return 0;
+ }
+
+ iterator.seek(seekPrefix);
+
+ while (iterator.hasNext()) {
+ // KeyName : OmKeyInfo-Object
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+
+ // Exit loop if the key doesn't match the seekPrefix.
+ if (!dbKey.startsWith(seekPrefix)) {
+ break;
+ }
+
+ OmKeyInfo keyInfo = kv.getValue();
+ if (keyInfo != null) {
+ DUResponse.DiskUsage diskUsage = new DUResponse.DiskUsage();
+ String objectName = keyInfo.getKeyName();
+ diskUsage.setSubpath(objectName);
+ diskUsage.setKey(true);
+ diskUsage.setSize(keyInfo.getDataSize());
+
+ if (withReplica) {
+ long keyDU = keyInfo.getReplicatedSize();
+ keyDataSizeWithReplica += keyDU;
+ diskUsage.setSizeWithReplica(keyDU);
+ }
+ // List all the keys for the OBS bucket if requested.
+ if (listFile) {
+ duData.add(diskUsage);
+ }
+ }
+ }
+
+ return keyDataSizeWithReplica;
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public long calculateDUUnderObject(long parentId)
+ throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public long getDirObjectId(String[] names) throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public long getDirObjectId(String[] names, int cutoff) throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Returns the keyInfo object from the KEY table.
+ * @return OmKeyInfo
+ */
+ @Override
+ public OmKeyInfo getKeyInfo(String[] names) throws IOException {
+ String ozoneKey = OM_KEY_PREFIX;
+ ozoneKey += String.join(OM_KEY_PREFIX, names);
Review Comment:
nit:
```suggestion
String ozoneKey = OM_KEY_PREFIX + String.join(OM_KEY_PREFIX, names);
```
I'm guessing there must be some utility method for this.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling OBS buckets NameSpaceSummaries.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ @Override
+ public EntityType determineKeyPath(String keyName)
+ throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ iterator.seek(key);
+ if (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+ if (dbKey.equals(key)) {
+ return EntityType.KEY;
+ }
+ }
+ return EntityType.UNKNOWN;
+ }
+
+ /**
+ * This method handles disk usage of direct keys.
+ *
+ * @param parentId parent OBS bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * for each direct key's DU
+ * @param listFile if listFile is enabled, append key DU as a children
+ * keys
+ * @param duData the current DU data
+ * @param normalizedPath the normalized path request
+ * @return the total DU of all direct keys
+ * @throws IOException IOE
+ */
+ @Override
+ public long handleDirectKeys(long parentId, boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ String normalizedPath) throws IOException {
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+ long keyDataSizeWithReplica = 0L;
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ String seekPrefix = OM_KEY_PREFIX +
+ vol +
+ OM_KEY_PREFIX +
+ bucket +
+ OM_KEY_PREFIX;
+
+ NSSummary nsSummary = getReconNamespaceSummaryManager()
+ .getNSSummary(parentId);
+ // Handle the case of an empty bucket.
+ if (nsSummary == null) {
+ return 0;
+ }
+
+ iterator.seek(seekPrefix);
+
+ while (iterator.hasNext()) {
+ // KeyName : OmKeyInfo-Object
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+
+ // Exit loop if the key doesn't match the seekPrefix.
+ if (!dbKey.startsWith(seekPrefix)) {
+ break;
+ }
+
+ OmKeyInfo keyInfo = kv.getValue();
+ if (keyInfo != null) {
+ DUResponse.DiskUsage diskUsage = new DUResponse.DiskUsage();
+ String objectName = keyInfo.getKeyName();
+ diskUsage.setSubpath(objectName);
+ diskUsage.setKey(true);
+ diskUsage.setSize(keyInfo.getDataSize());
+
+ if (withReplica) {
+ long keyDU = keyInfo.getReplicatedSize();
+ keyDataSizeWithReplica += keyDU;
+ diskUsage.setSizeWithReplica(keyDU);
+ }
+ // List all the keys for the OBS bucket if requested.
+ if (listFile) {
+ duData.add(diskUsage);
+ }
+ }
+ }
+
+ return keyDataSizeWithReplica;
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public long calculateDUUnderObject(long parentId)
+ throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public long getDirObjectId(String[] names) throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public long getDirObjectId(String[] names, int cutoff) throws IOException {
+ return Long.parseLong(null);
Review Comment:
Agreed.
`NumberFormatException: Cannot parse null string`
(also for other similar code)
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/TestNSSummaryTaskWithOBS.java:
##########
@@ -0,0 +1,556 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.RDBBatchOperation;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.NSSummaryEndpoint;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Nested;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import javax.ws.rs.core.Response;
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.*;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeDirToOm;
+import static org.mockito.Mockito.mock;
+
+public class TestNSSummaryTaskWithOBS {
+ private static ReconNamespaceSummaryManager reconNamespaceSummaryManager;
+ private static OMMetadataManager omMetadataManager;
+ private static ReconOMMetadataManager reconOMMetadataManager;
+ private static NSSummaryTaskWithOBS nSSummaryTaskWithOBS;
+ private static OzoneConfiguration omConfiguration;
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String KEY_ONE = "key1";
+ private static final String KEY_TWO = "key2";
+ private static final String KEY_THREE = "dir1/dir2/key3";
+ private static final String KEY_FOUR = "key4///////////";
+ private static final String KEY_FIVE = "//////////";
+ private static final String KEY_SIX = "key6";
+ private static final String KEY_SEVEN = "key7";
+
+ private static final String TEST_USER = "TestUser";
+
+ private static final long PARENT_OBJECT_ID_ZERO = 0L;
+ private static final long VOL_OBJECT_ID = 0L;
+ private static final long BUCKET_ONE_OBJECT_ID = 1L;
+ private static final long BUCKET_TWO_OBJECT_ID = 2L;
+ private static final long KEY_ONE_OBJECT_ID = 3L;
+ private static final long KEY_TWO_OBJECT_ID = 5L;
+ private static final long KEY_FOUR_OBJECT_ID = 6L;
+ private static final long KEY_THREE_OBJECT_ID = 8L;
+ private static final long KEY_FIVE_OBJECT_ID = 9L;
+ private static final long KEY_SIX_OBJECT_ID = 10L;
+ private static final long KEY_SEVEN_OBJECT_ID = 11L;
+
+
+ private static final long KEY_ONE_SIZE = 500L;
+ private static final long KEY_TWO_OLD_SIZE = 1025L;
+ private static final long KEY_TWO_UPDATE_SIZE = 1023L;
+ private static final long KEY_THREE_SIZE =
+ ReconConstants.MAX_FILE_SIZE_UPPER_BOUND - 100L;
+ private static final long KEY_FOUR_SIZE = 2050L;
+ private static final long KEY_FIVE_SIZE = 100L;
+ private static final long KEY_SIX_SIZE = 6000L;
+ private static final long KEY_SEVEN_SIZE = 7000L;
+
+ private static Set<Long> bucketOneAns = new HashSet<>();
+ private static Set<Long> bucketTwoAns = new HashSet<>();
+ private static Set<Long> dirOneAns = new HashSet<>();
+
+ private TestNSSummaryTaskWithOBS() {
+ }
+
+ @BeforeAll
+ public static void setUp(@TempDir File tmpDir) throws Exception {
+ initializeNewOmMetadataManager(new File(tmpDir, "om"));
+ OzoneManagerServiceProviderImpl ozoneManagerServiceProvider =
+ getMockOzoneManagerServiceProvider();
+ reconOMMetadataManager = getTestReconOmMetadataManager(omMetadataManager,
+ new File(tmpDir, "recon"));
+
+ ReconTestInjector reconTestInjector =
+ new ReconTestInjector.Builder(tmpDir)
+ .withReconOm(reconOMMetadataManager)
+ .withOmServiceProvider(ozoneManagerServiceProvider)
+ .withReconSqlDb()
+ .withContainerDB()
+ .build();
+ reconNamespaceSummaryManager =
+ reconTestInjector.getInstance(ReconNamespaceSummaryManager.class);
+
+ NSSummary nonExistentSummary =
+ reconNamespaceSummaryManager.getNSSummary(BUCKET_ONE_OBJECT_ID);
+ Assertions.assertNull(nonExistentSummary);
Review Comment:
nit: please import `assertNull` and other assertions
##########
qodana.yaml:
##########
Review Comment:
Is this file added intentionally? If so, please create a separate PR for it, and provide info on what it is and how to use it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on PR #4245:
URL: https://github.com/apache/ozone/pull/4245#issuecomment-1842809972
The decision to reconsider the proposed approach for creating namespace summaries for OBS keys with prefix-based filtering was made after discussions that happened in the [document](https://docs.google.com/document/d/1g2NCR-kgcSfLFQRBhZfYsMZmaZ6edlA_26Pq0-qfzT8/edit) They expressed concerns about two key aspects:
1. Performance Impact: We are concerned that the introduction of synthetic paths as outlined in the proposal could potentially have a negative impact on Recon performance. The additional complexity of generating and managing these synthetic paths might consume more resources and potentially slow down the system.
2. Lack of Business Use Case: There were questions the practical necessity of introducing this complexity for (OBS). In contrast to the Filesystem (FSO) layout, where directory tables exist, it was suggested that there might not be a strong business need to create synthetic paths and namespace summaries for individual OBS keys. As a result, they proposed focusing on generating bucket-level summaries for OBS keys as a more pragmatic approach.
Following these discussions, we've chosen to proceed with generating disk usage statistics at the bucket level. We've decided to put the prefix-based filtering on hold for now. If it becomes necessary in the future, we'll seek input from the community. For more information, please refer to the [document](https://docs.google.com/document/d/1g2NCR-kgcSfLFQRBhZfYsMZmaZ6edlA_26Pq0-qfzT8/edit).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on PR #4245:
URL: https://github.com/apache/ozone/pull/4245#issuecomment-1969358538
`KeyName` refers to the entire path under the bucket while `FileName` refers just to the key, or to put it another way, the last component of the path. We kept the variables the same in the Legacy code so that the tests can easily be compared to FSO. I would suggest doing the same with OBS if it's feasible.
To explain it with an example `/vol1/bucket1/dir1/dir2/key1`
KeyName: `/dir1/dir2/key1`
FileName: `key1`
The issue in these tests is that there is no depth in the key structure. That's why all the key names and the file names are the same.
Check FSO and Legacy
https://github.com/apache/ozone/blob/master/hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestNSSummaryEndpointWithFSO.java#L135-L159
https://github.com/apache/ozone/blob/master/hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestNSSummaryEndpointWithLegacy.java#L135-L159
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1379654354
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/EntityHandler.java:
##########
@@ -46,6 +48,10 @@
*/
public abstract class EntityHandler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
Review Comment:
@ArafatKhan2198 thanks for the patch. Are you using this LOG instance to log anywhere ? I don't see it though.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling Legacy buckets.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ OBSBucketHandler.class);
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ public EntityType determineKeyPath(String keyName)
+ throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ iterator.seek(key);
+ if (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+ if (dbKey.equals(key)) {
+ return EntityType.KEY;
+ }
+ }
+ return EntityType.UNKNOWN;
+ }
+
+ /**
+ * This method handles disk usage of direct keys.
+ *
+ * @param parentId parent OBS bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * for each direct key's DU
+ * @param listFile if listFile is enabled, append key DU as a children
+ * keys
+ * @param duData the current DU data
+ * @param normalizedPath the normalized path request
+ * @return the total DU of all direct keys
+ * @throws IOException IOE
+ */
+ public long handleDirectKeys(long parentId, boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ String normalizedPath) throws IOException {
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+ long keyDataSizeWithReplica = 0L;
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ String seekPrefix = OM_KEY_PREFIX +
+ vol +
+ OM_KEY_PREFIX +
+ bucket +
+ OM_KEY_PREFIX;
+
+ NSSummary nsSummary = getReconNamespaceSummaryManager()
+ .getNSSummary(parentId);
+ // Handle the case of an empty bucket.
+ if (nsSummary == null) {
+ return 0;
+ }
+
+ iterator.seek(seekPrefix);
+
+ while (iterator.hasNext()) {
+ // KeyName : OmKeyInfo-Object
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+
+ // Exit loop if the key doesn't match the seekPrefix.
+ if (!dbKey.startsWith(seekPrefix)) {
+ break;
+ }
+
+ OmKeyInfo keyInfo = kv.getValue();
+ if (keyInfo != null) {
+ DUResponse.DiskUsage diskUsage = new DUResponse.DiskUsage();
+ String objectName = keyInfo.getKeyName();
+ diskUsage.setSubpath(objectName);
+ diskUsage.setKey(true);
+ diskUsage.setSize(keyInfo.getDataSize());
+
+ if (withReplica) {
+ long keyDU = keyInfo.getReplicatedSize();
+ keyDataSizeWithReplica += keyDU;
+ diskUsage.setSizeWithReplica(keyDU);
+ }
+ // List all the keys for the OBS bucket if requested.
+ if (listFile) {
+ duData.add(diskUsage);
+ }
+ }
+ }
+
+ return keyDataSizeWithReplica;
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ public long calculateDUUnderObject(long parentId)
+ throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ public long getDirObjectId(String[] names) throws IOException {
Review Comment:
Shouldn't this method be treated as an overridden method and use @OverRide annotation ?
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling Legacy buckets.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ OBSBucketHandler.class);
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ public EntityType determineKeyPath(String keyName)
+ throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ iterator.seek(key);
+ if (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+ if (dbKey.equals(key)) {
+ return EntityType.KEY;
+ }
+ }
+ return EntityType.UNKNOWN;
+ }
+
+ /**
+ * This method handles disk usage of direct keys.
+ *
+ * @param parentId parent OBS bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * for each direct key's DU
+ * @param listFile if listFile is enabled, append key DU as a children
+ * keys
+ * @param duData the current DU data
+ * @param normalizedPath the normalized path request
+ * @return the total DU of all direct keys
+ * @throws IOException IOE
+ */
+ public long handleDirectKeys(long parentId, boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ String normalizedPath) throws IOException {
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+ long keyDataSizeWithReplica = 0L;
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ String seekPrefix = OM_KEY_PREFIX +
+ vol +
+ OM_KEY_PREFIX +
+ bucket +
+ OM_KEY_PREFIX;
+
+ NSSummary nsSummary = getReconNamespaceSummaryManager()
+ .getNSSummary(parentId);
+ // Handle the case of an empty bucket.
+ if (nsSummary == null) {
+ return 0;
+ }
+
+ iterator.seek(seekPrefix);
+
+ while (iterator.hasNext()) {
+ // KeyName : OmKeyInfo-Object
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+
+ // Exit loop if the key doesn't match the seekPrefix.
+ if (!dbKey.startsWith(seekPrefix)) {
+ break;
+ }
+
+ OmKeyInfo keyInfo = kv.getValue();
+ if (keyInfo != null) {
+ DUResponse.DiskUsage diskUsage = new DUResponse.DiskUsage();
+ String objectName = keyInfo.getKeyName();
+ diskUsage.setSubpath(objectName);
+ diskUsage.setKey(true);
+ diskUsage.setSize(keyInfo.getDataSize());
+
+ if (withReplica) {
+ long keyDU = keyInfo.getReplicatedSize();
+ keyDataSizeWithReplica += keyDU;
+ diskUsage.setSizeWithReplica(keyDU);
+ }
+ // List all the keys for the OBS bucket if requested.
+ if (listFile) {
+ duData.add(diskUsage);
+ }
+ }
+ }
+
+ return keyDataSizeWithReplica;
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ public long calculateDUUnderObject(long parentId)
Review Comment:
Shouldn't this method be treated as an overridden method and use @OverRide annotation ?
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling Legacy buckets.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ OBSBucketHandler.class);
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ public EntityType determineKeyPath(String keyName)
+ throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ iterator.seek(key);
+ if (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+ if (dbKey.equals(key)) {
+ return EntityType.KEY;
+ }
+ }
+ return EntityType.UNKNOWN;
+ }
+
+ /**
+ * This method handles disk usage of direct keys.
+ *
+ * @param parentId parent OBS bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * for each direct key's DU
+ * @param listFile if listFile is enabled, append key DU as a children
+ * keys
+ * @param duData the current DU data
+ * @param normalizedPath the normalized path request
+ * @return the total DU of all direct keys
+ * @throws IOException IOE
+ */
+ public long handleDirectKeys(long parentId, boolean withReplica,
Review Comment:
Shouldn't this method be treated as an overridden method and use @OverRide annotation ?
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling Legacy buckets.
Review Comment:
Pls correct the class comment explaining what this handler do.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling Legacy buckets.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ OBSBucketHandler.class);
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ public EntityType determineKeyPath(String keyName)
Review Comment:
Shouldn't this method be treated as an overridden method and use @OverRide annotation ?
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling Legacy buckets.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ OBSBucketHandler.class);
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ public EntityType determineKeyPath(String keyName)
+ throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ iterator.seek(key);
+ if (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+ if (dbKey.equals(key)) {
+ return EntityType.KEY;
+ }
+ }
+ return EntityType.UNKNOWN;
+ }
+
+ /**
+ * This method handles disk usage of direct keys.
+ *
+ * @param parentId parent OBS bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * for each direct key's DU
+ * @param listFile if listFile is enabled, append key DU as a children
+ * keys
+ * @param duData the current DU data
+ * @param normalizedPath the normalized path request
+ * @return the total DU of all direct keys
+ * @throws IOException IOE
+ */
+ public long handleDirectKeys(long parentId, boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ String normalizedPath) throws IOException {
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+ long keyDataSizeWithReplica = 0L;
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ String seekPrefix = OM_KEY_PREFIX +
+ vol +
+ OM_KEY_PREFIX +
+ bucket +
+ OM_KEY_PREFIX;
+
+ NSSummary nsSummary = getReconNamespaceSummaryManager()
+ .getNSSummary(parentId);
+ // Handle the case of an empty bucket.
+ if (nsSummary == null) {
+ return 0;
+ }
+
+ iterator.seek(seekPrefix);
+
+ while (iterator.hasNext()) {
+ // KeyName : OmKeyInfo-Object
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+
+ // Exit loop if the key doesn't match the seekPrefix.
+ if (!dbKey.startsWith(seekPrefix)) {
+ break;
+ }
+
+ OmKeyInfo keyInfo = kv.getValue();
+ if (keyInfo != null) {
+ DUResponse.DiskUsage diskUsage = new DUResponse.DiskUsage();
+ String objectName = keyInfo.getKeyName();
+ diskUsage.setSubpath(objectName);
+ diskUsage.setKey(true);
+ diskUsage.setSize(keyInfo.getDataSize());
+
+ if (withReplica) {
+ long keyDU = keyInfo.getReplicatedSize();
+ keyDataSizeWithReplica += keyDU;
+ diskUsage.setSizeWithReplica(keyDU);
+ }
+ // List all the keys for the OBS bucket if requested.
+ if (listFile) {
+ duData.add(diskUsage);
+ }
+ }
+ }
+
+ return keyDataSizeWithReplica;
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ public long calculateDUUnderObject(long parentId)
+ throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ public long getDirObjectId(String[] names) throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ public long getDirObjectId(String[] names, int cutoff) throws IOException {
+ return Long.parseLong(null);
+ }
+
+
+ public OmKeyInfo getKeyInfo(String[] names) throws IOException {
Review Comment:
Shouldn't this method be treated as an overridden method and use @Override annotation ?
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling Legacy buckets.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ OBSBucketHandler.class);
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ public EntityType determineKeyPath(String keyName)
+ throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ iterator.seek(key);
+ if (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+ if (dbKey.equals(key)) {
+ return EntityType.KEY;
+ }
+ }
+ return EntityType.UNKNOWN;
+ }
+
+ /**
+ * This method handles disk usage of direct keys.
+ *
+ * @param parentId parent OBS bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * for each direct key's DU
+ * @param listFile if listFile is enabled, append key DU as a children
+ * keys
+ * @param duData the current DU data
+ * @param normalizedPath the normalized path request
+ * @return the total DU of all direct keys
+ * @throws IOException IOE
+ */
+ public long handleDirectKeys(long parentId, boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ String normalizedPath) throws IOException {
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+ long keyDataSizeWithReplica = 0L;
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ String seekPrefix = OM_KEY_PREFIX +
+ vol +
+ OM_KEY_PREFIX +
+ bucket +
+ OM_KEY_PREFIX;
+
+ NSSummary nsSummary = getReconNamespaceSummaryManager()
+ .getNSSummary(parentId);
+ // Handle the case of an empty bucket.
+ if (nsSummary == null) {
+ return 0;
+ }
+
+ iterator.seek(seekPrefix);
+
+ while (iterator.hasNext()) {
+ // KeyName : OmKeyInfo-Object
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+
+ // Exit loop if the key doesn't match the seekPrefix.
+ if (!dbKey.startsWith(seekPrefix)) {
+ break;
+ }
+
+ OmKeyInfo keyInfo = kv.getValue();
+ if (keyInfo != null) {
+ DUResponse.DiskUsage diskUsage = new DUResponse.DiskUsage();
+ String objectName = keyInfo.getKeyName();
+ diskUsage.setSubpath(objectName);
+ diskUsage.setKey(true);
+ diskUsage.setSize(keyInfo.getDataSize());
+
+ if (withReplica) {
+ long keyDU = keyInfo.getReplicatedSize();
+ keyDataSizeWithReplica += keyDU;
+ diskUsage.setSizeWithReplica(keyDU);
+ }
+ // List all the keys for the OBS bucket if requested.
+ if (listFile) {
+ duData.add(diskUsage);
+ }
+ }
+ }
+
+ return keyDataSizeWithReplica;
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ public long calculateDUUnderObject(long parentId)
+ throws IOException {
+ return Long.parseLong(null);
Review Comment:
Why this should return null ? Shouldn't this be returning its own dataSize based on `parentId` because to provide data size at prefix level, this method should use recursion to go thru all keys starting with parentId and accumulate the data size till that parentId key.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling Legacy buckets.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ OBSBucketHandler.class);
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ public EntityType determineKeyPath(String keyName)
+ throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ iterator.seek(key);
+ if (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+ if (dbKey.equals(key)) {
+ return EntityType.KEY;
+ }
+ }
+ return EntityType.UNKNOWN;
+ }
+
+ /**
+ * This method handles disk usage of direct keys.
+ *
+ * @param parentId parent OBS bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * for each direct key's DU
+ * @param listFile if listFile is enabled, append key DU as a children
+ * keys
+ * @param duData the current DU data
+ * @param normalizedPath the normalized path request
+ * @return the total DU of all direct keys
+ * @throws IOException IOE
+ */
+ public long handleDirectKeys(long parentId, boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ String normalizedPath) throws IOException {
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+ long keyDataSizeWithReplica = 0L;
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ String seekPrefix = OM_KEY_PREFIX +
+ vol +
+ OM_KEY_PREFIX +
+ bucket +
+ OM_KEY_PREFIX;
+
+ NSSummary nsSummary = getReconNamespaceSummaryManager()
+ .getNSSummary(parentId);
+ // Handle the case of an empty bucket.
+ if (nsSummary == null) {
+ return 0;
+ }
+
+ iterator.seek(seekPrefix);
+
+ while (iterator.hasNext()) {
+ // KeyName : OmKeyInfo-Object
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+
+ // Exit loop if the key doesn't match the seekPrefix.
+ if (!dbKey.startsWith(seekPrefix)) {
+ break;
+ }
+
+ OmKeyInfo keyInfo = kv.getValue();
+ if (keyInfo != null) {
+ DUResponse.DiskUsage diskUsage = new DUResponse.DiskUsage();
+ String objectName = keyInfo.getKeyName();
+ diskUsage.setSubpath(objectName);
+ diskUsage.setKey(true);
+ diskUsage.setSize(keyInfo.getDataSize());
+
+ if (withReplica) {
+ long keyDU = keyInfo.getReplicatedSize();
+ keyDataSizeWithReplica += keyDU;
+ diskUsage.setSizeWithReplica(keyDU);
+ }
+ // List all the keys for the OBS bucket if requested.
+ if (listFile) {
+ duData.add(diskUsage);
+ }
+ }
+ }
+
+ return keyDataSizeWithReplica;
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ public long calculateDUUnderObject(long parentId)
+ throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ public long getDirObjectId(String[] names) throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ public long getDirObjectId(String[] names, int cutoff) throws IOException {
Review Comment:
Shouldn't this method be treated as an overridden method and use @OverRide annotation ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1461727434
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling OBS buckets NameSpaceSummaries.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ @Override
+ public EntityType determineKeyPath(String keyName)
+ throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ iterator.seek(key);
+ if (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+ if (dbKey.equals(key)) {
+ return EntityType.KEY;
+ }
+ }
+ return EntityType.UNKNOWN;
+ }
+
+ /**
+ * This method handles disk usage of direct keys.
+ *
+ * @param parentId parent OBS bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * for each direct key's DU
+ * @param listFile if listFile is enabled, append key DU as a children
+ * keys
+ * @param duData the current DU data
+ * @param normalizedPath the normalized path request
+ * @return the total DU of all direct keys
+ * @throws IOException IOE
+ */
+ @Override
+ public long handleDirectKeys(long parentId, boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ String normalizedPath) throws IOException {
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+ long keyDataSizeWithReplica = 0L;
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ String seekPrefix = OM_KEY_PREFIX +
+ vol +
+ OM_KEY_PREFIX +
+ bucket +
+ OM_KEY_PREFIX;
+
+ NSSummary nsSummary = getReconNamespaceSummaryManager()
+ .getNSSummary(parentId);
+ // Handle the case of an empty bucket.
+ if (nsSummary == null) {
+ return 0;
+ }
+
+ iterator.seek(seekPrefix);
+
+ while (iterator.hasNext()) {
+ // KeyName : OmKeyInfo-Object
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+
+ // Exit loop if the key doesn't match the seekPrefix.
+ if (!dbKey.startsWith(seekPrefix)) {
+ break;
+ }
+
+ OmKeyInfo keyInfo = kv.getValue();
+ if (keyInfo != null) {
+ DUResponse.DiskUsage diskUsage = new DUResponse.DiskUsage();
+ String objectName = keyInfo.getKeyName();
+ diskUsage.setSubpath(objectName);
+ diskUsage.setKey(true);
+ diskUsage.setSize(keyInfo.getDataSize());
+
+ if (withReplica) {
+ long keyDU = keyInfo.getReplicatedSize();
+ keyDataSizeWithReplica += keyDU;
+ diskUsage.setSizeWithReplica(keyDU);
+ }
+ // List all the keys for the OBS bucket if requested.
+ if (listFile) {
+ duData.add(diskUsage);
+ }
+ }
+ }
+
+ return keyDataSizeWithReplica;
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public long calculateDUUnderObject(long parentId)
+ throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public long getDirObjectId(String[] names) throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public long getDirObjectId(String[] names, int cutoff) throws IOException {
+ return Long.parseLong(null);
Review Comment:
This will throw NumberFormatException as null is not valid long. Can throw UnsupportedOperationException as this is not supported
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #4245:
URL: https://github.com/apache/ozone/pull/4245#issuecomment-1919765528
Thanks @ArafatKhan2198 for updating the patch. There are some test failures:
```
Check Quota Usage api access | FAIL |
```
https://github.com/ArafatKhan2198/ozone/actions/runs/7727734656/job/21067723313#step:6:9
```
Error: Tests run: 2, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.064 s <<< FAILURE! -- in org.apache.hadoop.ozone.recon.tasks.TestNSSummaryTask$TestProcess
Error: org.apache.hadoop.ozone.recon.tasks.TestNSSummaryTask$TestProcess.testProcessBucket -- Time elapsed: 0.037 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <null> but was: <or...@35a9e665>
at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
at org.junit.jupiter.api.AssertNull.failNotNull(AssertNull.java:54)
at org.junit.jupiter.api.AssertNull.assertNull(AssertNull.java:38)
at org.junit.jupiter.api.AssertNull.assertNull(AssertNull.java:33)
at org.junit.jupiter.api.Assertions.assertNull(Assertions.java:275)
at org.apache.hadoop.ozone.recon.tasks.TestNSSummaryTask$TestProcess.setUp(TestNSSummaryTask.java:236)
```
https://github.com/ArafatKhan2198/ozone/actions/runs/7727734656/job/21067286096#step:5:3044
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on a diff in pull request #4245: HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets.
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1107483186
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTask.java:
##########
@@ -100,6 +104,12 @@ public Pair<String, Boolean> process(OMUpdateEventBatch events) {
} else {
LOG.error("processWithFSO failed.");
}
+ if (success) {
+ success = nsSummaryTaskWithOBS.reprocessWithOBS(reconOMMetadataManager);
+ } else {
+ LOG.error("processWithFSO failed.");
Review Comment:
```suggestion
LOG.error("processWithOBS failed.");
```
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTask.java:
##########
@@ -100,6 +104,12 @@ public Pair<String, Boolean> process(OMUpdateEventBatch events) {
} else {
LOG.error("processWithFSO failed.");
}
+ if (success) {
+ success = nsSummaryTaskWithOBS.reprocessWithOBS(reconOMMetadataManager);
Review Comment:
```suggestion
success = nsSummaryTaskWithOBS.processWithOBS(reconOMMetadataManager);
```
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTaskWithOBS.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling OBS specific tasks.
+ */
+public class NSSummaryTaskWithOBS extends NSSummaryTaskDbEventHandler {
+
+ private static final BucketLayout BUCKET_LAYOUT = BucketLayout.OBJECT_STORE;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(NSSummaryTaskWithOBS.class);
+
+ private boolean enableFileSystemPaths;
+
+ public NSSummaryTaskWithOBS(ReconNamespaceSummaryManager
+ reconNamespaceSummaryManager,
+ ReconOMMetadataManager
+ reconOMMetadataManager,
+ OzoneConfiguration
+ ozoneConfiguration) {
+ super(reconNamespaceSummaryManager,
+ reconOMMetadataManager, ozoneConfiguration);
+ // true if FileSystemPaths enabled
+ enableFileSystemPaths = ozoneConfiguration
+ .getBoolean(OMConfigKeys.OZONE_OM_ENABLE_FILESYSTEM_PATHS,
+ OMConfigKeys.OZONE_OM_ENABLE_FILESYSTEM_PATHS_DEFAULT);
+ }
+
+
+ public boolean reprocessWithOBS(OMMetadataManager omMetadataManager) {
+ Map<Long, NSSummary> nsSummaryMap = new HashMap<>();
+
+ try {
+ Table<String, OmKeyInfo> keyTable =
+ omMetadataManager.getKeyTable(BUCKET_LAYOUT);
+
+ try (TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ keyTableIter = keyTable.iterator()) {
+
+ while (keyTableIter.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = keyTableIter.next();
+ OmKeyInfo keyInfo = kv.getValue();
+
+ // KeyTable entries belong to both Legacy and OBS buckets.
+ // Check bucket layout and if it's Legacy-FS
+ // continue to the next iteration.
+ String volumeName = keyInfo.getVolumeName();
+ String bucketName = keyInfo.getBucketName();
+ String bucketDBKey = omMetadataManager
+ .getBucketKey(volumeName, bucketName);
+ // Get bucket info from bucket table
+ OmBucketInfo omBucketInfo = omMetadataManager
+ .getBucketTable().getSkipCache(bucketDBKey);
+
+ if (!omBucketInfo.getBucketLayout()
+ .isObjectStore(enableFileSystemPaths)) {
+ continue;
+ }
+
+ setKeyParentID(keyInfo);
+
+ handlePutKeyEvent(keyInfo, nsSummaryMap);
+ if (!checkAndCallFlushToDB(nsSummaryMap)) {
+ return false;
+ }
+ }
+ }
+ } catch (IOException ioEx) {
+ LOG.error("Unable to reprocess Namespace Summary data in Recon DB. ",
+ ioEx);
+ return false;
+ }
+
+ // flush and commit left out entries at end
+ if (!flushAndCommitNSToDB(nsSummaryMap)) {
+ return false;
+ }
+ LOG.info("Completed a reprocess run of NSSummaryTaskWithOBS");
+ return true;
+ }
+
+ /**
+ * KeyTable entries don't have the parentId set.
+ * In order to reuse the existing FSO methods that rely on
+ * the parentId, we have to set it explicitly.
+ *
+ * @param keyInfo
+ * @throws IOException
+ */
+ private void setKeyParentID(OmKeyInfo keyInfo) throws IOException {
Review Comment:
`System.out.println` can be replaced with `LOG.debug`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on PR #4245:
URL: https://github.com/apache/ozone/pull/4245#issuecomment-1938224073
> Thanks @ArafatKhan2198 for updating the patch. There are some test failures:
>
> ```
> Check Quota Usage api access | FAIL |
> ```
>
> https://github.com/ArafatKhan2198/ozone/actions/runs/7727734656/job/21067723313#step:6:9
>
> ```
> Error: Tests run: 2, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.064 s <<< FAILURE! -- in org.apache.hadoop.ozone.recon.tasks.TestNSSummaryTask$TestProcess
> Error: org.apache.hadoop.ozone.recon.tasks.TestNSSummaryTask$TestProcess.testProcessBucket -- Time elapsed: 0.037 s <<< FAILURE!
> org.opentest4j.AssertionFailedError: expected: <null> but was: <or...@35a9e665>
> at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
> at org.junit.jupiter.api.AssertNull.failNotNull(AssertNull.java:54)
> at org.junit.jupiter.api.AssertNull.assertNull(AssertNull.java:38)
> at org.junit.jupiter.api.AssertNull.assertNull(AssertNull.java:33)
> at org.junit.jupiter.api.Assertions.assertNull(Assertions.java:275)
> at org.apache.hadoop.ozone.recon.tasks.TestNSSummaryTask$TestProcess.setUp(TestNSSummaryTask.java:236)
> ```
>
> https://github.com/ArafatKhan2198/ozone/actions/runs/7727734656/job/21067286096#step:5:3044
Thanks they are fixed now!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #4245:
URL: https://github.com/apache/ozone/pull/4245#issuecomment-1938807024
@ArafatKhan2198 please merge `master` into your branch and resolve compile error:
```
Error: /home/runner/work/ozone/ozone/hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestNSSummaryEndpointWithOBS.java:[1129,12] no suitable constructor found for SCMNodeStat(long,long,long)
Error: constructor org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat.SCMNodeStat() is not applicable
Error: (actual and formal argument lists differ in length)
Error: constructor org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat.SCMNodeStat(org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat) is not applicable
Error: (actual and formal argument lists differ in length)
Error: constructor org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat.SCMNodeStat(long,long,long,long,long) is not applicable
Error: (actual and formal argument lists differ in length)
```
https://github.com/apache/ozone/actions/runs/7862913313/job/21479359409?pr=4245#step:6:2865
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on PR #4245:
URL: https://github.com/apache/ozone/pull/4245#issuecomment-1790610795
> Thank you for your response, @xBis7
>
> **This patch will not implement prefix-based search**; its primary purpose is to list the keys within an OBS bucket. The support for prefix-based searching in OBS buckets will be added later when I have the available bandwidth to complete the design doc.
>
> We received several support tickets from customers who reported issues with NSSummary not functioning correctly due to the lack of OBS support. Therefore, the goal of this patch is to provide the essential functionality to address this issue and eliminate any confusion for now. This patch offers a straightforward implementation of how we can generate NSSummary for an OBS layout bucket.
>
> The addition of prefix-based OBS filtering can be considered as an enhancement in subsequent updates and will be tracked by this JIRA :- https://issues.apache.org/jira/browse/HDDS-9535
I think simply providing the datasize of single OBS bucket key will not solve the purpose and pain point for users and without adding providing datasize at prefix level will create more confusion when users will have mix of buckets.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on PR #4245:
URL: https://github.com/apache/ozone/pull/4245#issuecomment-1914347714
@devmadhuu @sumitagrawl @adoroszlai @dombizita please review again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #4245:
URL: https://github.com/apache/ozone/pull/4245#issuecomment-1914354760
@ArafatKhan2198 there are some failures in your fork:
* rat: https://github.com/ArafatKhan2198/ozone/actions/runs/7693146833/job/20961333207
* findbugs: https://github.com/ArafatKhan2198/ozone/actions/runs/7693146833/job/20961333054
* checkstyle: https://github.com/ArafatKhan2198/ozone/actions/runs/7693146833/job/20961332713
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1503758563
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestNSSummaryEndpointWithOBS.java:
##########
@@ -0,0 +1,1133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor.ONE;
+import static org.apache.hadoop.ozone.om.helpers.QuotaUtil.getReplicatedSize;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.StorageType;
+import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.container.ContainerReplica;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.handlers.BucketHandler;
+import org.apache.hadoop.ozone.recon.api.handlers.EntityHandler;
+import org.apache.hadoop.ozone.recon.api.types.BucketObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NamespaceSummaryResponse;
+import org.apache.hadoop.ozone.recon.api.types.QuotaUsageResponse;
+import org.apache.hadoop.ozone.recon.api.types.ResponseStatus;
+import org.apache.hadoop.ozone.recon.api.types.VolumeObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.FileSizeDistributionResponse;
+import org.apache.hadoop.ozone.recon.common.CommonUtils;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.scm.ReconNodeManager;
+import org.apache.hadoop.ozone.recon.scm.ReconStorageContainerManagerFacade;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.StorageContainerServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.spi.impl.StorageContainerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.tasks.NSSummaryTaskWithOBS;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import javax.ws.rs.core.Response;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.HashSet;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.apache.hadoop.hdds.protocol.MockDatanodeDetails.randomDatanodeDetails;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProvider;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for NSSummary REST APIs with OBS.
+ * Testing is done on a simple object store model with a flat hierarchy:
+ * vol
+ * / \
+ * bucket1 bucket2
+ * | |
+ * file1 file2
+ *
+ * This tests the Rest APIs for NSSummary in the context of OBS buckets,
+ * focusing on disk usage, quota usage, and file size distribution.
+ */
+public class TestNSSummaryEndpointWithOBS {
+ @TempDir
+ private Path temporaryFolder;
+
+ private ReconOMMetadataManager reconOMMetadataManager;
+ private NSSummaryEndpoint nsSummaryEndpoint;
+ private OzoneConfiguration conf;
+ private CommonUtils commonUtils;
+
+ private static final String TEST_PATH_UTILITY =
+ "/vol1/buck1/a/b/c/d/e/file1.txt";
+ private static final String PARENT_DIR = "vol1/buck1/a/b/c/d/e";
+ private static final String[] TEST_NAMES =
+ new String[]{"vol1", "buck1", "a", "b", "c", "d", "e", "file1.txt"};
+ private static final String TEST_KEY_NAMES = "a/b/c/d/e/file1.txt";
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String VOL_TWO = "vol2";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String BUCKET_THREE = "bucket3";
+ private static final String BUCKET_FOUR = "bucket4";
+ private static final String KEY_ONE = "file1";
+ private static final String KEY_TWO = "file2";
+ private static final String KEY_THREE = "file3";
+ private static final String KEY_FOUR = "file4";
+ private static final String KEY_FIVE = "file5";
+ private static final String KEY_SIX = "file6";
+ private static final String KEY_SEVEN = "file7";
+ private static final String KEY_EIGHT = "file8";
+ private static final String KEY_NINE = "file9";
+ private static final String KEY_TEN = "file10";
+ private static final String KEY_ELEVEN = "file11";
+ private static final String MULTI_BLOCK_KEY = "file7";
+ private static final String MULTI_BLOCK_FILE = "file7";
+
+ private static final String FILE_ONE = "file1";
+ private static final String FILE_TWO = "file2";
+ private static final String FILE_THREE = "file3";
+ private static final String FILE_FOUR = "file4";
+ private static final String FILE_FIVE = "file5";
+ private static final String FILE_SIX = "file6";
+ private static final String FILE_SEVEN = "file7";
+ private static final String FILE_EIGHT = "file8";
+ private static final String FILE_NINE = "file9";
+ private static final String FILE_TEN = "file10";
+ private static final String FILE_ELEVEN = "file11";
Review Comment:
This is how the other tests have been written `TestNSSummaryEndpointWithLegacy`, `TestNSSummaryEndpointWithOBS` I thought I would go with the same convention.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1504085055
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestNSSummaryEndpointWithOBS.java:
##########
@@ -0,0 +1,1133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor.ONE;
+import static org.apache.hadoop.ozone.om.helpers.QuotaUtil.getReplicatedSize;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.StorageType;
+import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.container.ContainerReplica;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.handlers.BucketHandler;
+import org.apache.hadoop.ozone.recon.api.handlers.EntityHandler;
+import org.apache.hadoop.ozone.recon.api.types.BucketObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NamespaceSummaryResponse;
+import org.apache.hadoop.ozone.recon.api.types.QuotaUsageResponse;
+import org.apache.hadoop.ozone.recon.api.types.ResponseStatus;
+import org.apache.hadoop.ozone.recon.api.types.VolumeObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.FileSizeDistributionResponse;
+import org.apache.hadoop.ozone.recon.common.CommonUtils;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.scm.ReconNodeManager;
+import org.apache.hadoop.ozone.recon.scm.ReconStorageContainerManagerFacade;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.StorageContainerServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.spi.impl.StorageContainerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.tasks.NSSummaryTaskWithOBS;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import javax.ws.rs.core.Response;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.HashSet;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.apache.hadoop.hdds.protocol.MockDatanodeDetails.randomDatanodeDetails;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProvider;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for NSSummary REST APIs with OBS.
+ * Testing is done on a simple object store model with a flat hierarchy:
+ * vol
+ * / \
+ * bucket1 bucket2
+ * | |
+ * file1 file2
+ *
+ * This tests the Rest APIs for NSSummary in the context of OBS buckets,
+ * focusing on disk usage, quota usage, and file size distribution.
+ */
+public class TestNSSummaryEndpointWithOBS {
+ @TempDir
+ private Path temporaryFolder;
+
+ private ReconOMMetadataManager reconOMMetadataManager;
+ private NSSummaryEndpoint nsSummaryEndpoint;
+ private OzoneConfiguration conf;
+ private CommonUtils commonUtils;
+
+ private static final String TEST_PATH_UTILITY =
+ "/vol1/buck1/a/b/c/d/e/file1.txt";
+ private static final String PARENT_DIR = "vol1/buck1/a/b/c/d/e";
+ private static final String[] TEST_NAMES =
+ new String[]{"vol1", "buck1", "a", "b", "c", "d", "e", "file1.txt"};
+ private static final String TEST_KEY_NAMES = "a/b/c/d/e/file1.txt";
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String VOL_TWO = "vol2";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String BUCKET_THREE = "bucket3";
+ private static final String BUCKET_FOUR = "bucket4";
+ private static final String KEY_ONE = "file1";
+ private static final String KEY_TWO = "file2";
+ private static final String KEY_THREE = "file3";
+ private static final String KEY_FOUR = "file4";
+ private static final String KEY_FIVE = "file5";
+ private static final String KEY_SIX = "file6";
+ private static final String KEY_SEVEN = "file7";
+ private static final String KEY_EIGHT = "file8";
+ private static final String KEY_NINE = "file9";
+ private static final String KEY_TEN = "file10";
+ private static final String KEY_ELEVEN = "file11";
+ private static final String MULTI_BLOCK_KEY = "file7";
+ private static final String MULTI_BLOCK_FILE = "file7";
+
+ private static final String FILE_ONE = "file1";
+ private static final String FILE_TWO = "file2";
+ private static final String FILE_THREE = "file3";
+ private static final String FILE_FOUR = "file4";
+ private static final String FILE_FIVE = "file5";
+ private static final String FILE_SIX = "file6";
+ private static final String FILE_SEVEN = "file7";
+ private static final String FILE_EIGHT = "file8";
+ private static final String FILE_NINE = "file9";
+ private static final String FILE_TEN = "file10";
+ private static final String FILE_ELEVEN = "file11";
+ // objects IDs
+ private static final long PARENT_OBJECT_ID_ZERO = 0L;
+ private static final long VOL_OBJECT_ID = 0L;
+ private static final long BUCKET_ONE_OBJECT_ID = 1L;
+ private static final long BUCKET_TWO_OBJECT_ID = 2L;
+ private static final long KEY_ONE_OBJECT_ID = 3L;
+ private static final long KEY_TWO_OBJECT_ID = 5L;
+ private static final long KEY_FOUR_OBJECT_ID = 6L;
+ private static final long KEY_THREE_OBJECT_ID = 8L;
+ private static final long KEY_FIVE_OBJECT_ID = 9L;
+ private static final long KEY_SIX_OBJECT_ID = 10L;
+ private static final long MULTI_BLOCK_KEY_OBJECT_ID = 13L;
+ private static final long KEY_SEVEN_OBJECT_ID = 13L;
+ private static final long VOL_TWO_OBJECT_ID = 14L;
+ private static final long BUCKET_THREE_OBJECT_ID = 15L;
+ private static final long BUCKET_FOUR_OBJECT_ID = 16L;
+ private static final long KEY_EIGHT_OBJECT_ID = 17L;
+ private static final long KEY_NINE_OBJECT_ID = 19L;
+ private static final long KEY_TEN_OBJECT_ID = 20L;
+ private static final long KEY_ELEVEN_OBJECT_ID = 21L;
+
+ // container IDs
+ private static final long CONTAINER_ONE_ID = 1L;
+ private static final long CONTAINER_TWO_ID = 2L;
+ private static final long CONTAINER_THREE_ID = 3L;
+ private static final long CONTAINER_FOUR_ID = 4L;
+ private static final long CONTAINER_FIVE_ID = 5L;
+ private static final long CONTAINER_SIX_ID = 6L;
+
+ // replication factors
+ private static final int CONTAINER_ONE_REPLICA_COUNT = 3;
+ private static final int CONTAINER_TWO_REPLICA_COUNT = 2;
+ private static final int CONTAINER_THREE_REPLICA_COUNT = 4;
+ private static final int CONTAINER_FOUR_REPLICA_COUNT = 5;
+ private static final int CONTAINER_FIVE_REPLICA_COUNT = 2;
+ private static final int CONTAINER_SIX_REPLICA_COUNT = 3;
+
+ // block lengths
+ private static final long BLOCK_ONE_LENGTH = 1000L;
+ private static final long BLOCK_TWO_LENGTH = 2000L;
+ private static final long BLOCK_THREE_LENGTH = 3000L;
+ private static final long BLOCK_FOUR_LENGTH = 4000L;
+ private static final long BLOCK_FIVE_LENGTH = 5000L;
+ private static final long BLOCK_SIX_LENGTH = 6000L;
+
+ // data size in bytes
+ private static final long KEY_ONE_SIZE = 500L; // bin 0
+ private static final long KEY_TWO_SIZE = OzoneConsts.KB + 1; // bin 1
+ private static final long KEY_THREE_SIZE = 4 * OzoneConsts.KB + 1; // bin 3
+ private static final long KEY_FOUR_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_FIVE_SIZE = 100L; // bin 0
+ private static final long KEY_SIX_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_SEVEN_SIZE = 4 * OzoneConsts.KB + 1;
+ private static final long KEY_EIGHT_SIZE = OzoneConsts.KB + 1; // bin 1
+ private static final long KEY_NINE_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_TEN_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_ELEVEN_SIZE = OzoneConsts.KB + 1; // bin 1
+
+ private static final long FILE1_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_ONE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE2_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_TWO_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE3_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_THREE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE4_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_FOUR_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE5_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_FIVE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE6_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_SIX_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));;
+ private static final long FILE7_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_SEVEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));;
+ private static final long FILE8_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_EIGHT_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE9_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_NINE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE10_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_TEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE11_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_ELEVEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+
+ private static final long MULTI_BLOCK_KEY_SIZE_WITH_REPLICA
+ = FILE7_SIZE_WITH_REPLICA;
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_ROOT
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA
+ + FILE4_SIZE_WITH_REPLICA
+ + FILE5_SIZE_WITH_REPLICA
+ + FILE8_SIZE_WITH_REPLICA
+ + FILE9_SIZE_WITH_REPLICA
+ + FILE10_SIZE_WITH_REPLICA
+ + FILE11_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_VOL
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA
+ + FILE4_SIZE_WITH_REPLICA
+ + FILE5_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_BUCKET1
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_KEY
+ = FILE4_SIZE_WITH_REPLICA;
+
+ // quota in bytes
+ private static final long ROOT_QUOTA = 2 * (2 * OzoneConsts.MB);
+ private static final long VOL_QUOTA = 2 * OzoneConsts.MB;
+ private static final long VOL_TWO_QUOTA = 2 * OzoneConsts.MB;
+ private static final long BUCKET_ONE_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_TWO_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_THREE_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_FOUR_QUOTA = OzoneConsts.MB;
+
+ // mock client's path requests
+ private static final String TEST_USER = "TestUser";
+ private static final String ROOT_PATH = "/";
+ private static final String VOL_PATH = "/vol";
+ private static final String VOL_TWO_PATH = "/vol2";
+ private static final String BUCKET_ONE_PATH = "/vol/bucket1";
+ private static final String BUCKET_TWO_PATH = "/vol/bucket2";
+ private static final String KEY_PATH = "/vol/bucket2/file4";
+ private static final String MULTI_BLOCK_KEY_PATH = "/vol/bucket1/file7";
+ private static final String INVALID_PATH = "/vol/path/not/found";
+
+ // some expected answers
+ private static final long ROOT_DATA_SIZE =
+ KEY_ONE_SIZE + KEY_TWO_SIZE + KEY_THREE_SIZE + KEY_FOUR_SIZE +
+ KEY_FIVE_SIZE + KEY_EIGHT_SIZE + KEY_NINE_SIZE + KEY_TEN_SIZE +
+ KEY_ELEVEN_SIZE;
+ private static final long VOL_DATA_SIZE = KEY_ONE_SIZE + KEY_TWO_SIZE +
+ KEY_THREE_SIZE + KEY_FOUR_SIZE + KEY_FIVE_SIZE;
+
+ private static final long VOL_TWO_DATA_SIZE =
+ KEY_EIGHT_SIZE + KEY_NINE_SIZE + KEY_TEN_SIZE + KEY_ELEVEN_SIZE;
+
+ private static final long BUCKET_ONE_DATA_SIZE = KEY_ONE_SIZE + KEY_TWO_SIZE +
+ KEY_THREE_SIZE;
+
+ private static final long BUCKET_TWO_DATA_SIZE =
+ KEY_FOUR_SIZE + KEY_FIVE_SIZE;
+
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ conf = new OzoneConfiguration();
+ OMMetadataManager omMetadataManager = initializeNewOmMetadataManager(
+ Files.createDirectory(temporaryFolder.resolve(
+ "JunitOmDBDir")).toFile(), conf);
+ OzoneManagerServiceProviderImpl ozoneManagerServiceProvider =
+ getMockOzoneManagerServiceProvider();
+ reconOMMetadataManager = getTestReconOmMetadataManager(omMetadataManager,
+ Files.createDirectory(temporaryFolder.resolve(
+ "omMetadatDir")).toFile());
+
+ ReconTestInjector reconTestInjector =
+ new ReconTestInjector.Builder(temporaryFolder.toFile())
+ .withReconOm(reconOMMetadataManager)
+ .withOmServiceProvider(ozoneManagerServiceProvider)
+ .withReconSqlDb()
+ .withContainerDB()
+ .addBinding(OzoneStorageContainerManager.class,
+ getMockReconSCM())
+ .addBinding(StorageContainerServiceProvider.class,
+ mock(StorageContainerServiceProviderImpl.class))
+ .addBinding(NSSummaryEndpoint.class)
+ .build();
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager =
+ reconTestInjector.getInstance(ReconNamespaceSummaryManager.class);
+ nsSummaryEndpoint = reconTestInjector.getInstance(NSSummaryEndpoint.class);
+
+ // populate OM DB and reprocess into Recon RocksDB
+ populateOMDB();
+ NSSummaryTaskWithOBS nsSummaryTaskWithOBS =
+ new NSSummaryTaskWithOBS(reconNamespaceSummaryManager,
+ reconOMMetadataManager, conf);
+ nsSummaryTaskWithOBS.reprocessWithOBS(reconOMMetadataManager);
+ commonUtils = new CommonUtils();
+ }
+
+ @Test
+ public void testUtility() {
+ String[] names = EntityHandler.parseRequestPath(TEST_PATH_UTILITY);
+ assertArrayEquals(TEST_NAMES, names);
+ String keyName = BucketHandler.getKeyName(names);
+ assertEquals(TEST_KEY_NAMES, keyName);
+ String subpath = BucketHandler.buildSubpath(PARENT_DIR, "file1.txt");
+ assertEquals(TEST_PATH_UTILITY, subpath);
+ }
+
+ @Test
+ public void testGetBasicInfoRoot() throws Exception {
+ // Test root basics
+ Response rootResponse = nsSummaryEndpoint.getBasicInfo(ROOT_PATH);
+ NamespaceSummaryResponse rootResponseObj =
+ (NamespaceSummaryResponse) rootResponse.getEntity();
+ assertEquals(EntityType.ROOT, rootResponseObj.getEntityType());
+ assertEquals(2, rootResponseObj.getCountStats().getNumVolume());
+ assertEquals(4, rootResponseObj.getCountStats().getNumBucket());
+ assertEquals(9, rootResponseObj.getCountStats().getNumTotalKey());
+ }
+
+ @Test
+ public void testGetBasicInfoVol() throws Exception {
+ // Test volume basics
+ Response volResponse = nsSummaryEndpoint.getBasicInfo(VOL_PATH);
+ NamespaceSummaryResponse volResponseObj =
+ (NamespaceSummaryResponse) volResponse.getEntity();
+ assertEquals(EntityType.VOLUME,
+ volResponseObj.getEntityType());
+ assertEquals(2, volResponseObj.getCountStats().getNumBucket());
+ assertEquals(5, volResponseObj.getCountStats().getNumTotalKey());
+ assertEquals("TestUser", ((VolumeObjectDBInfo) volResponseObj.
+ getObjectDBInfo()).getAdmin());
+ assertEquals("TestUser", ((VolumeObjectDBInfo) volResponseObj.
+ getObjectDBInfo()).getOwner());
+ assertEquals("vol", volResponseObj.getObjectDBInfo().getName());
+ assertEquals(2097152, volResponseObj.getObjectDBInfo().getQuotaInBytes());
+ assertEquals(-1, volResponseObj.getObjectDBInfo().getQuotaInNamespace());
+ }
+
+ @Test
+ public void testGetBasicInfoBucketOne() throws Exception {
+ // Test bucket 1's basics
+ Response bucketOneResponse =
+ nsSummaryEndpoint.getBasicInfo(BUCKET_ONE_PATH);
+ NamespaceSummaryResponse bucketOneObj =
+ (NamespaceSummaryResponse) bucketOneResponse.getEntity();
+ assertEquals(EntityType.BUCKET, bucketOneObj.getEntityType());
+ assertEquals(3, bucketOneObj.getCountStats().getNumTotalKey());
+ assertEquals("vol",
+ ((BucketObjectDBInfo) bucketOneObj.getObjectDBInfo()).getVolumeName());
+ assertEquals(StorageType.DISK,
+ ((BucketObjectDBInfo)
+ bucketOneObj.getObjectDBInfo()).getStorageType());
+ assertEquals(getBucketLayout(),
+ ((BucketObjectDBInfo)
+ bucketOneObj.getObjectDBInfo()).getBucketLayout());
+ assertEquals("bucket1",
+ ((BucketObjectDBInfo) bucketOneObj.getObjectDBInfo()).getName());
+ }
+
+ @Test
+ public void testGetBasicInfoBucketTwo() throws Exception {
+ // Test bucket 2's basics
+ commonUtils.testNSSummaryBasicInfoBucketTwo(
+ BucketLayout.OBJECT_STORE,
+ nsSummaryEndpoint);
+ }
+
+ @Test
+ public void testGetBasicInfoNoPath() throws Exception {
+ // Test invalid path
+ commonUtils.testNSSummaryBasicInfoNoPath(nsSummaryEndpoint);
+ }
+
+ @Test
+ public void testGetBasicInfoKey() throws Exception {
+ // Test key
+ commonUtils.testNSSummaryBasicInfoKey(nsSummaryEndpoint);
+ }
+
+ @Test
+ public void testDiskUsageRoot() throws Exception {
+ // root level DU
+ Response rootResponse = nsSummaryEndpoint.getDiskUsage(ROOT_PATH,
+ false, false);
+ DUResponse duRootRes = (DUResponse) rootResponse.getEntity();
+ assertEquals(2, duRootRes.getCount());
+ List<DUResponse.DiskUsage> duRootData = duRootRes.getDuData();
+ // sort based on subpath
+ Collections.sort(duRootData,
+ Comparator.comparing(DUResponse.DiskUsage::getSubpath));
+ DUResponse.DiskUsage duVol1 = duRootData.get(0);
+ DUResponse.DiskUsage duVol2 = duRootData.get(1);
+ assertEquals(VOL_PATH, duVol1.getSubpath());
+ assertEquals(VOL_TWO_PATH, duVol2.getSubpath());
+ assertEquals(VOL_DATA_SIZE, duVol1.getSize());
+ assertEquals(VOL_TWO_DATA_SIZE, duVol2.getSize());
+ }
+
+ @Test
+ public void testDiskUsageVolume() throws Exception {
+ // volume level DU
+ Response volResponse = nsSummaryEndpoint.getDiskUsage(VOL_PATH,
+ false, false);
+ DUResponse duVolRes = (DUResponse) volResponse.getEntity();
+ assertEquals(2, duVolRes.getCount());
+ List<DUResponse.DiskUsage> duData = duVolRes.getDuData();
+ // sort based on subpath
+ Collections.sort(duData,
+ Comparator.comparing(DUResponse.DiskUsage::getSubpath));
+ DUResponse.DiskUsage duBucket1 = duData.get(0);
+ DUResponse.DiskUsage duBucket2 = duData.get(1);
+ assertEquals(BUCKET_ONE_PATH, duBucket1.getSubpath());
+ assertEquals(BUCKET_TWO_PATH, duBucket2.getSubpath());
+ assertEquals(BUCKET_ONE_DATA_SIZE, duBucket1.getSize());
+ assertEquals(BUCKET_TWO_DATA_SIZE, duBucket2.getSize());
+ }
+
+ @Test
+ public void testDiskUsageBucket() throws Exception {
+ // bucket level DU
+ Response bucketResponse = nsSummaryEndpoint.getDiskUsage(BUCKET_ONE_PATH,
+ false, false);
+ DUResponse duBucketResponse = (DUResponse) bucketResponse.getEntity();
+ // There are no sub-paths under this OBS bucket.
+ assertEquals(0, duBucketResponse.getCount());
+ assertEquals(BUCKET_ONE_DATA_SIZE, duBucketResponse.getSize());
+ }
+
+ @Test
+ public void testDiskUsageKey() throws Exception {
+ // key level DU
+ Response keyResponse = nsSummaryEndpoint.getDiskUsage(KEY_PATH,
+ false, false);
+ DUResponse keyObj = (DUResponse) keyResponse.getEntity();
+ assertEquals(0, keyObj.getCount());
+ assertEquals(KEY_FOUR_SIZE, keyObj.getSize());
+ }
+
+ @Test
+ public void testDiskUsageUnknown() throws Exception {
+ // invalid path check
+ Response invalidResponse = nsSummaryEndpoint.getDiskUsage(INVALID_PATH,
+ false, false);
+ DUResponse invalidObj = (DUResponse) invalidResponse.getEntity();
+ assertEquals(ResponseStatus.PATH_NOT_FOUND,
+ invalidObj.getStatus());
+ }
Review Comment:
No they trigger different endpoints.
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestNSSummaryEndpointWithOBS.java:
##########
@@ -0,0 +1,1133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor.ONE;
+import static org.apache.hadoop.ozone.om.helpers.QuotaUtil.getReplicatedSize;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.StorageType;
+import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.container.ContainerReplica;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.handlers.BucketHandler;
+import org.apache.hadoop.ozone.recon.api.handlers.EntityHandler;
+import org.apache.hadoop.ozone.recon.api.types.BucketObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NamespaceSummaryResponse;
+import org.apache.hadoop.ozone.recon.api.types.QuotaUsageResponse;
+import org.apache.hadoop.ozone.recon.api.types.ResponseStatus;
+import org.apache.hadoop.ozone.recon.api.types.VolumeObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.FileSizeDistributionResponse;
+import org.apache.hadoop.ozone.recon.common.CommonUtils;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.scm.ReconNodeManager;
+import org.apache.hadoop.ozone.recon.scm.ReconStorageContainerManagerFacade;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.StorageContainerServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.spi.impl.StorageContainerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.tasks.NSSummaryTaskWithOBS;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import javax.ws.rs.core.Response;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.HashSet;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.apache.hadoop.hdds.protocol.MockDatanodeDetails.randomDatanodeDetails;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProvider;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for NSSummary REST APIs with OBS.
+ * Testing is done on a simple object store model with a flat hierarchy:
+ * vol
+ * / \
+ * bucket1 bucket2
+ * | |
+ * file1 file2
+ *
+ * This tests the Rest APIs for NSSummary in the context of OBS buckets,
+ * focusing on disk usage, quota usage, and file size distribution.
+ */
+public class TestNSSummaryEndpointWithOBS {
+ @TempDir
+ private Path temporaryFolder;
+
+ private ReconOMMetadataManager reconOMMetadataManager;
+ private NSSummaryEndpoint nsSummaryEndpoint;
+ private OzoneConfiguration conf;
+ private CommonUtils commonUtils;
+
+ private static final String TEST_PATH_UTILITY =
+ "/vol1/buck1/a/b/c/d/e/file1.txt";
+ private static final String PARENT_DIR = "vol1/buck1/a/b/c/d/e";
+ private static final String[] TEST_NAMES =
+ new String[]{"vol1", "buck1", "a", "b", "c", "d", "e", "file1.txt"};
+ private static final String TEST_KEY_NAMES = "a/b/c/d/e/file1.txt";
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String VOL_TWO = "vol2";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String BUCKET_THREE = "bucket3";
+ private static final String BUCKET_FOUR = "bucket4";
+ private static final String KEY_ONE = "file1";
+ private static final String KEY_TWO = "file2";
+ private static final String KEY_THREE = "file3";
+ private static final String KEY_FOUR = "file4";
+ private static final String KEY_FIVE = "file5";
+ private static final String KEY_SIX = "file6";
+ private static final String KEY_SEVEN = "file7";
+ private static final String KEY_EIGHT = "file8";
+ private static final String KEY_NINE = "file9";
+ private static final String KEY_TEN = "file10";
+ private static final String KEY_ELEVEN = "file11";
+ private static final String MULTI_BLOCK_KEY = "file7";
+ private static final String MULTI_BLOCK_FILE = "file7";
+
+ private static final String FILE_ONE = "file1";
+ private static final String FILE_TWO = "file2";
+ private static final String FILE_THREE = "file3";
+ private static final String FILE_FOUR = "file4";
+ private static final String FILE_FIVE = "file5";
+ private static final String FILE_SIX = "file6";
+ private static final String FILE_SEVEN = "file7";
+ private static final String FILE_EIGHT = "file8";
+ private static final String FILE_NINE = "file9";
+ private static final String FILE_TEN = "file10";
+ private static final String FILE_ELEVEN = "file11";
+ // objects IDs
+ private static final long PARENT_OBJECT_ID_ZERO = 0L;
+ private static final long VOL_OBJECT_ID = 0L;
+ private static final long BUCKET_ONE_OBJECT_ID = 1L;
+ private static final long BUCKET_TWO_OBJECT_ID = 2L;
+ private static final long KEY_ONE_OBJECT_ID = 3L;
+ private static final long KEY_TWO_OBJECT_ID = 5L;
+ private static final long KEY_FOUR_OBJECT_ID = 6L;
+ private static final long KEY_THREE_OBJECT_ID = 8L;
+ private static final long KEY_FIVE_OBJECT_ID = 9L;
+ private static final long KEY_SIX_OBJECT_ID = 10L;
+ private static final long MULTI_BLOCK_KEY_OBJECT_ID = 13L;
+ private static final long KEY_SEVEN_OBJECT_ID = 13L;
+ private static final long VOL_TWO_OBJECT_ID = 14L;
+ private static final long BUCKET_THREE_OBJECT_ID = 15L;
+ private static final long BUCKET_FOUR_OBJECT_ID = 16L;
+ private static final long KEY_EIGHT_OBJECT_ID = 17L;
+ private static final long KEY_NINE_OBJECT_ID = 19L;
+ private static final long KEY_TEN_OBJECT_ID = 20L;
+ private static final long KEY_ELEVEN_OBJECT_ID = 21L;
+
+ // container IDs
+ private static final long CONTAINER_ONE_ID = 1L;
+ private static final long CONTAINER_TWO_ID = 2L;
+ private static final long CONTAINER_THREE_ID = 3L;
+ private static final long CONTAINER_FOUR_ID = 4L;
+ private static final long CONTAINER_FIVE_ID = 5L;
+ private static final long CONTAINER_SIX_ID = 6L;
+
+ // replication factors
+ private static final int CONTAINER_ONE_REPLICA_COUNT = 3;
+ private static final int CONTAINER_TWO_REPLICA_COUNT = 2;
+ private static final int CONTAINER_THREE_REPLICA_COUNT = 4;
+ private static final int CONTAINER_FOUR_REPLICA_COUNT = 5;
+ private static final int CONTAINER_FIVE_REPLICA_COUNT = 2;
+ private static final int CONTAINER_SIX_REPLICA_COUNT = 3;
+
+ // block lengths
+ private static final long BLOCK_ONE_LENGTH = 1000L;
+ private static final long BLOCK_TWO_LENGTH = 2000L;
+ private static final long BLOCK_THREE_LENGTH = 3000L;
+ private static final long BLOCK_FOUR_LENGTH = 4000L;
+ private static final long BLOCK_FIVE_LENGTH = 5000L;
+ private static final long BLOCK_SIX_LENGTH = 6000L;
+
+ // data size in bytes
+ private static final long KEY_ONE_SIZE = 500L; // bin 0
+ private static final long KEY_TWO_SIZE = OzoneConsts.KB + 1; // bin 1
+ private static final long KEY_THREE_SIZE = 4 * OzoneConsts.KB + 1; // bin 3
+ private static final long KEY_FOUR_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_FIVE_SIZE = 100L; // bin 0
+ private static final long KEY_SIX_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_SEVEN_SIZE = 4 * OzoneConsts.KB + 1;
+ private static final long KEY_EIGHT_SIZE = OzoneConsts.KB + 1; // bin 1
+ private static final long KEY_NINE_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_TEN_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_ELEVEN_SIZE = OzoneConsts.KB + 1; // bin 1
+
+ private static final long FILE1_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_ONE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE2_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_TWO_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE3_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_THREE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE4_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_FOUR_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE5_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_FIVE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE6_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_SIX_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));;
+ private static final long FILE7_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_SEVEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));;
+ private static final long FILE8_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_EIGHT_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE9_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_NINE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE10_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_TEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE11_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_ELEVEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+
+ private static final long MULTI_BLOCK_KEY_SIZE_WITH_REPLICA
+ = FILE7_SIZE_WITH_REPLICA;
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_ROOT
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA
+ + FILE4_SIZE_WITH_REPLICA
+ + FILE5_SIZE_WITH_REPLICA
+ + FILE8_SIZE_WITH_REPLICA
+ + FILE9_SIZE_WITH_REPLICA
+ + FILE10_SIZE_WITH_REPLICA
+ + FILE11_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_VOL
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA
+ + FILE4_SIZE_WITH_REPLICA
+ + FILE5_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_BUCKET1
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_KEY
+ = FILE4_SIZE_WITH_REPLICA;
+
+ // quota in bytes
+ private static final long ROOT_QUOTA = 2 * (2 * OzoneConsts.MB);
+ private static final long VOL_QUOTA = 2 * OzoneConsts.MB;
+ private static final long VOL_TWO_QUOTA = 2 * OzoneConsts.MB;
+ private static final long BUCKET_ONE_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_TWO_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_THREE_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_FOUR_QUOTA = OzoneConsts.MB;
+
+ // mock client's path requests
+ private static final String TEST_USER = "TestUser";
+ private static final String ROOT_PATH = "/";
+ private static final String VOL_PATH = "/vol";
+ private static final String VOL_TWO_PATH = "/vol2";
+ private static final String BUCKET_ONE_PATH = "/vol/bucket1";
+ private static final String BUCKET_TWO_PATH = "/vol/bucket2";
+ private static final String KEY_PATH = "/vol/bucket2/file4";
+ private static final String MULTI_BLOCK_KEY_PATH = "/vol/bucket1/file7";
+ private static final String INVALID_PATH = "/vol/path/not/found";
+
+ // some expected answers
+ private static final long ROOT_DATA_SIZE =
+ KEY_ONE_SIZE + KEY_TWO_SIZE + KEY_THREE_SIZE + KEY_FOUR_SIZE +
+ KEY_FIVE_SIZE + KEY_EIGHT_SIZE + KEY_NINE_SIZE + KEY_TEN_SIZE +
+ KEY_ELEVEN_SIZE;
+ private static final long VOL_DATA_SIZE = KEY_ONE_SIZE + KEY_TWO_SIZE +
+ KEY_THREE_SIZE + KEY_FOUR_SIZE + KEY_FIVE_SIZE;
+
+ private static final long VOL_TWO_DATA_SIZE =
+ KEY_EIGHT_SIZE + KEY_NINE_SIZE + KEY_TEN_SIZE + KEY_ELEVEN_SIZE;
+
+ private static final long BUCKET_ONE_DATA_SIZE = KEY_ONE_SIZE + KEY_TWO_SIZE +
+ KEY_THREE_SIZE;
+
+ private static final long BUCKET_TWO_DATA_SIZE =
+ KEY_FOUR_SIZE + KEY_FIVE_SIZE;
+
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ conf = new OzoneConfiguration();
+ OMMetadataManager omMetadataManager = initializeNewOmMetadataManager(
+ Files.createDirectory(temporaryFolder.resolve(
+ "JunitOmDBDir")).toFile(), conf);
+ OzoneManagerServiceProviderImpl ozoneManagerServiceProvider =
+ getMockOzoneManagerServiceProvider();
+ reconOMMetadataManager = getTestReconOmMetadataManager(omMetadataManager,
+ Files.createDirectory(temporaryFolder.resolve(
+ "omMetadatDir")).toFile());
+
+ ReconTestInjector reconTestInjector =
+ new ReconTestInjector.Builder(temporaryFolder.toFile())
+ .withReconOm(reconOMMetadataManager)
+ .withOmServiceProvider(ozoneManagerServiceProvider)
+ .withReconSqlDb()
+ .withContainerDB()
+ .addBinding(OzoneStorageContainerManager.class,
+ getMockReconSCM())
+ .addBinding(StorageContainerServiceProvider.class,
+ mock(StorageContainerServiceProviderImpl.class))
+ .addBinding(NSSummaryEndpoint.class)
+ .build();
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager =
+ reconTestInjector.getInstance(ReconNamespaceSummaryManager.class);
+ nsSummaryEndpoint = reconTestInjector.getInstance(NSSummaryEndpoint.class);
+
+ // populate OM DB and reprocess into Recon RocksDB
+ populateOMDB();
+ NSSummaryTaskWithOBS nsSummaryTaskWithOBS =
+ new NSSummaryTaskWithOBS(reconNamespaceSummaryManager,
+ reconOMMetadataManager, conf);
+ nsSummaryTaskWithOBS.reprocessWithOBS(reconOMMetadataManager);
+ commonUtils = new CommonUtils();
+ }
+
+ @Test
+ public void testUtility() {
+ String[] names = EntityHandler.parseRequestPath(TEST_PATH_UTILITY);
+ assertArrayEquals(TEST_NAMES, names);
+ String keyName = BucketHandler.getKeyName(names);
+ assertEquals(TEST_KEY_NAMES, keyName);
+ String subpath = BucketHandler.buildSubpath(PARENT_DIR, "file1.txt");
+ assertEquals(TEST_PATH_UTILITY, subpath);
+ }
+
+ @Test
+ public void testGetBasicInfoRoot() throws Exception {
+ // Test root basics
+ Response rootResponse = nsSummaryEndpoint.getBasicInfo(ROOT_PATH);
+ NamespaceSummaryResponse rootResponseObj =
+ (NamespaceSummaryResponse) rootResponse.getEntity();
+ assertEquals(EntityType.ROOT, rootResponseObj.getEntityType());
+ assertEquals(2, rootResponseObj.getCountStats().getNumVolume());
+ assertEquals(4, rootResponseObj.getCountStats().getNumBucket());
+ assertEquals(9, rootResponseObj.getCountStats().getNumTotalKey());
+ }
+
+ @Test
+ public void testGetBasicInfoVol() throws Exception {
+ // Test volume basics
+ Response volResponse = nsSummaryEndpoint.getBasicInfo(VOL_PATH);
+ NamespaceSummaryResponse volResponseObj =
+ (NamespaceSummaryResponse) volResponse.getEntity();
+ assertEquals(EntityType.VOLUME,
+ volResponseObj.getEntityType());
+ assertEquals(2, volResponseObj.getCountStats().getNumBucket());
+ assertEquals(5, volResponseObj.getCountStats().getNumTotalKey());
+ assertEquals("TestUser", ((VolumeObjectDBInfo) volResponseObj.
+ getObjectDBInfo()).getAdmin());
+ assertEquals("TestUser", ((VolumeObjectDBInfo) volResponseObj.
+ getObjectDBInfo()).getOwner());
+ assertEquals("vol", volResponseObj.getObjectDBInfo().getName());
+ assertEquals(2097152, volResponseObj.getObjectDBInfo().getQuotaInBytes());
+ assertEquals(-1, volResponseObj.getObjectDBInfo().getQuotaInNamespace());
Review Comment:
Changed thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1504083228
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestNSSummaryEndpointWithOBS.java:
##########
@@ -0,0 +1,1133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor.ONE;
+import static org.apache.hadoop.ozone.om.helpers.QuotaUtil.getReplicatedSize;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.StorageType;
+import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.container.ContainerReplica;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.handlers.BucketHandler;
+import org.apache.hadoop.ozone.recon.api.handlers.EntityHandler;
+import org.apache.hadoop.ozone.recon.api.types.BucketObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NamespaceSummaryResponse;
+import org.apache.hadoop.ozone.recon.api.types.QuotaUsageResponse;
+import org.apache.hadoop.ozone.recon.api.types.ResponseStatus;
+import org.apache.hadoop.ozone.recon.api.types.VolumeObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.FileSizeDistributionResponse;
+import org.apache.hadoop.ozone.recon.common.CommonUtils;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.scm.ReconNodeManager;
+import org.apache.hadoop.ozone.recon.scm.ReconStorageContainerManagerFacade;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.StorageContainerServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.spi.impl.StorageContainerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.tasks.NSSummaryTaskWithOBS;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import javax.ws.rs.core.Response;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.HashSet;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.apache.hadoop.hdds.protocol.MockDatanodeDetails.randomDatanodeDetails;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProvider;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for NSSummary REST APIs with OBS.
+ * Testing is done on a simple object store model with a flat hierarchy:
+ * vol
+ * / \
+ * bucket1 bucket2
+ * | |
+ * file1 file2
+ *
+ * This tests the Rest APIs for NSSummary in the context of OBS buckets,
+ * focusing on disk usage, quota usage, and file size distribution.
+ */
+public class TestNSSummaryEndpointWithOBS {
+ @TempDir
+ private Path temporaryFolder;
+
+ private ReconOMMetadataManager reconOMMetadataManager;
+ private NSSummaryEndpoint nsSummaryEndpoint;
+ private OzoneConfiguration conf;
+ private CommonUtils commonUtils;
+
+ private static final String TEST_PATH_UTILITY =
+ "/vol1/buck1/a/b/c/d/e/file1.txt";
+ private static final String PARENT_DIR = "vol1/buck1/a/b/c/d/e";
+ private static final String[] TEST_NAMES =
+ new String[]{"vol1", "buck1", "a", "b", "c", "d", "e", "file1.txt"};
+ private static final String TEST_KEY_NAMES = "a/b/c/d/e/file1.txt";
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String VOL_TWO = "vol2";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String BUCKET_THREE = "bucket3";
+ private static final String BUCKET_FOUR = "bucket4";
+ private static final String KEY_ONE = "file1";
+ private static final String KEY_TWO = "file2";
+ private static final String KEY_THREE = "file3";
+ private static final String KEY_FOUR = "file4";
+ private static final String KEY_FIVE = "file5";
+ private static final String KEY_SIX = "file6";
+ private static final String KEY_SEVEN = "file7";
+ private static final String KEY_EIGHT = "file8";
+ private static final String KEY_NINE = "file9";
+ private static final String KEY_TEN = "file10";
+ private static final String KEY_ELEVEN = "file11";
+ private static final String MULTI_BLOCK_KEY = "file7";
+ private static final String MULTI_BLOCK_FILE = "file7";
+
+ private static final String FILE_ONE = "file1";
+ private static final String FILE_TWO = "file2";
+ private static final String FILE_THREE = "file3";
+ private static final String FILE_FOUR = "file4";
+ private static final String FILE_FIVE = "file5";
+ private static final String FILE_SIX = "file6";
+ private static final String FILE_SEVEN = "file7";
+ private static final String FILE_EIGHT = "file8";
+ private static final String FILE_NINE = "file9";
+ private static final String FILE_TEN = "file10";
+ private static final String FILE_ELEVEN = "file11";
+ // objects IDs
+ private static final long PARENT_OBJECT_ID_ZERO = 0L;
+ private static final long VOL_OBJECT_ID = 0L;
+ private static final long BUCKET_ONE_OBJECT_ID = 1L;
+ private static final long BUCKET_TWO_OBJECT_ID = 2L;
+ private static final long KEY_ONE_OBJECT_ID = 3L;
+ private static final long KEY_TWO_OBJECT_ID = 5L;
+ private static final long KEY_FOUR_OBJECT_ID = 6L;
+ private static final long KEY_THREE_OBJECT_ID = 8L;
+ private static final long KEY_FIVE_OBJECT_ID = 9L;
+ private static final long KEY_SIX_OBJECT_ID = 10L;
+ private static final long MULTI_BLOCK_KEY_OBJECT_ID = 13L;
+ private static final long KEY_SEVEN_OBJECT_ID = 13L;
+ private static final long VOL_TWO_OBJECT_ID = 14L;
+ private static final long BUCKET_THREE_OBJECT_ID = 15L;
+ private static final long BUCKET_FOUR_OBJECT_ID = 16L;
+ private static final long KEY_EIGHT_OBJECT_ID = 17L;
+ private static final long KEY_NINE_OBJECT_ID = 19L;
+ private static final long KEY_TEN_OBJECT_ID = 20L;
+ private static final long KEY_ELEVEN_OBJECT_ID = 21L;
+
+ // container IDs
+ private static final long CONTAINER_ONE_ID = 1L;
+ private static final long CONTAINER_TWO_ID = 2L;
+ private static final long CONTAINER_THREE_ID = 3L;
+ private static final long CONTAINER_FOUR_ID = 4L;
+ private static final long CONTAINER_FIVE_ID = 5L;
+ private static final long CONTAINER_SIX_ID = 6L;
+
+ // replication factors
+ private static final int CONTAINER_ONE_REPLICA_COUNT = 3;
+ private static final int CONTAINER_TWO_REPLICA_COUNT = 2;
+ private static final int CONTAINER_THREE_REPLICA_COUNT = 4;
+ private static final int CONTAINER_FOUR_REPLICA_COUNT = 5;
+ private static final int CONTAINER_FIVE_REPLICA_COUNT = 2;
+ private static final int CONTAINER_SIX_REPLICA_COUNT = 3;
+
+ // block lengths
+ private static final long BLOCK_ONE_LENGTH = 1000L;
+ private static final long BLOCK_TWO_LENGTH = 2000L;
+ private static final long BLOCK_THREE_LENGTH = 3000L;
+ private static final long BLOCK_FOUR_LENGTH = 4000L;
+ private static final long BLOCK_FIVE_LENGTH = 5000L;
+ private static final long BLOCK_SIX_LENGTH = 6000L;
+
+ // data size in bytes
+ private static final long KEY_ONE_SIZE = 500L; // bin 0
+ private static final long KEY_TWO_SIZE = OzoneConsts.KB + 1; // bin 1
+ private static final long KEY_THREE_SIZE = 4 * OzoneConsts.KB + 1; // bin 3
+ private static final long KEY_FOUR_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_FIVE_SIZE = 100L; // bin 0
+ private static final long KEY_SIX_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_SEVEN_SIZE = 4 * OzoneConsts.KB + 1;
+ private static final long KEY_EIGHT_SIZE = OzoneConsts.KB + 1; // bin 1
+ private static final long KEY_NINE_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_TEN_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_ELEVEN_SIZE = OzoneConsts.KB + 1; // bin 1
+
+ private static final long FILE1_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_ONE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE2_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_TWO_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE3_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_THREE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE4_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_FOUR_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE5_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_FIVE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE6_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_SIX_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));;
+ private static final long FILE7_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_SEVEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));;
+ private static final long FILE8_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_EIGHT_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE9_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_NINE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE10_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_TEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE11_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_ELEVEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+
+ private static final long MULTI_BLOCK_KEY_SIZE_WITH_REPLICA
+ = FILE7_SIZE_WITH_REPLICA;
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_ROOT
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA
+ + FILE4_SIZE_WITH_REPLICA
+ + FILE5_SIZE_WITH_REPLICA
+ + FILE8_SIZE_WITH_REPLICA
+ + FILE9_SIZE_WITH_REPLICA
+ + FILE10_SIZE_WITH_REPLICA
+ + FILE11_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_VOL
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA
+ + FILE4_SIZE_WITH_REPLICA
+ + FILE5_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_BUCKET1
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_KEY
+ = FILE4_SIZE_WITH_REPLICA;
+
+ // quota in bytes
+ private static final long ROOT_QUOTA = 2 * (2 * OzoneConsts.MB);
+ private static final long VOL_QUOTA = 2 * OzoneConsts.MB;
+ private static final long VOL_TWO_QUOTA = 2 * OzoneConsts.MB;
+ private static final long BUCKET_ONE_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_TWO_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_THREE_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_FOUR_QUOTA = OzoneConsts.MB;
+
+ // mock client's path requests
+ private static final String TEST_USER = "TestUser";
+ private static final String ROOT_PATH = "/";
+ private static final String VOL_PATH = "/vol";
+ private static final String VOL_TWO_PATH = "/vol2";
+ private static final String BUCKET_ONE_PATH = "/vol/bucket1";
+ private static final String BUCKET_TWO_PATH = "/vol/bucket2";
+ private static final String KEY_PATH = "/vol/bucket2/file4";
+ private static final String MULTI_BLOCK_KEY_PATH = "/vol/bucket1/file7";
+ private static final String INVALID_PATH = "/vol/path/not/found";
+
+ // some expected answers
+ private static final long ROOT_DATA_SIZE =
+ KEY_ONE_SIZE + KEY_TWO_SIZE + KEY_THREE_SIZE + KEY_FOUR_SIZE +
+ KEY_FIVE_SIZE + KEY_EIGHT_SIZE + KEY_NINE_SIZE + KEY_TEN_SIZE +
+ KEY_ELEVEN_SIZE;
+ private static final long VOL_DATA_SIZE = KEY_ONE_SIZE + KEY_TWO_SIZE +
+ KEY_THREE_SIZE + KEY_FOUR_SIZE + KEY_FIVE_SIZE;
+
+ private static final long VOL_TWO_DATA_SIZE =
+ KEY_EIGHT_SIZE + KEY_NINE_SIZE + KEY_TEN_SIZE + KEY_ELEVEN_SIZE;
+
+ private static final long BUCKET_ONE_DATA_SIZE = KEY_ONE_SIZE + KEY_TWO_SIZE +
+ KEY_THREE_SIZE;
+
+ private static final long BUCKET_TWO_DATA_SIZE =
+ KEY_FOUR_SIZE + KEY_FIVE_SIZE;
+
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ conf = new OzoneConfiguration();
+ OMMetadataManager omMetadataManager = initializeNewOmMetadataManager(
+ Files.createDirectory(temporaryFolder.resolve(
+ "JunitOmDBDir")).toFile(), conf);
+ OzoneManagerServiceProviderImpl ozoneManagerServiceProvider =
+ getMockOzoneManagerServiceProvider();
+ reconOMMetadataManager = getTestReconOmMetadataManager(omMetadataManager,
+ Files.createDirectory(temporaryFolder.resolve(
+ "omMetadatDir")).toFile());
+
+ ReconTestInjector reconTestInjector =
+ new ReconTestInjector.Builder(temporaryFolder.toFile())
+ .withReconOm(reconOMMetadataManager)
+ .withOmServiceProvider(ozoneManagerServiceProvider)
+ .withReconSqlDb()
+ .withContainerDB()
+ .addBinding(OzoneStorageContainerManager.class,
+ getMockReconSCM())
+ .addBinding(StorageContainerServiceProvider.class,
+ mock(StorageContainerServiceProviderImpl.class))
+ .addBinding(NSSummaryEndpoint.class)
+ .build();
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager =
+ reconTestInjector.getInstance(ReconNamespaceSummaryManager.class);
+ nsSummaryEndpoint = reconTestInjector.getInstance(NSSummaryEndpoint.class);
+
+ // populate OM DB and reprocess into Recon RocksDB
+ populateOMDB();
+ NSSummaryTaskWithOBS nsSummaryTaskWithOBS =
+ new NSSummaryTaskWithOBS(reconNamespaceSummaryManager,
+ reconOMMetadataManager, conf);
+ nsSummaryTaskWithOBS.reprocessWithOBS(reconOMMetadataManager);
+ commonUtils = new CommonUtils();
+ }
+
+ @Test
+ public void testUtility() {
+ String[] names = EntityHandler.parseRequestPath(TEST_PATH_UTILITY);
+ assertArrayEquals(TEST_NAMES, names);
+ String keyName = BucketHandler.getKeyName(names);
+ assertEquals(TEST_KEY_NAMES, keyName);
+ String subpath = BucketHandler.buildSubpath(PARENT_DIR, "file1.txt");
+ assertEquals(TEST_PATH_UTILITY, subpath);
+ }
+
+ @Test
+ public void testGetBasicInfoRoot() throws Exception {
+ // Test root basics
+ Response rootResponse = nsSummaryEndpoint.getBasicInfo(ROOT_PATH);
+ NamespaceSummaryResponse rootResponseObj =
+ (NamespaceSummaryResponse) rootResponse.getEntity();
+ assertEquals(EntityType.ROOT, rootResponseObj.getEntityType());
+ assertEquals(2, rootResponseObj.getCountStats().getNumVolume());
+ assertEquals(4, rootResponseObj.getCountStats().getNumBucket());
+ assertEquals(9, rootResponseObj.getCountStats().getNumTotalKey());
+ }
+
+ @Test
+ public void testGetBasicInfoVol() throws Exception {
+ // Test volume basics
+ Response volResponse = nsSummaryEndpoint.getBasicInfo(VOL_PATH);
+ NamespaceSummaryResponse volResponseObj =
+ (NamespaceSummaryResponse) volResponse.getEntity();
+ assertEquals(EntityType.VOLUME,
+ volResponseObj.getEntityType());
+ assertEquals(2, volResponseObj.getCountStats().getNumBucket());
+ assertEquals(5, volResponseObj.getCountStats().getNumTotalKey());
+ assertEquals("TestUser", ((VolumeObjectDBInfo) volResponseObj.
+ getObjectDBInfo()).getAdmin());
+ assertEquals("TestUser", ((VolumeObjectDBInfo) volResponseObj.
+ getObjectDBInfo()).getOwner());
+ assertEquals("vol", volResponseObj.getObjectDBInfo().getName());
+ assertEquals(2097152, volResponseObj.getObjectDBInfo().getQuotaInBytes());
+ assertEquals(-1, volResponseObj.getObjectDBInfo().getQuotaInNamespace());
+ }
+
+ @Test
+ public void testGetBasicInfoBucketOne() throws Exception {
+ // Test bucket 1's basics
+ Response bucketOneResponse =
+ nsSummaryEndpoint.getBasicInfo(BUCKET_ONE_PATH);
+ NamespaceSummaryResponse bucketOneObj =
+ (NamespaceSummaryResponse) bucketOneResponse.getEntity();
+ assertEquals(EntityType.BUCKET, bucketOneObj.getEntityType());
+ assertEquals(3, bucketOneObj.getCountStats().getNumTotalKey());
+ assertEquals("vol",
+ ((BucketObjectDBInfo) bucketOneObj.getObjectDBInfo()).getVolumeName());
+ assertEquals(StorageType.DISK,
+ ((BucketObjectDBInfo)
+ bucketOneObj.getObjectDBInfo()).getStorageType());
+ assertEquals(getBucketLayout(),
+ ((BucketObjectDBInfo)
+ bucketOneObj.getObjectDBInfo()).getBucketLayout());
+ assertEquals("bucket1",
+ ((BucketObjectDBInfo) bucketOneObj.getObjectDBInfo()).getName());
+ }
+
+ @Test
+ public void testGetBasicInfoBucketTwo() throws Exception {
+ // Test bucket 2's basics
+ commonUtils.testNSSummaryBasicInfoBucketTwo(
+ BucketLayout.OBJECT_STORE,
+ nsSummaryEndpoint);
+ }
+
+ @Test
+ public void testGetBasicInfoNoPath() throws Exception {
+ // Test invalid path
+ commonUtils.testNSSummaryBasicInfoNoPath(nsSummaryEndpoint);
+ }
+
+ @Test
+ public void testGetBasicInfoKey() throws Exception {
+ // Test key
+ commonUtils.testNSSummaryBasicInfoKey(nsSummaryEndpoint);
+ }
+
+ @Test
+ public void testDiskUsageRoot() throws Exception {
+ // root level DU
+ Response rootResponse = nsSummaryEndpoint.getDiskUsage(ROOT_PATH,
+ false, false);
+ DUResponse duRootRes = (DUResponse) rootResponse.getEntity();
+ assertEquals(2, duRootRes.getCount());
+ List<DUResponse.DiskUsage> duRootData = duRootRes.getDuData();
+ // sort based on subpath
+ Collections.sort(duRootData,
+ Comparator.comparing(DUResponse.DiskUsage::getSubpath));
+ DUResponse.DiskUsage duVol1 = duRootData.get(0);
+ DUResponse.DiskUsage duVol2 = duRootData.get(1);
+ assertEquals(VOL_PATH, duVol1.getSubpath());
+ assertEquals(VOL_TWO_PATH, duVol2.getSubpath());
+ assertEquals(VOL_DATA_SIZE, duVol1.getSize());
+ assertEquals(VOL_TWO_DATA_SIZE, duVol2.getSize());
+ }
+
+ @Test
+ public void testDiskUsageVolume() throws Exception {
+ // volume level DU
+ Response volResponse = nsSummaryEndpoint.getDiskUsage(VOL_PATH,
+ false, false);
+ DUResponse duVolRes = (DUResponse) volResponse.getEntity();
+ assertEquals(2, duVolRes.getCount());
+ List<DUResponse.DiskUsage> duData = duVolRes.getDuData();
+ // sort based on subpath
+ Collections.sort(duData,
+ Comparator.comparing(DUResponse.DiskUsage::getSubpath));
+ DUResponse.DiskUsage duBucket1 = duData.get(0);
+ DUResponse.DiskUsage duBucket2 = duData.get(1);
+ assertEquals(BUCKET_ONE_PATH, duBucket1.getSubpath());
+ assertEquals(BUCKET_TWO_PATH, duBucket2.getSubpath());
+ assertEquals(BUCKET_ONE_DATA_SIZE, duBucket1.getSize());
+ assertEquals(BUCKET_TWO_DATA_SIZE, duBucket2.getSize());
+ }
+
+ @Test
+ public void testDiskUsageBucket() throws Exception {
+ // bucket level DU
+ Response bucketResponse = nsSummaryEndpoint.getDiskUsage(BUCKET_ONE_PATH,
+ false, false);
+ DUResponse duBucketResponse = (DUResponse) bucketResponse.getEntity();
+ // There are no sub-paths under this OBS bucket.
+ assertEquals(0, duBucketResponse.getCount());
+ assertEquals(BUCKET_ONE_DATA_SIZE, duBucketResponse.getSize());
+ }
+
+ @Test
+ public void testDiskUsageKey() throws Exception {
+ // key level DU
+ Response keyResponse = nsSummaryEndpoint.getDiskUsage(KEY_PATH,
+ false, false);
+ DUResponse keyObj = (DUResponse) keyResponse.getEntity();
+ assertEquals(0, keyObj.getCount());
+ assertEquals(KEY_FOUR_SIZE, keyObj.getSize());
+ }
Review Comment:
`testGetBasicInfoKey` triggers the `/summary` endpoint whereas `testDiskUsageKey` over here triggers the `/du` endpoint, which display completely different data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on a diff in pull request #4245: HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets.
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1107482893
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTask.java:
##########
@@ -100,6 +104,12 @@ public Pair<String, Boolean> process(OMUpdateEventBatch events) {
} else {
LOG.error("processWithFSO failed.");
}
+ if (success) {
+ success = nsSummaryTaskWithOBS.reprocessWithOBS(reconOMMetadataManager);
Review Comment:
```suggestion
success = nsSummaryTaskWithOBS.processWithOBS(events);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1469200159
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling OBS buckets NameSpaceSummaries.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ @Override
+ public EntityType determineKeyPath(String keyName)
+ throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ iterator.seek(key);
+ if (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+ if (dbKey.equals(key)) {
+ return EntityType.KEY;
+ }
+ }
+ return EntityType.UNKNOWN;
+ }
+
+ /**
+ * This method handles disk usage of direct keys.
+ *
+ * @param parentId parent OBS bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * for each direct key's DU
+ * @param listFile if listFile is enabled, append key DU as a children
+ * keys
+ * @param duData the current DU data
+ * @param normalizedPath the normalized path request
+ * @return the total DU of all direct keys
+ * @throws IOException IOE
+ */
+ @Override
+ public long handleDirectKeys(long parentId, boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ String normalizedPath) throws IOException {
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+ long keyDataSizeWithReplica = 0L;
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ String seekPrefix = OM_KEY_PREFIX +
+ vol +
+ OM_KEY_PREFIX +
+ bucket +
+ OM_KEY_PREFIX;
+
+ NSSummary nsSummary = getReconNamespaceSummaryManager()
+ .getNSSummary(parentId);
+ // Handle the case of an empty bucket.
+ if (nsSummary == null) {
+ return 0;
+ }
+
+ iterator.seek(seekPrefix);
+
+ while (iterator.hasNext()) {
+ // KeyName : OmKeyInfo-Object
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+
+ // Exit loop if the key doesn't match the seekPrefix.
+ if (!dbKey.startsWith(seekPrefix)) {
+ break;
+ }
+
+ OmKeyInfo keyInfo = kv.getValue();
+ if (keyInfo != null) {
+ DUResponse.DiskUsage diskUsage = new DUResponse.DiskUsage();
+ String objectName = keyInfo.getKeyName();
+ diskUsage.setSubpath(objectName);
+ diskUsage.setKey(true);
+ diskUsage.setSize(keyInfo.getDataSize());
+
+ if (withReplica) {
+ long keyDU = keyInfo.getReplicatedSize();
+ keyDataSizeWithReplica += keyDU;
+ diskUsage.setSizeWithReplica(keyDU);
+ }
+ // List all the keys for the OBS bucket if requested.
+ if (listFile) {
+ duData.add(diskUsage);
+ }
+ }
+ }
+
+ return keyDataSizeWithReplica;
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public long calculateDUUnderObject(long parentId)
+ throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public long getDirObjectId(String[] names) throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public long getDirObjectId(String[] names, int cutoff) throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Returns the keyInfo object from the KEY table.
+ * @return OmKeyInfo
+ */
+ @Override
+ public OmKeyInfo getKeyInfo(String[] names) throws IOException {
+ String ozoneKey = OM_KEY_PREFIX;
+ ozoneKey += String.join(OM_KEY_PREFIX, names);
Review Comment:
I have checked the utility classes in recon, and currently there is no utility method available for this specific line. All other handlers within the codebase also utilize the same join method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1469209088
##########
qodana.yaml:
##########
Review Comment:
I believe the generated file got added mistakenly, I have removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on PR #4245:
URL: https://github.com/apache/ozone/pull/4245#issuecomment-1790595726
Thank you for your response, @xBis7
**This patch will not implement prefix-based search**; its primary purpose is to list the keys within an OBS bucket. The support for prefix-based searching in OBS buckets will be added later when I have the available bandwidth to complete the design doc.
We received several support tickets from customers who reported issues with NSSummary not functioning correctly due to the lack of OBS support. Therefore, the goal of this patch is to provide the essential functionality to address this issue and eliminate any confusion for now. This patch offers a straightforward implementation of how we can generate NSSummary for an OBS layout bucket.
The addition of prefix-based OBS filtering can be considered as an enhancement in subsequent updates and will be tracked by this JIRA :- https://issues.apache.org/jira/browse/HDDS-9535
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on PR #4245:
URL: https://github.com/apache/ozone/pull/4245#issuecomment-1790131287
@adoroszlai @devmadhuu @GeorgeJahad Could you please take a look!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "dombizita (via GitHub)" <gi...@apache.org>.
dombizita commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1495572410
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestNSSummaryEndpointWithOBS.java:
##########
@@ -0,0 +1,1133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor.ONE;
+import static org.apache.hadoop.ozone.om.helpers.QuotaUtil.getReplicatedSize;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.StorageType;
+import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.container.ContainerReplica;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.handlers.BucketHandler;
+import org.apache.hadoop.ozone.recon.api.handlers.EntityHandler;
+import org.apache.hadoop.ozone.recon.api.types.BucketObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NamespaceSummaryResponse;
+import org.apache.hadoop.ozone.recon.api.types.QuotaUsageResponse;
+import org.apache.hadoop.ozone.recon.api.types.ResponseStatus;
+import org.apache.hadoop.ozone.recon.api.types.VolumeObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.FileSizeDistributionResponse;
+import org.apache.hadoop.ozone.recon.common.CommonUtils;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.scm.ReconNodeManager;
+import org.apache.hadoop.ozone.recon.scm.ReconStorageContainerManagerFacade;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.StorageContainerServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.spi.impl.StorageContainerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.tasks.NSSummaryTaskWithOBS;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import javax.ws.rs.core.Response;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.HashSet;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.apache.hadoop.hdds.protocol.MockDatanodeDetails.randomDatanodeDetails;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProvider;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for NSSummary REST APIs with OBS.
+ * Testing is done on a simple object store model with a flat hierarchy:
+ * vol
+ * / \
+ * bucket1 bucket2
+ * | |
+ * file1 file2
+ *
+ * This tests the Rest APIs for NSSummary in the context of OBS buckets,
+ * focusing on disk usage, quota usage, and file size distribution.
+ */
+public class TestNSSummaryEndpointWithOBS {
+ @TempDir
+ private Path temporaryFolder;
+
+ private ReconOMMetadataManager reconOMMetadataManager;
+ private NSSummaryEndpoint nsSummaryEndpoint;
+ private OzoneConfiguration conf;
+ private CommonUtils commonUtils;
+
+ private static final String TEST_PATH_UTILITY =
+ "/vol1/buck1/a/b/c/d/e/file1.txt";
+ private static final String PARENT_DIR = "vol1/buck1/a/b/c/d/e";
+ private static final String[] TEST_NAMES =
+ new String[]{"vol1", "buck1", "a", "b", "c", "d", "e", "file1.txt"};
+ private static final String TEST_KEY_NAMES = "a/b/c/d/e/file1.txt";
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String VOL_TWO = "vol2";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String BUCKET_THREE = "bucket3";
+ private static final String BUCKET_FOUR = "bucket4";
+ private static final String KEY_ONE = "file1";
+ private static final String KEY_TWO = "file2";
+ private static final String KEY_THREE = "file3";
+ private static final String KEY_FOUR = "file4";
+ private static final String KEY_FIVE = "file5";
+ private static final String KEY_SIX = "file6";
+ private static final String KEY_SEVEN = "file7";
+ private static final String KEY_EIGHT = "file8";
+ private static final String KEY_NINE = "file9";
+ private static final String KEY_TEN = "file10";
+ private static final String KEY_ELEVEN = "file11";
+ private static final String MULTI_BLOCK_KEY = "file7";
+ private static final String MULTI_BLOCK_FILE = "file7";
+
+ private static final String FILE_ONE = "file1";
+ private static final String FILE_TWO = "file2";
+ private static final String FILE_THREE = "file3";
+ private static final String FILE_FOUR = "file4";
+ private static final String FILE_FIVE = "file5";
+ private static final String FILE_SIX = "file6";
+ private static final String FILE_SEVEN = "file7";
Review Comment:
Same here: these two are never used, why?
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestNSSummaryEndpointWithOBS.java:
##########
@@ -0,0 +1,1133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor.ONE;
+import static org.apache.hadoop.ozone.om.helpers.QuotaUtil.getReplicatedSize;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.StorageType;
+import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.container.ContainerReplica;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.handlers.BucketHandler;
+import org.apache.hadoop.ozone.recon.api.handlers.EntityHandler;
+import org.apache.hadoop.ozone.recon.api.types.BucketObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NamespaceSummaryResponse;
+import org.apache.hadoop.ozone.recon.api.types.QuotaUsageResponse;
+import org.apache.hadoop.ozone.recon.api.types.ResponseStatus;
+import org.apache.hadoop.ozone.recon.api.types.VolumeObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.FileSizeDistributionResponse;
+import org.apache.hadoop.ozone.recon.common.CommonUtils;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.scm.ReconNodeManager;
+import org.apache.hadoop.ozone.recon.scm.ReconStorageContainerManagerFacade;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.StorageContainerServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.spi.impl.StorageContainerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.tasks.NSSummaryTaskWithOBS;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import javax.ws.rs.core.Response;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.HashSet;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.apache.hadoop.hdds.protocol.MockDatanodeDetails.randomDatanodeDetails;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProvider;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for NSSummary REST APIs with OBS.
+ * Testing is done on a simple object store model with a flat hierarchy:
+ * vol
+ * / \
+ * bucket1 bucket2
+ * | |
+ * file1 file2
+ *
+ * This tests the Rest APIs for NSSummary in the context of OBS buckets,
+ * focusing on disk usage, quota usage, and file size distribution.
+ */
+public class TestNSSummaryEndpointWithOBS {
+ @TempDir
+ private Path temporaryFolder;
+
+ private ReconOMMetadataManager reconOMMetadataManager;
+ private NSSummaryEndpoint nsSummaryEndpoint;
+ private OzoneConfiguration conf;
+ private CommonUtils commonUtils;
+
+ private static final String TEST_PATH_UTILITY =
+ "/vol1/buck1/a/b/c/d/e/file1.txt";
+ private static final String PARENT_DIR = "vol1/buck1/a/b/c/d/e";
+ private static final String[] TEST_NAMES =
+ new String[]{"vol1", "buck1", "a", "b", "c", "d", "e", "file1.txt"};
+ private static final String TEST_KEY_NAMES = "a/b/c/d/e/file1.txt";
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String VOL_TWO = "vol2";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String BUCKET_THREE = "bucket3";
+ private static final String BUCKET_FOUR = "bucket4";
+ private static final String KEY_ONE = "file1";
+ private static final String KEY_TWO = "file2";
+ private static final String KEY_THREE = "file3";
+ private static final String KEY_FOUR = "file4";
+ private static final String KEY_FIVE = "file5";
+ private static final String KEY_SIX = "file6";
+ private static final String KEY_SEVEN = "file7";
Review Comment:
These two are never used, why?
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestNSSummaryEndpointWithOBS.java:
##########
@@ -0,0 +1,1133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor.ONE;
+import static org.apache.hadoop.ozone.om.helpers.QuotaUtil.getReplicatedSize;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.StorageType;
+import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.container.ContainerReplica;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.handlers.BucketHandler;
+import org.apache.hadoop.ozone.recon.api.handlers.EntityHandler;
+import org.apache.hadoop.ozone.recon.api.types.BucketObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NamespaceSummaryResponse;
+import org.apache.hadoop.ozone.recon.api.types.QuotaUsageResponse;
+import org.apache.hadoop.ozone.recon.api.types.ResponseStatus;
+import org.apache.hadoop.ozone.recon.api.types.VolumeObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.FileSizeDistributionResponse;
+import org.apache.hadoop.ozone.recon.common.CommonUtils;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.scm.ReconNodeManager;
+import org.apache.hadoop.ozone.recon.scm.ReconStorageContainerManagerFacade;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.StorageContainerServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.spi.impl.StorageContainerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.tasks.NSSummaryTaskWithOBS;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import javax.ws.rs.core.Response;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.HashSet;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.apache.hadoop.hdds.protocol.MockDatanodeDetails.randomDatanodeDetails;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProvider;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for NSSummary REST APIs with OBS.
+ * Testing is done on a simple object store model with a flat hierarchy:
+ * vol
+ * / \
+ * bucket1 bucket2
+ * | |
+ * file1 file2
+ *
+ * This tests the Rest APIs for NSSummary in the context of OBS buckets,
+ * focusing on disk usage, quota usage, and file size distribution.
+ */
+public class TestNSSummaryEndpointWithOBS {
+ @TempDir
+ private Path temporaryFolder;
+
+ private ReconOMMetadataManager reconOMMetadataManager;
+ private NSSummaryEndpoint nsSummaryEndpoint;
+ private OzoneConfiguration conf;
+ private CommonUtils commonUtils;
+
+ private static final String TEST_PATH_UTILITY =
+ "/vol1/buck1/a/b/c/d/e/file1.txt";
+ private static final String PARENT_DIR = "vol1/buck1/a/b/c/d/e";
+ private static final String[] TEST_NAMES =
+ new String[]{"vol1", "buck1", "a", "b", "c", "d", "e", "file1.txt"};
+ private static final String TEST_KEY_NAMES = "a/b/c/d/e/file1.txt";
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String VOL_TWO = "vol2";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String BUCKET_THREE = "bucket3";
+ private static final String BUCKET_FOUR = "bucket4";
+ private static final String KEY_ONE = "file1";
+ private static final String KEY_TWO = "file2";
+ private static final String KEY_THREE = "file3";
+ private static final String KEY_FOUR = "file4";
+ private static final String KEY_FIVE = "file5";
+ private static final String KEY_SIX = "file6";
+ private static final String KEY_SEVEN = "file7";
+ private static final String KEY_EIGHT = "file8";
+ private static final String KEY_NINE = "file9";
+ private static final String KEY_TEN = "file10";
+ private static final String KEY_ELEVEN = "file11";
+ private static final String MULTI_BLOCK_KEY = "file7";
+ private static final String MULTI_BLOCK_FILE = "file7";
+
+ private static final String FILE_ONE = "file1";
+ private static final String FILE_TWO = "file2";
+ private static final String FILE_THREE = "file3";
+ private static final String FILE_FOUR = "file4";
+ private static final String FILE_FIVE = "file5";
+ private static final String FILE_SIX = "file6";
+ private static final String FILE_SEVEN = "file7";
+ private static final String FILE_EIGHT = "file8";
+ private static final String FILE_NINE = "file9";
+ private static final String FILE_TEN = "file10";
+ private static final String FILE_ELEVEN = "file11";
+ // objects IDs
+ private static final long PARENT_OBJECT_ID_ZERO = 0L;
+ private static final long VOL_OBJECT_ID = 0L;
+ private static final long BUCKET_ONE_OBJECT_ID = 1L;
+ private static final long BUCKET_TWO_OBJECT_ID = 2L;
+ private static final long KEY_ONE_OBJECT_ID = 3L;
+ private static final long KEY_TWO_OBJECT_ID = 5L;
+ private static final long KEY_FOUR_OBJECT_ID = 6L;
+ private static final long KEY_THREE_OBJECT_ID = 8L;
+ private static final long KEY_FIVE_OBJECT_ID = 9L;
+ private static final long KEY_SIX_OBJECT_ID = 10L;
+ private static final long MULTI_BLOCK_KEY_OBJECT_ID = 13L;
+ private static final long KEY_SEVEN_OBJECT_ID = 13L;
+ private static final long VOL_TWO_OBJECT_ID = 14L;
+ private static final long BUCKET_THREE_OBJECT_ID = 15L;
+ private static final long BUCKET_FOUR_OBJECT_ID = 16L;
+ private static final long KEY_EIGHT_OBJECT_ID = 17L;
+ private static final long KEY_NINE_OBJECT_ID = 19L;
+ private static final long KEY_TEN_OBJECT_ID = 20L;
+ private static final long KEY_ELEVEN_OBJECT_ID = 21L;
+
+ // container IDs
+ private static final long CONTAINER_ONE_ID = 1L;
+ private static final long CONTAINER_TWO_ID = 2L;
+ private static final long CONTAINER_THREE_ID = 3L;
+ private static final long CONTAINER_FOUR_ID = 4L;
+ private static final long CONTAINER_FIVE_ID = 5L;
+ private static final long CONTAINER_SIX_ID = 6L;
+
+ // replication factors
+ private static final int CONTAINER_ONE_REPLICA_COUNT = 3;
+ private static final int CONTAINER_TWO_REPLICA_COUNT = 2;
+ private static final int CONTAINER_THREE_REPLICA_COUNT = 4;
+ private static final int CONTAINER_FOUR_REPLICA_COUNT = 5;
+ private static final int CONTAINER_FIVE_REPLICA_COUNT = 2;
+ private static final int CONTAINER_SIX_REPLICA_COUNT = 3;
+
+ // block lengths
+ private static final long BLOCK_ONE_LENGTH = 1000L;
+ private static final long BLOCK_TWO_LENGTH = 2000L;
+ private static final long BLOCK_THREE_LENGTH = 3000L;
+ private static final long BLOCK_FOUR_LENGTH = 4000L;
+ private static final long BLOCK_FIVE_LENGTH = 5000L;
+ private static final long BLOCK_SIX_LENGTH = 6000L;
+
+ // data size in bytes
+ private static final long KEY_ONE_SIZE = 500L; // bin 0
+ private static final long KEY_TWO_SIZE = OzoneConsts.KB + 1; // bin 1
+ private static final long KEY_THREE_SIZE = 4 * OzoneConsts.KB + 1; // bin 3
+ private static final long KEY_FOUR_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_FIVE_SIZE = 100L; // bin 0
+ private static final long KEY_SIX_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_SEVEN_SIZE = 4 * OzoneConsts.KB + 1;
+ private static final long KEY_EIGHT_SIZE = OzoneConsts.KB + 1; // bin 1
+ private static final long KEY_NINE_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_TEN_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_ELEVEN_SIZE = OzoneConsts.KB + 1; // bin 1
+
+ private static final long FILE1_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_ONE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE2_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_TWO_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE3_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_THREE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE4_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_FOUR_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE5_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_FIVE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE6_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_SIX_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));;
+ private static final long FILE7_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_SEVEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));;
+ private static final long FILE8_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_EIGHT_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE9_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_NINE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE10_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_TEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE11_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_ELEVEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+
+ private static final long MULTI_BLOCK_KEY_SIZE_WITH_REPLICA
+ = FILE7_SIZE_WITH_REPLICA;
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_ROOT
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA
+ + FILE4_SIZE_WITH_REPLICA
+ + FILE5_SIZE_WITH_REPLICA
+ + FILE8_SIZE_WITH_REPLICA
+ + FILE9_SIZE_WITH_REPLICA
+ + FILE10_SIZE_WITH_REPLICA
+ + FILE11_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_VOL
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA
+ + FILE4_SIZE_WITH_REPLICA
+ + FILE5_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_BUCKET1
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_KEY
+ = FILE4_SIZE_WITH_REPLICA;
+
+ // quota in bytes
+ private static final long ROOT_QUOTA = 2 * (2 * OzoneConsts.MB);
+ private static final long VOL_QUOTA = 2 * OzoneConsts.MB;
+ private static final long VOL_TWO_QUOTA = 2 * OzoneConsts.MB;
+ private static final long BUCKET_ONE_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_TWO_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_THREE_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_FOUR_QUOTA = OzoneConsts.MB;
+
+ // mock client's path requests
+ private static final String TEST_USER = "TestUser";
+ private static final String ROOT_PATH = "/";
+ private static final String VOL_PATH = "/vol";
+ private static final String VOL_TWO_PATH = "/vol2";
+ private static final String BUCKET_ONE_PATH = "/vol/bucket1";
+ private static final String BUCKET_TWO_PATH = "/vol/bucket2";
+ private static final String KEY_PATH = "/vol/bucket2/file4";
+ private static final String MULTI_BLOCK_KEY_PATH = "/vol/bucket1/file7";
+ private static final String INVALID_PATH = "/vol/path/not/found";
+
+ // some expected answers
+ private static final long ROOT_DATA_SIZE =
+ KEY_ONE_SIZE + KEY_TWO_SIZE + KEY_THREE_SIZE + KEY_FOUR_SIZE +
+ KEY_FIVE_SIZE + KEY_EIGHT_SIZE + KEY_NINE_SIZE + KEY_TEN_SIZE +
+ KEY_ELEVEN_SIZE;
+ private static final long VOL_DATA_SIZE = KEY_ONE_SIZE + KEY_TWO_SIZE +
+ KEY_THREE_SIZE + KEY_FOUR_SIZE + KEY_FIVE_SIZE;
+
+ private static final long VOL_TWO_DATA_SIZE =
+ KEY_EIGHT_SIZE + KEY_NINE_SIZE + KEY_TEN_SIZE + KEY_ELEVEN_SIZE;
+
+ private static final long BUCKET_ONE_DATA_SIZE = KEY_ONE_SIZE + KEY_TWO_SIZE +
+ KEY_THREE_SIZE;
+
+ private static final long BUCKET_TWO_DATA_SIZE =
+ KEY_FOUR_SIZE + KEY_FIVE_SIZE;
+
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ conf = new OzoneConfiguration();
+ OMMetadataManager omMetadataManager = initializeNewOmMetadataManager(
+ Files.createDirectory(temporaryFolder.resolve(
+ "JunitOmDBDir")).toFile(), conf);
+ OzoneManagerServiceProviderImpl ozoneManagerServiceProvider =
+ getMockOzoneManagerServiceProvider();
+ reconOMMetadataManager = getTestReconOmMetadataManager(omMetadataManager,
+ Files.createDirectory(temporaryFolder.resolve(
+ "omMetadatDir")).toFile());
+
+ ReconTestInjector reconTestInjector =
+ new ReconTestInjector.Builder(temporaryFolder.toFile())
+ .withReconOm(reconOMMetadataManager)
+ .withOmServiceProvider(ozoneManagerServiceProvider)
+ .withReconSqlDb()
+ .withContainerDB()
+ .addBinding(OzoneStorageContainerManager.class,
+ getMockReconSCM())
+ .addBinding(StorageContainerServiceProvider.class,
+ mock(StorageContainerServiceProviderImpl.class))
+ .addBinding(NSSummaryEndpoint.class)
+ .build();
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager =
+ reconTestInjector.getInstance(ReconNamespaceSummaryManager.class);
+ nsSummaryEndpoint = reconTestInjector.getInstance(NSSummaryEndpoint.class);
+
+ // populate OM DB and reprocess into Recon RocksDB
+ populateOMDB();
+ NSSummaryTaskWithOBS nsSummaryTaskWithOBS =
+ new NSSummaryTaskWithOBS(reconNamespaceSummaryManager,
+ reconOMMetadataManager, conf);
+ nsSummaryTaskWithOBS.reprocessWithOBS(reconOMMetadataManager);
+ commonUtils = new CommonUtils();
+ }
+
+ @Test
+ public void testUtility() {
+ String[] names = EntityHandler.parseRequestPath(TEST_PATH_UTILITY);
+ assertArrayEquals(TEST_NAMES, names);
+ String keyName = BucketHandler.getKeyName(names);
+ assertEquals(TEST_KEY_NAMES, keyName);
+ String subpath = BucketHandler.buildSubpath(PARENT_DIR, "file1.txt");
+ assertEquals(TEST_PATH_UTILITY, subpath);
+ }
+
+ @Test
+ public void testGetBasicInfoRoot() throws Exception {
+ // Test root basics
+ Response rootResponse = nsSummaryEndpoint.getBasicInfo(ROOT_PATH);
+ NamespaceSummaryResponse rootResponseObj =
+ (NamespaceSummaryResponse) rootResponse.getEntity();
+ assertEquals(EntityType.ROOT, rootResponseObj.getEntityType());
+ assertEquals(2, rootResponseObj.getCountStats().getNumVolume());
+ assertEquals(4, rootResponseObj.getCountStats().getNumBucket());
+ assertEquals(9, rootResponseObj.getCountStats().getNumTotalKey());
+ }
+
+ @Test
+ public void testGetBasicInfoVol() throws Exception {
+ // Test volume basics
+ Response volResponse = nsSummaryEndpoint.getBasicInfo(VOL_PATH);
+ NamespaceSummaryResponse volResponseObj =
+ (NamespaceSummaryResponse) volResponse.getEntity();
+ assertEquals(EntityType.VOLUME,
+ volResponseObj.getEntityType());
+ assertEquals(2, volResponseObj.getCountStats().getNumBucket());
+ assertEquals(5, volResponseObj.getCountStats().getNumTotalKey());
+ assertEquals("TestUser", ((VolumeObjectDBInfo) volResponseObj.
+ getObjectDBInfo()).getAdmin());
+ assertEquals("TestUser", ((VolumeObjectDBInfo) volResponseObj.
+ getObjectDBInfo()).getOwner());
+ assertEquals("vol", volResponseObj.getObjectDBInfo().getName());
+ assertEquals(2097152, volResponseObj.getObjectDBInfo().getQuotaInBytes());
+ assertEquals(-1, volResponseObj.getObjectDBInfo().getQuotaInNamespace());
Review Comment:
I think we should assert the constants here, like:
```
assertEquals(TEST_USER, ((VolumeObjectDBInfo) volResponseObj.getObjectDBInfo()).getAdmin());
```
This happened later as well, I won't comment everywhere, please check the whole class.
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestNSSummaryEndpointWithOBS.java:
##########
@@ -0,0 +1,1133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor.ONE;
+import static org.apache.hadoop.ozone.om.helpers.QuotaUtil.getReplicatedSize;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.StorageType;
+import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.container.ContainerReplica;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.handlers.BucketHandler;
+import org.apache.hadoop.ozone.recon.api.handlers.EntityHandler;
+import org.apache.hadoop.ozone.recon.api.types.BucketObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NamespaceSummaryResponse;
+import org.apache.hadoop.ozone.recon.api.types.QuotaUsageResponse;
+import org.apache.hadoop.ozone.recon.api.types.ResponseStatus;
+import org.apache.hadoop.ozone.recon.api.types.VolumeObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.FileSizeDistributionResponse;
+import org.apache.hadoop.ozone.recon.common.CommonUtils;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.scm.ReconNodeManager;
+import org.apache.hadoop.ozone.recon.scm.ReconStorageContainerManagerFacade;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.StorageContainerServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.spi.impl.StorageContainerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.tasks.NSSummaryTaskWithOBS;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import javax.ws.rs.core.Response;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.HashSet;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.apache.hadoop.hdds.protocol.MockDatanodeDetails.randomDatanodeDetails;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProvider;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for NSSummary REST APIs with OBS.
+ * Testing is done on a simple object store model with a flat hierarchy:
+ * vol
+ * / \
+ * bucket1 bucket2
+ * | |
+ * file1 file2
+ *
+ * This tests the Rest APIs for NSSummary in the context of OBS buckets,
+ * focusing on disk usage, quota usage, and file size distribution.
+ */
+public class TestNSSummaryEndpointWithOBS {
+ @TempDir
+ private Path temporaryFolder;
+
+ private ReconOMMetadataManager reconOMMetadataManager;
+ private NSSummaryEndpoint nsSummaryEndpoint;
+ private OzoneConfiguration conf;
+ private CommonUtils commonUtils;
+
+ private static final String TEST_PATH_UTILITY =
+ "/vol1/buck1/a/b/c/d/e/file1.txt";
+ private static final String PARENT_DIR = "vol1/buck1/a/b/c/d/e";
+ private static final String[] TEST_NAMES =
+ new String[]{"vol1", "buck1", "a", "b", "c", "d", "e", "file1.txt"};
+ private static final String TEST_KEY_NAMES = "a/b/c/d/e/file1.txt";
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String VOL_TWO = "vol2";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String BUCKET_THREE = "bucket3";
+ private static final String BUCKET_FOUR = "bucket4";
+ private static final String KEY_ONE = "file1";
+ private static final String KEY_TWO = "file2";
+ private static final String KEY_THREE = "file3";
+ private static final String KEY_FOUR = "file4";
+ private static final String KEY_FIVE = "file5";
+ private static final String KEY_SIX = "file6";
+ private static final String KEY_SEVEN = "file7";
+ private static final String KEY_EIGHT = "file8";
+ private static final String KEY_NINE = "file9";
+ private static final String KEY_TEN = "file10";
+ private static final String KEY_ELEVEN = "file11";
+ private static final String MULTI_BLOCK_KEY = "file7";
+ private static final String MULTI_BLOCK_FILE = "file7";
+
+ private static final String FILE_ONE = "file1";
+ private static final String FILE_TWO = "file2";
+ private static final String FILE_THREE = "file3";
+ private static final String FILE_FOUR = "file4";
+ private static final String FILE_FIVE = "file5";
+ private static final String FILE_SIX = "file6";
+ private static final String FILE_SEVEN = "file7";
+ private static final String FILE_EIGHT = "file8";
+ private static final String FILE_NINE = "file9";
+ private static final String FILE_TEN = "file10";
+ private static final String FILE_ELEVEN = "file11";
+ // objects IDs
+ private static final long PARENT_OBJECT_ID_ZERO = 0L;
+ private static final long VOL_OBJECT_ID = 0L;
+ private static final long BUCKET_ONE_OBJECT_ID = 1L;
+ private static final long BUCKET_TWO_OBJECT_ID = 2L;
+ private static final long KEY_ONE_OBJECT_ID = 3L;
+ private static final long KEY_TWO_OBJECT_ID = 5L;
+ private static final long KEY_FOUR_OBJECT_ID = 6L;
+ private static final long KEY_THREE_OBJECT_ID = 8L;
+ private static final long KEY_FIVE_OBJECT_ID = 9L;
+ private static final long KEY_SIX_OBJECT_ID = 10L;
+ private static final long MULTI_BLOCK_KEY_OBJECT_ID = 13L;
+ private static final long KEY_SEVEN_OBJECT_ID = 13L;
+ private static final long VOL_TWO_OBJECT_ID = 14L;
+ private static final long BUCKET_THREE_OBJECT_ID = 15L;
+ private static final long BUCKET_FOUR_OBJECT_ID = 16L;
+ private static final long KEY_EIGHT_OBJECT_ID = 17L;
+ private static final long KEY_NINE_OBJECT_ID = 19L;
+ private static final long KEY_TEN_OBJECT_ID = 20L;
+ private static final long KEY_ELEVEN_OBJECT_ID = 21L;
+
+ // container IDs
+ private static final long CONTAINER_ONE_ID = 1L;
+ private static final long CONTAINER_TWO_ID = 2L;
+ private static final long CONTAINER_THREE_ID = 3L;
+ private static final long CONTAINER_FOUR_ID = 4L;
+ private static final long CONTAINER_FIVE_ID = 5L;
+ private static final long CONTAINER_SIX_ID = 6L;
+
+ // replication factors
+ private static final int CONTAINER_ONE_REPLICA_COUNT = 3;
+ private static final int CONTAINER_TWO_REPLICA_COUNT = 2;
+ private static final int CONTAINER_THREE_REPLICA_COUNT = 4;
+ private static final int CONTAINER_FOUR_REPLICA_COUNT = 5;
+ private static final int CONTAINER_FIVE_REPLICA_COUNT = 2;
+ private static final int CONTAINER_SIX_REPLICA_COUNT = 3;
+
+ // block lengths
+ private static final long BLOCK_ONE_LENGTH = 1000L;
+ private static final long BLOCK_TWO_LENGTH = 2000L;
+ private static final long BLOCK_THREE_LENGTH = 3000L;
+ private static final long BLOCK_FOUR_LENGTH = 4000L;
+ private static final long BLOCK_FIVE_LENGTH = 5000L;
+ private static final long BLOCK_SIX_LENGTH = 6000L;
+
+ // data size in bytes
+ private static final long KEY_ONE_SIZE = 500L; // bin 0
+ private static final long KEY_TWO_SIZE = OzoneConsts.KB + 1; // bin 1
+ private static final long KEY_THREE_SIZE = 4 * OzoneConsts.KB + 1; // bin 3
+ private static final long KEY_FOUR_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_FIVE_SIZE = 100L; // bin 0
+ private static final long KEY_SIX_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_SEVEN_SIZE = 4 * OzoneConsts.KB + 1;
+ private static final long KEY_EIGHT_SIZE = OzoneConsts.KB + 1; // bin 1
+ private static final long KEY_NINE_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_TEN_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_ELEVEN_SIZE = OzoneConsts.KB + 1; // bin 1
+
+ private static final long FILE1_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_ONE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE2_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_TWO_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE3_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_THREE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE4_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_FOUR_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE5_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_FIVE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE6_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_SIX_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));;
+ private static final long FILE7_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_SEVEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));;
+ private static final long FILE8_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_EIGHT_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE9_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_NINE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE10_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_TEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE11_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_ELEVEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+
+ private static final long MULTI_BLOCK_KEY_SIZE_WITH_REPLICA
+ = FILE7_SIZE_WITH_REPLICA;
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_ROOT
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA
+ + FILE4_SIZE_WITH_REPLICA
+ + FILE5_SIZE_WITH_REPLICA
+ + FILE8_SIZE_WITH_REPLICA
+ + FILE9_SIZE_WITH_REPLICA
+ + FILE10_SIZE_WITH_REPLICA
+ + FILE11_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_VOL
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA
+ + FILE4_SIZE_WITH_REPLICA
+ + FILE5_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_BUCKET1
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_KEY
+ = FILE4_SIZE_WITH_REPLICA;
+
+ // quota in bytes
+ private static final long ROOT_QUOTA = 2 * (2 * OzoneConsts.MB);
+ private static final long VOL_QUOTA = 2 * OzoneConsts.MB;
+ private static final long VOL_TWO_QUOTA = 2 * OzoneConsts.MB;
+ private static final long BUCKET_ONE_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_TWO_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_THREE_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_FOUR_QUOTA = OzoneConsts.MB;
+
+ // mock client's path requests
+ private static final String TEST_USER = "TestUser";
+ private static final String ROOT_PATH = "/";
+ private static final String VOL_PATH = "/vol";
+ private static final String VOL_TWO_PATH = "/vol2";
+ private static final String BUCKET_ONE_PATH = "/vol/bucket1";
+ private static final String BUCKET_TWO_PATH = "/vol/bucket2";
+ private static final String KEY_PATH = "/vol/bucket2/file4";
+ private static final String MULTI_BLOCK_KEY_PATH = "/vol/bucket1/file7";
+ private static final String INVALID_PATH = "/vol/path/not/found";
+
+ // some expected answers
+ private static final long ROOT_DATA_SIZE =
+ KEY_ONE_SIZE + KEY_TWO_SIZE + KEY_THREE_SIZE + KEY_FOUR_SIZE +
+ KEY_FIVE_SIZE + KEY_EIGHT_SIZE + KEY_NINE_SIZE + KEY_TEN_SIZE +
+ KEY_ELEVEN_SIZE;
+ private static final long VOL_DATA_SIZE = KEY_ONE_SIZE + KEY_TWO_SIZE +
+ KEY_THREE_SIZE + KEY_FOUR_SIZE + KEY_FIVE_SIZE;
+
+ private static final long VOL_TWO_DATA_SIZE =
+ KEY_EIGHT_SIZE + KEY_NINE_SIZE + KEY_TEN_SIZE + KEY_ELEVEN_SIZE;
+
+ private static final long BUCKET_ONE_DATA_SIZE = KEY_ONE_SIZE + KEY_TWO_SIZE +
+ KEY_THREE_SIZE;
+
+ private static final long BUCKET_TWO_DATA_SIZE =
+ KEY_FOUR_SIZE + KEY_FIVE_SIZE;
+
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ conf = new OzoneConfiguration();
+ OMMetadataManager omMetadataManager = initializeNewOmMetadataManager(
+ Files.createDirectory(temporaryFolder.resolve(
+ "JunitOmDBDir")).toFile(), conf);
+ OzoneManagerServiceProviderImpl ozoneManagerServiceProvider =
+ getMockOzoneManagerServiceProvider();
+ reconOMMetadataManager = getTestReconOmMetadataManager(omMetadataManager,
+ Files.createDirectory(temporaryFolder.resolve(
+ "omMetadatDir")).toFile());
+
+ ReconTestInjector reconTestInjector =
+ new ReconTestInjector.Builder(temporaryFolder.toFile())
+ .withReconOm(reconOMMetadataManager)
+ .withOmServiceProvider(ozoneManagerServiceProvider)
+ .withReconSqlDb()
+ .withContainerDB()
+ .addBinding(OzoneStorageContainerManager.class,
+ getMockReconSCM())
+ .addBinding(StorageContainerServiceProvider.class,
+ mock(StorageContainerServiceProviderImpl.class))
+ .addBinding(NSSummaryEndpoint.class)
+ .build();
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager =
+ reconTestInjector.getInstance(ReconNamespaceSummaryManager.class);
+ nsSummaryEndpoint = reconTestInjector.getInstance(NSSummaryEndpoint.class);
+
+ // populate OM DB and reprocess into Recon RocksDB
+ populateOMDB();
+ NSSummaryTaskWithOBS nsSummaryTaskWithOBS =
+ new NSSummaryTaskWithOBS(reconNamespaceSummaryManager,
+ reconOMMetadataManager, conf);
+ nsSummaryTaskWithOBS.reprocessWithOBS(reconOMMetadataManager);
+ commonUtils = new CommonUtils();
+ }
+
+ @Test
+ public void testUtility() {
+ String[] names = EntityHandler.parseRequestPath(TEST_PATH_UTILITY);
+ assertArrayEquals(TEST_NAMES, names);
+ String keyName = BucketHandler.getKeyName(names);
+ assertEquals(TEST_KEY_NAMES, keyName);
+ String subpath = BucketHandler.buildSubpath(PARENT_DIR, "file1.txt");
+ assertEquals(TEST_PATH_UTILITY, subpath);
+ }
+
+ @Test
+ public void testGetBasicInfoRoot() throws Exception {
+ // Test root basics
+ Response rootResponse = nsSummaryEndpoint.getBasicInfo(ROOT_PATH);
+ NamespaceSummaryResponse rootResponseObj =
+ (NamespaceSummaryResponse) rootResponse.getEntity();
+ assertEquals(EntityType.ROOT, rootResponseObj.getEntityType());
+ assertEquals(2, rootResponseObj.getCountStats().getNumVolume());
+ assertEquals(4, rootResponseObj.getCountStats().getNumBucket());
+ assertEquals(9, rootResponseObj.getCountStats().getNumTotalKey());
+ }
+
+ @Test
+ public void testGetBasicInfoVol() throws Exception {
+ // Test volume basics
+ Response volResponse = nsSummaryEndpoint.getBasicInfo(VOL_PATH);
+ NamespaceSummaryResponse volResponseObj =
+ (NamespaceSummaryResponse) volResponse.getEntity();
+ assertEquals(EntityType.VOLUME,
+ volResponseObj.getEntityType());
+ assertEquals(2, volResponseObj.getCountStats().getNumBucket());
+ assertEquals(5, volResponseObj.getCountStats().getNumTotalKey());
+ assertEquals("TestUser", ((VolumeObjectDBInfo) volResponseObj.
+ getObjectDBInfo()).getAdmin());
+ assertEquals("TestUser", ((VolumeObjectDBInfo) volResponseObj.
+ getObjectDBInfo()).getOwner());
+ assertEquals("vol", volResponseObj.getObjectDBInfo().getName());
+ assertEquals(2097152, volResponseObj.getObjectDBInfo().getQuotaInBytes());
+ assertEquals(-1, volResponseObj.getObjectDBInfo().getQuotaInNamespace());
+ }
+
+ @Test
+ public void testGetBasicInfoBucketOne() throws Exception {
+ // Test bucket 1's basics
+ Response bucketOneResponse =
+ nsSummaryEndpoint.getBasicInfo(BUCKET_ONE_PATH);
+ NamespaceSummaryResponse bucketOneObj =
+ (NamespaceSummaryResponse) bucketOneResponse.getEntity();
+ assertEquals(EntityType.BUCKET, bucketOneObj.getEntityType());
+ assertEquals(3, bucketOneObj.getCountStats().getNumTotalKey());
+ assertEquals("vol",
+ ((BucketObjectDBInfo) bucketOneObj.getObjectDBInfo()).getVolumeName());
+ assertEquals(StorageType.DISK,
+ ((BucketObjectDBInfo)
+ bucketOneObj.getObjectDBInfo()).getStorageType());
+ assertEquals(getBucketLayout(),
+ ((BucketObjectDBInfo)
+ bucketOneObj.getObjectDBInfo()).getBucketLayout());
+ assertEquals("bucket1",
+ ((BucketObjectDBInfo) bucketOneObj.getObjectDBInfo()).getName());
+ }
+
+ @Test
+ public void testGetBasicInfoBucketTwo() throws Exception {
+ // Test bucket 2's basics
+ commonUtils.testNSSummaryBasicInfoBucketTwo(
+ BucketLayout.OBJECT_STORE,
+ nsSummaryEndpoint);
+ }
+
+ @Test
+ public void testGetBasicInfoNoPath() throws Exception {
+ // Test invalid path
+ commonUtils.testNSSummaryBasicInfoNoPath(nsSummaryEndpoint);
+ }
+
+ @Test
+ public void testGetBasicInfoKey() throws Exception {
+ // Test key
+ commonUtils.testNSSummaryBasicInfoKey(nsSummaryEndpoint);
+ }
+
+ @Test
+ public void testDiskUsageRoot() throws Exception {
+ // root level DU
+ Response rootResponse = nsSummaryEndpoint.getDiskUsage(ROOT_PATH,
+ false, false);
+ DUResponse duRootRes = (DUResponse) rootResponse.getEntity();
+ assertEquals(2, duRootRes.getCount());
+ List<DUResponse.DiskUsage> duRootData = duRootRes.getDuData();
+ // sort based on subpath
+ Collections.sort(duRootData,
+ Comparator.comparing(DUResponse.DiskUsage::getSubpath));
+ DUResponse.DiskUsage duVol1 = duRootData.get(0);
+ DUResponse.DiskUsage duVol2 = duRootData.get(1);
+ assertEquals(VOL_PATH, duVol1.getSubpath());
+ assertEquals(VOL_TWO_PATH, duVol2.getSubpath());
+ assertEquals(VOL_DATA_SIZE, duVol1.getSize());
+ assertEquals(VOL_TWO_DATA_SIZE, duVol2.getSize());
+ }
+
+ @Test
+ public void testDiskUsageVolume() throws Exception {
+ // volume level DU
+ Response volResponse = nsSummaryEndpoint.getDiskUsage(VOL_PATH,
+ false, false);
+ DUResponse duVolRes = (DUResponse) volResponse.getEntity();
+ assertEquals(2, duVolRes.getCount());
+ List<DUResponse.DiskUsage> duData = duVolRes.getDuData();
+ // sort based on subpath
+ Collections.sort(duData,
+ Comparator.comparing(DUResponse.DiskUsage::getSubpath));
+ DUResponse.DiskUsage duBucket1 = duData.get(0);
+ DUResponse.DiskUsage duBucket2 = duData.get(1);
+ assertEquals(BUCKET_ONE_PATH, duBucket1.getSubpath());
+ assertEquals(BUCKET_TWO_PATH, duBucket2.getSubpath());
+ assertEquals(BUCKET_ONE_DATA_SIZE, duBucket1.getSize());
+ assertEquals(BUCKET_TWO_DATA_SIZE, duBucket2.getSize());
+ }
+
+ @Test
+ public void testDiskUsageBucket() throws Exception {
+ // bucket level DU
+ Response bucketResponse = nsSummaryEndpoint.getDiskUsage(BUCKET_ONE_PATH,
+ false, false);
+ DUResponse duBucketResponse = (DUResponse) bucketResponse.getEntity();
+ // There are no sub-paths under this OBS bucket.
+ assertEquals(0, duBucketResponse.getCount());
+ assertEquals(BUCKET_ONE_DATA_SIZE, duBucketResponse.getSize());
+ }
+
+ @Test
+ public void testDiskUsageKey() throws Exception {
+ // key level DU
+ Response keyResponse = nsSummaryEndpoint.getDiskUsage(KEY_PATH,
+ false, false);
+ DUResponse keyObj = (DUResponse) keyResponse.getEntity();
+ assertEquals(0, keyObj.getCount());
+ assertEquals(KEY_FOUR_SIZE, keyObj.getSize());
+ }
+
+ @Test
+ public void testDiskUsageUnknown() throws Exception {
+ // invalid path check
+ Response invalidResponse = nsSummaryEndpoint.getDiskUsage(INVALID_PATH,
+ false, false);
+ DUResponse invalidObj = (DUResponse) invalidResponse.getEntity();
+ assertEquals(ResponseStatus.PATH_NOT_FOUND,
+ invalidObj.getStatus());
+ }
Review Comment:
Isn't this does the same as `testGetBasicInfoNoPath()`?
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestNSSummaryEndpointWithOBS.java:
##########
@@ -0,0 +1,1133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor.ONE;
+import static org.apache.hadoop.ozone.om.helpers.QuotaUtil.getReplicatedSize;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.StorageType;
+import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.container.ContainerReplica;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.handlers.BucketHandler;
+import org.apache.hadoop.ozone.recon.api.handlers.EntityHandler;
+import org.apache.hadoop.ozone.recon.api.types.BucketObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NamespaceSummaryResponse;
+import org.apache.hadoop.ozone.recon.api.types.QuotaUsageResponse;
+import org.apache.hadoop.ozone.recon.api.types.ResponseStatus;
+import org.apache.hadoop.ozone.recon.api.types.VolumeObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.FileSizeDistributionResponse;
+import org.apache.hadoop.ozone.recon.common.CommonUtils;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.scm.ReconNodeManager;
+import org.apache.hadoop.ozone.recon.scm.ReconStorageContainerManagerFacade;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.StorageContainerServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.spi.impl.StorageContainerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.tasks.NSSummaryTaskWithOBS;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import javax.ws.rs.core.Response;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.HashSet;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.apache.hadoop.hdds.protocol.MockDatanodeDetails.randomDatanodeDetails;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProvider;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for NSSummary REST APIs with OBS.
+ * Testing is done on a simple object store model with a flat hierarchy:
+ * vol
+ * / \
+ * bucket1 bucket2
+ * | |
+ * file1 file2
+ *
+ * This tests the Rest APIs for NSSummary in the context of OBS buckets,
+ * focusing on disk usage, quota usage, and file size distribution.
+ */
+public class TestNSSummaryEndpointWithOBS {
+ @TempDir
+ private Path temporaryFolder;
+
+ private ReconOMMetadataManager reconOMMetadataManager;
+ private NSSummaryEndpoint nsSummaryEndpoint;
+ private OzoneConfiguration conf;
+ private CommonUtils commonUtils;
+
+ private static final String TEST_PATH_UTILITY =
+ "/vol1/buck1/a/b/c/d/e/file1.txt";
+ private static final String PARENT_DIR = "vol1/buck1/a/b/c/d/e";
+ private static final String[] TEST_NAMES =
+ new String[]{"vol1", "buck1", "a", "b", "c", "d", "e", "file1.txt"};
+ private static final String TEST_KEY_NAMES = "a/b/c/d/e/file1.txt";
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String VOL_TWO = "vol2";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String BUCKET_THREE = "bucket3";
+ private static final String BUCKET_FOUR = "bucket4";
+ private static final String KEY_ONE = "file1";
+ private static final String KEY_TWO = "file2";
+ private static final String KEY_THREE = "file3";
+ private static final String KEY_FOUR = "file4";
+ private static final String KEY_FIVE = "file5";
+ private static final String KEY_SIX = "file6";
+ private static final String KEY_SEVEN = "file7";
+ private static final String KEY_EIGHT = "file8";
+ private static final String KEY_NINE = "file9";
+ private static final String KEY_TEN = "file10";
+ private static final String KEY_ELEVEN = "file11";
+ private static final String MULTI_BLOCK_KEY = "file7";
+ private static final String MULTI_BLOCK_FILE = "file7";
+
+ private static final String FILE_ONE = "file1";
+ private static final String FILE_TWO = "file2";
+ private static final String FILE_THREE = "file3";
+ private static final String FILE_FOUR = "file4";
+ private static final String FILE_FIVE = "file5";
+ private static final String FILE_SIX = "file6";
+ private static final String FILE_SEVEN = "file7";
+ private static final String FILE_EIGHT = "file8";
+ private static final String FILE_NINE = "file9";
+ private static final String FILE_TEN = "file10";
+ private static final String FILE_ELEVEN = "file11";
+ // objects IDs
+ private static final long PARENT_OBJECT_ID_ZERO = 0L;
+ private static final long VOL_OBJECT_ID = 0L;
+ private static final long BUCKET_ONE_OBJECT_ID = 1L;
+ private static final long BUCKET_TWO_OBJECT_ID = 2L;
+ private static final long KEY_ONE_OBJECT_ID = 3L;
+ private static final long KEY_TWO_OBJECT_ID = 5L;
+ private static final long KEY_FOUR_OBJECT_ID = 6L;
+ private static final long KEY_THREE_OBJECT_ID = 8L;
+ private static final long KEY_FIVE_OBJECT_ID = 9L;
+ private static final long KEY_SIX_OBJECT_ID = 10L;
+ private static final long MULTI_BLOCK_KEY_OBJECT_ID = 13L;
+ private static final long KEY_SEVEN_OBJECT_ID = 13L;
+ private static final long VOL_TWO_OBJECT_ID = 14L;
+ private static final long BUCKET_THREE_OBJECT_ID = 15L;
+ private static final long BUCKET_FOUR_OBJECT_ID = 16L;
+ private static final long KEY_EIGHT_OBJECT_ID = 17L;
+ private static final long KEY_NINE_OBJECT_ID = 19L;
+ private static final long KEY_TEN_OBJECT_ID = 20L;
+ private static final long KEY_ELEVEN_OBJECT_ID = 21L;
+
+ // container IDs
+ private static final long CONTAINER_ONE_ID = 1L;
+ private static final long CONTAINER_TWO_ID = 2L;
+ private static final long CONTAINER_THREE_ID = 3L;
+ private static final long CONTAINER_FOUR_ID = 4L;
+ private static final long CONTAINER_FIVE_ID = 5L;
+ private static final long CONTAINER_SIX_ID = 6L;
+
+ // replication factors
+ private static final int CONTAINER_ONE_REPLICA_COUNT = 3;
+ private static final int CONTAINER_TWO_REPLICA_COUNT = 2;
+ private static final int CONTAINER_THREE_REPLICA_COUNT = 4;
+ private static final int CONTAINER_FOUR_REPLICA_COUNT = 5;
+ private static final int CONTAINER_FIVE_REPLICA_COUNT = 2;
+ private static final int CONTAINER_SIX_REPLICA_COUNT = 3;
+
+ // block lengths
+ private static final long BLOCK_ONE_LENGTH = 1000L;
+ private static final long BLOCK_TWO_LENGTH = 2000L;
+ private static final long BLOCK_THREE_LENGTH = 3000L;
+ private static final long BLOCK_FOUR_LENGTH = 4000L;
+ private static final long BLOCK_FIVE_LENGTH = 5000L;
+ private static final long BLOCK_SIX_LENGTH = 6000L;
+
+ // data size in bytes
+ private static final long KEY_ONE_SIZE = 500L; // bin 0
+ private static final long KEY_TWO_SIZE = OzoneConsts.KB + 1; // bin 1
+ private static final long KEY_THREE_SIZE = 4 * OzoneConsts.KB + 1; // bin 3
+ private static final long KEY_FOUR_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_FIVE_SIZE = 100L; // bin 0
+ private static final long KEY_SIX_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_SEVEN_SIZE = 4 * OzoneConsts.KB + 1;
+ private static final long KEY_EIGHT_SIZE = OzoneConsts.KB + 1; // bin 1
+ private static final long KEY_NINE_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_TEN_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_ELEVEN_SIZE = OzoneConsts.KB + 1; // bin 1
+
+ private static final long FILE1_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_ONE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE2_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_TWO_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE3_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_THREE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE4_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_FOUR_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE5_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_FIVE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE6_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_SIX_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));;
+ private static final long FILE7_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_SEVEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));;
+ private static final long FILE8_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_EIGHT_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE9_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_NINE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE10_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_TEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE11_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_ELEVEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+
+ private static final long MULTI_BLOCK_KEY_SIZE_WITH_REPLICA
+ = FILE7_SIZE_WITH_REPLICA;
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_ROOT
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA
+ + FILE4_SIZE_WITH_REPLICA
+ + FILE5_SIZE_WITH_REPLICA
+ + FILE8_SIZE_WITH_REPLICA
+ + FILE9_SIZE_WITH_REPLICA
+ + FILE10_SIZE_WITH_REPLICA
+ + FILE11_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_VOL
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA
+ + FILE4_SIZE_WITH_REPLICA
+ + FILE5_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_BUCKET1
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_KEY
+ = FILE4_SIZE_WITH_REPLICA;
+
+ // quota in bytes
+ private static final long ROOT_QUOTA = 2 * (2 * OzoneConsts.MB);
+ private static final long VOL_QUOTA = 2 * OzoneConsts.MB;
+ private static final long VOL_TWO_QUOTA = 2 * OzoneConsts.MB;
+ private static final long BUCKET_ONE_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_TWO_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_THREE_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_FOUR_QUOTA = OzoneConsts.MB;
+
+ // mock client's path requests
+ private static final String TEST_USER = "TestUser";
+ private static final String ROOT_PATH = "/";
+ private static final String VOL_PATH = "/vol";
+ private static final String VOL_TWO_PATH = "/vol2";
+ private static final String BUCKET_ONE_PATH = "/vol/bucket1";
+ private static final String BUCKET_TWO_PATH = "/vol/bucket2";
+ private static final String KEY_PATH = "/vol/bucket2/file4";
+ private static final String MULTI_BLOCK_KEY_PATH = "/vol/bucket1/file7";
+ private static final String INVALID_PATH = "/vol/path/not/found";
Review Comment:
Shouldn't these use the constants added before? So if we change the volume/bucket names it will be reflected here as well.
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestNSSummaryEndpointWithOBS.java:
##########
@@ -0,0 +1,1133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor.ONE;
+import static org.apache.hadoop.ozone.om.helpers.QuotaUtil.getReplicatedSize;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.StorageType;
+import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.container.ContainerReplica;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.handlers.BucketHandler;
+import org.apache.hadoop.ozone.recon.api.handlers.EntityHandler;
+import org.apache.hadoop.ozone.recon.api.types.BucketObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NamespaceSummaryResponse;
+import org.apache.hadoop.ozone.recon.api.types.QuotaUsageResponse;
+import org.apache.hadoop.ozone.recon.api.types.ResponseStatus;
+import org.apache.hadoop.ozone.recon.api.types.VolumeObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.FileSizeDistributionResponse;
+import org.apache.hadoop.ozone.recon.common.CommonUtils;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.scm.ReconNodeManager;
+import org.apache.hadoop.ozone.recon.scm.ReconStorageContainerManagerFacade;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.StorageContainerServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.spi.impl.StorageContainerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.tasks.NSSummaryTaskWithOBS;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import javax.ws.rs.core.Response;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.HashSet;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.apache.hadoop.hdds.protocol.MockDatanodeDetails.randomDatanodeDetails;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProvider;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for NSSummary REST APIs with OBS.
+ * Testing is done on a simple object store model with a flat hierarchy:
+ * vol
+ * / \
+ * bucket1 bucket2
+ * | |
+ * file1 file2
+ *
+ * This tests the Rest APIs for NSSummary in the context of OBS buckets,
+ * focusing on disk usage, quota usage, and file size distribution.
+ */
+public class TestNSSummaryEndpointWithOBS {
+ @TempDir
+ private Path temporaryFolder;
+
+ private ReconOMMetadataManager reconOMMetadataManager;
+ private NSSummaryEndpoint nsSummaryEndpoint;
+ private OzoneConfiguration conf;
+ private CommonUtils commonUtils;
+
+ private static final String TEST_PATH_UTILITY =
+ "/vol1/buck1/a/b/c/d/e/file1.txt";
+ private static final String PARENT_DIR = "vol1/buck1/a/b/c/d/e";
+ private static final String[] TEST_NAMES =
+ new String[]{"vol1", "buck1", "a", "b", "c", "d", "e", "file1.txt"};
+ private static final String TEST_KEY_NAMES = "a/b/c/d/e/file1.txt";
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String VOL_TWO = "vol2";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String BUCKET_THREE = "bucket3";
+ private static final String BUCKET_FOUR = "bucket4";
+ private static final String KEY_ONE = "file1";
+ private static final String KEY_TWO = "file2";
+ private static final String KEY_THREE = "file3";
+ private static final String KEY_FOUR = "file4";
+ private static final String KEY_FIVE = "file5";
+ private static final String KEY_SIX = "file6";
+ private static final String KEY_SEVEN = "file7";
+ private static final String KEY_EIGHT = "file8";
+ private static final String KEY_NINE = "file9";
+ private static final String KEY_TEN = "file10";
+ private static final String KEY_ELEVEN = "file11";
+ private static final String MULTI_BLOCK_KEY = "file7";
+ private static final String MULTI_BLOCK_FILE = "file7";
+
+ private static final String FILE_ONE = "file1";
+ private static final String FILE_TWO = "file2";
+ private static final String FILE_THREE = "file3";
+ private static final String FILE_FOUR = "file4";
+ private static final String FILE_FIVE = "file5";
+ private static final String FILE_SIX = "file6";
+ private static final String FILE_SEVEN = "file7";
+ private static final String FILE_EIGHT = "file8";
+ private static final String FILE_NINE = "file9";
+ private static final String FILE_TEN = "file10";
+ private static final String FILE_ELEVEN = "file11";
Review Comment:
I don't get why do we need these file constants beside the key constants, as the values are also the same. I couldn't find the reason in the code either.
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestNSSummaryEndpointWithOBS.java:
##########
@@ -0,0 +1,1133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor.ONE;
+import static org.apache.hadoop.ozone.om.helpers.QuotaUtil.getReplicatedSize;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.StorageType;
+import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.container.ContainerReplica;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.handlers.BucketHandler;
+import org.apache.hadoop.ozone.recon.api.handlers.EntityHandler;
+import org.apache.hadoop.ozone.recon.api.types.BucketObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NamespaceSummaryResponse;
+import org.apache.hadoop.ozone.recon.api.types.QuotaUsageResponse;
+import org.apache.hadoop.ozone.recon.api.types.ResponseStatus;
+import org.apache.hadoop.ozone.recon.api.types.VolumeObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.FileSizeDistributionResponse;
+import org.apache.hadoop.ozone.recon.common.CommonUtils;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.scm.ReconNodeManager;
+import org.apache.hadoop.ozone.recon.scm.ReconStorageContainerManagerFacade;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.StorageContainerServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.spi.impl.StorageContainerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.tasks.NSSummaryTaskWithOBS;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import javax.ws.rs.core.Response;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.HashSet;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.apache.hadoop.hdds.protocol.MockDatanodeDetails.randomDatanodeDetails;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProvider;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for NSSummary REST APIs with OBS.
+ * Testing is done on a simple object store model with a flat hierarchy:
+ * vol
+ * / \
+ * bucket1 bucket2
+ * | |
+ * file1 file2
+ *
+ * This tests the Rest APIs for NSSummary in the context of OBS buckets,
+ * focusing on disk usage, quota usage, and file size distribution.
+ */
+public class TestNSSummaryEndpointWithOBS {
+ @TempDir
+ private Path temporaryFolder;
+
+ private ReconOMMetadataManager reconOMMetadataManager;
+ private NSSummaryEndpoint nsSummaryEndpoint;
+ private OzoneConfiguration conf;
+ private CommonUtils commonUtils;
+
+ private static final String TEST_PATH_UTILITY =
+ "/vol1/buck1/a/b/c/d/e/file1.txt";
+ private static final String PARENT_DIR = "vol1/buck1/a/b/c/d/e";
+ private static final String[] TEST_NAMES =
+ new String[]{"vol1", "buck1", "a", "b", "c", "d", "e", "file1.txt"};
+ private static final String TEST_KEY_NAMES = "a/b/c/d/e/file1.txt";
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String VOL_TWO = "vol2";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String BUCKET_THREE = "bucket3";
+ private static final String BUCKET_FOUR = "bucket4";
+ private static final String KEY_ONE = "file1";
+ private static final String KEY_TWO = "file2";
+ private static final String KEY_THREE = "file3";
+ private static final String KEY_FOUR = "file4";
+ private static final String KEY_FIVE = "file5";
+ private static final String KEY_SIX = "file6";
+ private static final String KEY_SEVEN = "file7";
+ private static final String KEY_EIGHT = "file8";
+ private static final String KEY_NINE = "file9";
+ private static final String KEY_TEN = "file10";
+ private static final String KEY_ELEVEN = "file11";
+ private static final String MULTI_BLOCK_KEY = "file7";
+ private static final String MULTI_BLOCK_FILE = "file7";
Review Comment:
Why are these `file7` too?
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/TestNSSummaryTaskWithOBS.java:
##########
@@ -0,0 +1,548 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.utils.db.RDBBatchOperation;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Nested;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Set;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProviderWithFSO;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+/**
+ * Unit test for NSSummaryTaskWithOBS.
+ */
+public final class TestNSSummaryTaskWithOBS implements Serializable {
+ private static ReconNamespaceSummaryManager reconNamespaceSummaryManager;
+ private static OMMetadataManager omMetadataManager;
+ private static ReconOMMetadataManager reconOMMetadataManager;
+ private static NSSummaryTaskWithOBS nSSummaryTaskWithOBS;
+ private static OzoneConfiguration omConfiguration;
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String KEY_ONE = "key1";
+ private static final String KEY_TWO = "key2";
+ private static final String KEY_THREE = "dir1/dir2/key3";
+ private static final String KEY_FOUR = "key4///////////";
+ private static final String KEY_FIVE = "//////////";
+ private static final String KEY_SIX = "key6";
+ private static final String KEY_SEVEN = "key7";
+
+ private static final String TEST_USER = "TestUser";
+
+ private static final long PARENT_OBJECT_ID_ZERO = 0L;
+ private static final long VOL_OBJECT_ID = 0L;
+ private static final long BUCKET_ONE_OBJECT_ID = 1L;
+ private static final long BUCKET_TWO_OBJECT_ID = 2L;
+ private static final long KEY_ONE_OBJECT_ID = 3L;
+ private static final long KEY_TWO_OBJECT_ID = 5L;
+ private static final long KEY_FOUR_OBJECT_ID = 6L;
+ private static final long KEY_THREE_OBJECT_ID = 8L;
+ private static final long KEY_FIVE_OBJECT_ID = 9L;
+ private static final long KEY_SIX_OBJECT_ID = 10L;
+ private static final long KEY_SEVEN_OBJECT_ID = 11L;
+
+
+ private static final long KEY_ONE_SIZE = 500L;
+ private static final long KEY_TWO_OLD_SIZE = 1025L;
+ private static final long KEY_TWO_UPDATE_SIZE = 1023L;
+ private static final long KEY_THREE_SIZE =
+ ReconConstants.MAX_FILE_SIZE_UPPER_BOUND - 100L;
+ private static final long KEY_FOUR_SIZE = 2050L;
+ private static final long KEY_FIVE_SIZE = 100L;
+ private static final long KEY_SIX_SIZE = 6000L;
+ private static final long KEY_SEVEN_SIZE = 7000L;
+
+ private TestNSSummaryTaskWithOBS() {
+ }
+
+ @BeforeAll
+ public static void setUp(@TempDir File tmpDir) throws Exception {
+ initializeNewOmMetadataManager(new File(tmpDir, "om"));
+ OzoneManagerServiceProviderImpl ozoneManagerServiceProvider =
+ getMockOzoneManagerServiceProviderWithFSO();
+ reconOMMetadataManager = getTestReconOmMetadataManager(omMetadataManager,
+ new File(tmpDir, "recon"));
+
+ ReconTestInjector reconTestInjector =
+ new ReconTestInjector.Builder(tmpDir)
+ .withReconOm(reconOMMetadataManager)
+ .withOmServiceProvider(ozoneManagerServiceProvider)
+ .withReconSqlDb()
+ .withContainerDB()
+ .build();
+ reconNamespaceSummaryManager =
+ reconTestInjector.getInstance(ReconNamespaceSummaryManager.class);
+
+ NSSummary nonExistentSummary =
+ reconNamespaceSummaryManager.getNSSummary(BUCKET_ONE_OBJECT_ID);
+ assertNull(nonExistentSummary);
+
+ populateOMDB();
+
+ nSSummaryTaskWithOBS = new NSSummaryTaskWithOBS(
+ reconNamespaceSummaryManager,
+ reconOMMetadataManager, omConfiguration);
+ }
+
+ /**
+ * Nested class for testing NSSummaryTaskWithOBS reprocess.
+ */
+ @Nested
+ public class TestReprocess {
+
+ private NSSummary nsSummaryForBucket1;
+ private NSSummary nsSummaryForBucket2;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ // write a NSSummary prior to reprocess
+ // verify it got cleaned up after.
+ NSSummary staleNSSummary = new NSSummary();
+ RDBBatchOperation rdbBatchOperation = new RDBBatchOperation();
+ reconNamespaceSummaryManager.batchStoreNSSummaries(rdbBatchOperation, -1L,
+ staleNSSummary);
+ reconNamespaceSummaryManager.commitBatchOperation(rdbBatchOperation);
+
+ // Verify commit
+ assertNotNull(reconNamespaceSummaryManager.getNSSummary(-1L));
+
+ // reinit Recon RocksDB's namespace CF.
+ reconNamespaceSummaryManager.clearNSSummaryTable();
+
+ nSSummaryTaskWithOBS.reprocessWithOBS(reconOMMetadataManager);
+ assertNull(reconNamespaceSummaryManager.getNSSummary(-1L));
+
+ nsSummaryForBucket1 =
+ reconNamespaceSummaryManager.getNSSummary(BUCKET_ONE_OBJECT_ID);
+ nsSummaryForBucket2 =
+ reconNamespaceSummaryManager.getNSSummary(BUCKET_TWO_OBJECT_ID);
+ assertNotNull(nsSummaryForBucket1);
+ assertNotNull(nsSummaryForBucket2);
+ }
+
+ @Test
+ public void testReprocessNSSummaryNull() throws IOException {
+ assertNull(reconNamespaceSummaryManager.getNSSummary(-1L));
+ }
+
+ @Test
+ public void testReprocessGetFiles() {
+ assertEquals(3, nsSummaryForBucket1.getNumOfFiles());
+ assertEquals(2, nsSummaryForBucket2.getNumOfFiles());
+
+ assertEquals(KEY_ONE_SIZE + KEY_TWO_OLD_SIZE + KEY_THREE_SIZE,
+ nsSummaryForBucket1.getSizeOfFiles());
+ assertEquals(KEY_FOUR_SIZE + KEY_FIVE_SIZE,
+ nsSummaryForBucket2.getSizeOfFiles());
+ }
+
+ @Test
+ public void testReprocessFileBucketSize() {
+ int[] fileDistBucket1 = nsSummaryForBucket1.getFileSizeBucket();
+ int[] fileDistBucket2 = nsSummaryForBucket2.getFileSizeBucket();
+ assertEquals(ReconConstants.NUM_OF_FILE_SIZE_BINS,
+ fileDistBucket1.length);
+ assertEquals(ReconConstants.NUM_OF_FILE_SIZE_BINS,
+ fileDistBucket2.length);
+
+ // Check for 1's and 0's in fileDistBucket1
+ int[] expectedIndexes1 = {0, 1, 40};
+ for (int index = 0; index < fileDistBucket1.length; index++) {
+ if (contains(expectedIndexes1, index)) {
+ assertEquals(1, fileDistBucket1[index]);
+ } else {
+ assertEquals(0, fileDistBucket1[index]);
+ }
+ }
+
+ // Check for 1's and 0's in fileDistBucket2
+ int[] expectedIndexes2 = {0, 2};
+ for (int index = 0; index < fileDistBucket2.length; index++) {
+ if (contains(expectedIndexes2, index)) {
+ assertEquals(1, fileDistBucket2[index]);
+ } else {
+ assertEquals(0, fileDistBucket2[index]);
+ }
+ }
+ }
+
+ }
+
+ /**
+ * Nested class for testing NSSummaryTaskWithLegacy process.
Review Comment:
```suggestion
* Nested class for testing NSSummaryTaskWithOBS process.
```
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestNSSummaryEndpointWithOBS.java:
##########
@@ -0,0 +1,1133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor.ONE;
+import static org.apache.hadoop.ozone.om.helpers.QuotaUtil.getReplicatedSize;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.StorageType;
+import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.container.ContainerReplica;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.handlers.BucketHandler;
+import org.apache.hadoop.ozone.recon.api.handlers.EntityHandler;
+import org.apache.hadoop.ozone.recon.api.types.BucketObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NamespaceSummaryResponse;
+import org.apache.hadoop.ozone.recon.api.types.QuotaUsageResponse;
+import org.apache.hadoop.ozone.recon.api.types.ResponseStatus;
+import org.apache.hadoop.ozone.recon.api.types.VolumeObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.FileSizeDistributionResponse;
+import org.apache.hadoop.ozone.recon.common.CommonUtils;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.scm.ReconNodeManager;
+import org.apache.hadoop.ozone.recon.scm.ReconStorageContainerManagerFacade;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.StorageContainerServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.spi.impl.StorageContainerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.tasks.NSSummaryTaskWithOBS;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import javax.ws.rs.core.Response;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.HashSet;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.apache.hadoop.hdds.protocol.MockDatanodeDetails.randomDatanodeDetails;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProvider;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for NSSummary REST APIs with OBS.
+ * Testing is done on a simple object store model with a flat hierarchy:
+ * vol
+ * / \
+ * bucket1 bucket2
+ * | |
+ * file1 file2
+ *
+ * This tests the Rest APIs for NSSummary in the context of OBS buckets,
+ * focusing on disk usage, quota usage, and file size distribution.
+ */
+public class TestNSSummaryEndpointWithOBS {
+ @TempDir
+ private Path temporaryFolder;
+
+ private ReconOMMetadataManager reconOMMetadataManager;
+ private NSSummaryEndpoint nsSummaryEndpoint;
+ private OzoneConfiguration conf;
+ private CommonUtils commonUtils;
+
+ private static final String TEST_PATH_UTILITY =
+ "/vol1/buck1/a/b/c/d/e/file1.txt";
+ private static final String PARENT_DIR = "vol1/buck1/a/b/c/d/e";
+ private static final String[] TEST_NAMES =
+ new String[]{"vol1", "buck1", "a", "b", "c", "d", "e", "file1.txt"};
+ private static final String TEST_KEY_NAMES = "a/b/c/d/e/file1.txt";
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String VOL_TWO = "vol2";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String BUCKET_THREE = "bucket3";
+ private static final String BUCKET_FOUR = "bucket4";
+ private static final String KEY_ONE = "file1";
+ private static final String KEY_TWO = "file2";
+ private static final String KEY_THREE = "file3";
+ private static final String KEY_FOUR = "file4";
+ private static final String KEY_FIVE = "file5";
+ private static final String KEY_SIX = "file6";
+ private static final String KEY_SEVEN = "file7";
+ private static final String KEY_EIGHT = "file8";
+ private static final String KEY_NINE = "file9";
+ private static final String KEY_TEN = "file10";
+ private static final String KEY_ELEVEN = "file11";
+ private static final String MULTI_BLOCK_KEY = "file7";
+ private static final String MULTI_BLOCK_FILE = "file7";
+
+ private static final String FILE_ONE = "file1";
+ private static final String FILE_TWO = "file2";
+ private static final String FILE_THREE = "file3";
+ private static final String FILE_FOUR = "file4";
+ private static final String FILE_FIVE = "file5";
+ private static final String FILE_SIX = "file6";
+ private static final String FILE_SEVEN = "file7";
+ private static final String FILE_EIGHT = "file8";
+ private static final String FILE_NINE = "file9";
+ private static final String FILE_TEN = "file10";
+ private static final String FILE_ELEVEN = "file11";
+ // objects IDs
+ private static final long PARENT_OBJECT_ID_ZERO = 0L;
+ private static final long VOL_OBJECT_ID = 0L;
+ private static final long BUCKET_ONE_OBJECT_ID = 1L;
+ private static final long BUCKET_TWO_OBJECT_ID = 2L;
+ private static final long KEY_ONE_OBJECT_ID = 3L;
+ private static final long KEY_TWO_OBJECT_ID = 5L;
+ private static final long KEY_FOUR_OBJECT_ID = 6L;
+ private static final long KEY_THREE_OBJECT_ID = 8L;
+ private static final long KEY_FIVE_OBJECT_ID = 9L;
+ private static final long KEY_SIX_OBJECT_ID = 10L;
+ private static final long MULTI_BLOCK_KEY_OBJECT_ID = 13L;
+ private static final long KEY_SEVEN_OBJECT_ID = 13L;
+ private static final long VOL_TWO_OBJECT_ID = 14L;
+ private static final long BUCKET_THREE_OBJECT_ID = 15L;
+ private static final long BUCKET_FOUR_OBJECT_ID = 16L;
+ private static final long KEY_EIGHT_OBJECT_ID = 17L;
+ private static final long KEY_NINE_OBJECT_ID = 19L;
+ private static final long KEY_TEN_OBJECT_ID = 20L;
+ private static final long KEY_ELEVEN_OBJECT_ID = 21L;
+
+ // container IDs
+ private static final long CONTAINER_ONE_ID = 1L;
+ private static final long CONTAINER_TWO_ID = 2L;
+ private static final long CONTAINER_THREE_ID = 3L;
+ private static final long CONTAINER_FOUR_ID = 4L;
+ private static final long CONTAINER_FIVE_ID = 5L;
+ private static final long CONTAINER_SIX_ID = 6L;
+
+ // replication factors
+ private static final int CONTAINER_ONE_REPLICA_COUNT = 3;
+ private static final int CONTAINER_TWO_REPLICA_COUNT = 2;
+ private static final int CONTAINER_THREE_REPLICA_COUNT = 4;
+ private static final int CONTAINER_FOUR_REPLICA_COUNT = 5;
+ private static final int CONTAINER_FIVE_REPLICA_COUNT = 2;
+ private static final int CONTAINER_SIX_REPLICA_COUNT = 3;
+
+ // block lengths
+ private static final long BLOCK_ONE_LENGTH = 1000L;
+ private static final long BLOCK_TWO_LENGTH = 2000L;
+ private static final long BLOCK_THREE_LENGTH = 3000L;
+ private static final long BLOCK_FOUR_LENGTH = 4000L;
+ private static final long BLOCK_FIVE_LENGTH = 5000L;
+ private static final long BLOCK_SIX_LENGTH = 6000L;
+
+ // data size in bytes
+ private static final long KEY_ONE_SIZE = 500L; // bin 0
+ private static final long KEY_TWO_SIZE = OzoneConsts.KB + 1; // bin 1
+ private static final long KEY_THREE_SIZE = 4 * OzoneConsts.KB + 1; // bin 3
+ private static final long KEY_FOUR_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_FIVE_SIZE = 100L; // bin 0
+ private static final long KEY_SIX_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_SEVEN_SIZE = 4 * OzoneConsts.KB + 1;
+ private static final long KEY_EIGHT_SIZE = OzoneConsts.KB + 1; // bin 1
+ private static final long KEY_NINE_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_TEN_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_ELEVEN_SIZE = OzoneConsts.KB + 1; // bin 1
+
+ private static final long FILE1_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_ONE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE2_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_TWO_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE3_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_THREE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE4_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_FOUR_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE5_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_FIVE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE6_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_SIX_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));;
+ private static final long FILE7_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_SEVEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));;
+ private static final long FILE8_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_EIGHT_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE9_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_NINE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE10_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_TEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE11_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_ELEVEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+
+ private static final long MULTI_BLOCK_KEY_SIZE_WITH_REPLICA
+ = FILE7_SIZE_WITH_REPLICA;
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_ROOT
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA
+ + FILE4_SIZE_WITH_REPLICA
+ + FILE5_SIZE_WITH_REPLICA
+ + FILE8_SIZE_WITH_REPLICA
+ + FILE9_SIZE_WITH_REPLICA
+ + FILE10_SIZE_WITH_REPLICA
+ + FILE11_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_VOL
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA
+ + FILE4_SIZE_WITH_REPLICA
+ + FILE5_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_BUCKET1
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_KEY
+ = FILE4_SIZE_WITH_REPLICA;
+
+ // quota in bytes
+ private static final long ROOT_QUOTA = 2 * (2 * OzoneConsts.MB);
+ private static final long VOL_QUOTA = 2 * OzoneConsts.MB;
+ private static final long VOL_TWO_QUOTA = 2 * OzoneConsts.MB;
+ private static final long BUCKET_ONE_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_TWO_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_THREE_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_FOUR_QUOTA = OzoneConsts.MB;
+
+ // mock client's path requests
+ private static final String TEST_USER = "TestUser";
+ private static final String ROOT_PATH = "/";
+ private static final String VOL_PATH = "/vol";
+ private static final String VOL_TWO_PATH = "/vol2";
+ private static final String BUCKET_ONE_PATH = "/vol/bucket1";
+ private static final String BUCKET_TWO_PATH = "/vol/bucket2";
+ private static final String KEY_PATH = "/vol/bucket2/file4";
+ private static final String MULTI_BLOCK_KEY_PATH = "/vol/bucket1/file7";
+ private static final String INVALID_PATH = "/vol/path/not/found";
+
+ // some expected answers
+ private static final long ROOT_DATA_SIZE =
+ KEY_ONE_SIZE + KEY_TWO_SIZE + KEY_THREE_SIZE + KEY_FOUR_SIZE +
+ KEY_FIVE_SIZE + KEY_EIGHT_SIZE + KEY_NINE_SIZE + KEY_TEN_SIZE +
+ KEY_ELEVEN_SIZE;
+ private static final long VOL_DATA_SIZE = KEY_ONE_SIZE + KEY_TWO_SIZE +
+ KEY_THREE_SIZE + KEY_FOUR_SIZE + KEY_FIVE_SIZE;
+
+ private static final long VOL_TWO_DATA_SIZE =
+ KEY_EIGHT_SIZE + KEY_NINE_SIZE + KEY_TEN_SIZE + KEY_ELEVEN_SIZE;
+
+ private static final long BUCKET_ONE_DATA_SIZE = KEY_ONE_SIZE + KEY_TWO_SIZE +
+ KEY_THREE_SIZE;
+
+ private static final long BUCKET_TWO_DATA_SIZE =
+ KEY_FOUR_SIZE + KEY_FIVE_SIZE;
+
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ conf = new OzoneConfiguration();
+ OMMetadataManager omMetadataManager = initializeNewOmMetadataManager(
+ Files.createDirectory(temporaryFolder.resolve(
+ "JunitOmDBDir")).toFile(), conf);
+ OzoneManagerServiceProviderImpl ozoneManagerServiceProvider =
+ getMockOzoneManagerServiceProvider();
+ reconOMMetadataManager = getTestReconOmMetadataManager(omMetadataManager,
+ Files.createDirectory(temporaryFolder.resolve(
+ "omMetadatDir")).toFile());
+
+ ReconTestInjector reconTestInjector =
+ new ReconTestInjector.Builder(temporaryFolder.toFile())
+ .withReconOm(reconOMMetadataManager)
+ .withOmServiceProvider(ozoneManagerServiceProvider)
+ .withReconSqlDb()
+ .withContainerDB()
+ .addBinding(OzoneStorageContainerManager.class,
+ getMockReconSCM())
+ .addBinding(StorageContainerServiceProvider.class,
+ mock(StorageContainerServiceProviderImpl.class))
+ .addBinding(NSSummaryEndpoint.class)
+ .build();
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager =
+ reconTestInjector.getInstance(ReconNamespaceSummaryManager.class);
+ nsSummaryEndpoint = reconTestInjector.getInstance(NSSummaryEndpoint.class);
+
+ // populate OM DB and reprocess into Recon RocksDB
+ populateOMDB();
+ NSSummaryTaskWithOBS nsSummaryTaskWithOBS =
+ new NSSummaryTaskWithOBS(reconNamespaceSummaryManager,
+ reconOMMetadataManager, conf);
+ nsSummaryTaskWithOBS.reprocessWithOBS(reconOMMetadataManager);
+ commonUtils = new CommonUtils();
+ }
+
+ @Test
+ public void testUtility() {
+ String[] names = EntityHandler.parseRequestPath(TEST_PATH_UTILITY);
+ assertArrayEquals(TEST_NAMES, names);
+ String keyName = BucketHandler.getKeyName(names);
+ assertEquals(TEST_KEY_NAMES, keyName);
+ String subpath = BucketHandler.buildSubpath(PARENT_DIR, "file1.txt");
+ assertEquals(TEST_PATH_UTILITY, subpath);
+ }
+
+ @Test
+ public void testGetBasicInfoRoot() throws Exception {
+ // Test root basics
+ Response rootResponse = nsSummaryEndpoint.getBasicInfo(ROOT_PATH);
+ NamespaceSummaryResponse rootResponseObj =
+ (NamespaceSummaryResponse) rootResponse.getEntity();
+ assertEquals(EntityType.ROOT, rootResponseObj.getEntityType());
+ assertEquals(2, rootResponseObj.getCountStats().getNumVolume());
+ assertEquals(4, rootResponseObj.getCountStats().getNumBucket());
+ assertEquals(9, rootResponseObj.getCountStats().getNumTotalKey());
+ }
+
+ @Test
+ public void testGetBasicInfoVol() throws Exception {
+ // Test volume basics
+ Response volResponse = nsSummaryEndpoint.getBasicInfo(VOL_PATH);
+ NamespaceSummaryResponse volResponseObj =
+ (NamespaceSummaryResponse) volResponse.getEntity();
+ assertEquals(EntityType.VOLUME,
+ volResponseObj.getEntityType());
+ assertEquals(2, volResponseObj.getCountStats().getNumBucket());
+ assertEquals(5, volResponseObj.getCountStats().getNumTotalKey());
+ assertEquals("TestUser", ((VolumeObjectDBInfo) volResponseObj.
+ getObjectDBInfo()).getAdmin());
+ assertEquals("TestUser", ((VolumeObjectDBInfo) volResponseObj.
+ getObjectDBInfo()).getOwner());
+ assertEquals("vol", volResponseObj.getObjectDBInfo().getName());
+ assertEquals(2097152, volResponseObj.getObjectDBInfo().getQuotaInBytes());
+ assertEquals(-1, volResponseObj.getObjectDBInfo().getQuotaInNamespace());
+ }
+
+ @Test
+ public void testGetBasicInfoBucketOne() throws Exception {
+ // Test bucket 1's basics
+ Response bucketOneResponse =
+ nsSummaryEndpoint.getBasicInfo(BUCKET_ONE_PATH);
+ NamespaceSummaryResponse bucketOneObj =
+ (NamespaceSummaryResponse) bucketOneResponse.getEntity();
+ assertEquals(EntityType.BUCKET, bucketOneObj.getEntityType());
+ assertEquals(3, bucketOneObj.getCountStats().getNumTotalKey());
+ assertEquals("vol",
+ ((BucketObjectDBInfo) bucketOneObj.getObjectDBInfo()).getVolumeName());
+ assertEquals(StorageType.DISK,
+ ((BucketObjectDBInfo)
+ bucketOneObj.getObjectDBInfo()).getStorageType());
+ assertEquals(getBucketLayout(),
+ ((BucketObjectDBInfo)
+ bucketOneObj.getObjectDBInfo()).getBucketLayout());
+ assertEquals("bucket1",
+ ((BucketObjectDBInfo) bucketOneObj.getObjectDBInfo()).getName());
+ }
+
+ @Test
+ public void testGetBasicInfoBucketTwo() throws Exception {
+ // Test bucket 2's basics
+ commonUtils.testNSSummaryBasicInfoBucketTwo(
+ BucketLayout.OBJECT_STORE,
+ nsSummaryEndpoint);
+ }
+
+ @Test
+ public void testGetBasicInfoNoPath() throws Exception {
+ // Test invalid path
+ commonUtils.testNSSummaryBasicInfoNoPath(nsSummaryEndpoint);
+ }
+
+ @Test
+ public void testGetBasicInfoKey() throws Exception {
+ // Test key
+ commonUtils.testNSSummaryBasicInfoKey(nsSummaryEndpoint);
+ }
+
+ @Test
+ public void testDiskUsageRoot() throws Exception {
+ // root level DU
+ Response rootResponse = nsSummaryEndpoint.getDiskUsage(ROOT_PATH,
+ false, false);
+ DUResponse duRootRes = (DUResponse) rootResponse.getEntity();
+ assertEquals(2, duRootRes.getCount());
+ List<DUResponse.DiskUsage> duRootData = duRootRes.getDuData();
+ // sort based on subpath
+ Collections.sort(duRootData,
+ Comparator.comparing(DUResponse.DiskUsage::getSubpath));
+ DUResponse.DiskUsage duVol1 = duRootData.get(0);
+ DUResponse.DiskUsage duVol2 = duRootData.get(1);
+ assertEquals(VOL_PATH, duVol1.getSubpath());
+ assertEquals(VOL_TWO_PATH, duVol2.getSubpath());
+ assertEquals(VOL_DATA_SIZE, duVol1.getSize());
+ assertEquals(VOL_TWO_DATA_SIZE, duVol2.getSize());
+ }
+
+ @Test
+ public void testDiskUsageVolume() throws Exception {
+ // volume level DU
+ Response volResponse = nsSummaryEndpoint.getDiskUsage(VOL_PATH,
+ false, false);
+ DUResponse duVolRes = (DUResponse) volResponse.getEntity();
+ assertEquals(2, duVolRes.getCount());
+ List<DUResponse.DiskUsage> duData = duVolRes.getDuData();
+ // sort based on subpath
+ Collections.sort(duData,
+ Comparator.comparing(DUResponse.DiskUsage::getSubpath));
+ DUResponse.DiskUsage duBucket1 = duData.get(0);
+ DUResponse.DiskUsage duBucket2 = duData.get(1);
+ assertEquals(BUCKET_ONE_PATH, duBucket1.getSubpath());
+ assertEquals(BUCKET_TWO_PATH, duBucket2.getSubpath());
+ assertEquals(BUCKET_ONE_DATA_SIZE, duBucket1.getSize());
+ assertEquals(BUCKET_TWO_DATA_SIZE, duBucket2.getSize());
+ }
+
+ @Test
+ public void testDiskUsageBucket() throws Exception {
+ // bucket level DU
+ Response bucketResponse = nsSummaryEndpoint.getDiskUsage(BUCKET_ONE_PATH,
+ false, false);
+ DUResponse duBucketResponse = (DUResponse) bucketResponse.getEntity();
+ // There are no sub-paths under this OBS bucket.
+ assertEquals(0, duBucketResponse.getCount());
+ assertEquals(BUCKET_ONE_DATA_SIZE, duBucketResponse.getSize());
+ }
+
+ @Test
+ public void testDiskUsageKey() throws Exception {
+ // key level DU
+ Response keyResponse = nsSummaryEndpoint.getDiskUsage(KEY_PATH,
+ false, false);
+ DUResponse keyObj = (DUResponse) keyResponse.getEntity();
+ assertEquals(0, keyObj.getCount());
+ assertEquals(KEY_FOUR_SIZE, keyObj.getSize());
+ }
Review Comment:
What's the difference between this and the `testGetBasicInfoKey()` test? Also the `testGetBasicInfoKey()` from the `CommonUtils` seems hard to use in other test classes, or at least we should create the exact same key with the same size, replication, ... to be able to run the test successfully.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1504085417
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestNSSummaryEndpointWithOBS.java:
##########
@@ -0,0 +1,1133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor.ONE;
+import static org.apache.hadoop.ozone.om.helpers.QuotaUtil.getReplicatedSize;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.StorageType;
+import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.container.ContainerReplica;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.handlers.BucketHandler;
+import org.apache.hadoop.ozone.recon.api.handlers.EntityHandler;
+import org.apache.hadoop.ozone.recon.api.types.BucketObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NamespaceSummaryResponse;
+import org.apache.hadoop.ozone.recon.api.types.QuotaUsageResponse;
+import org.apache.hadoop.ozone.recon.api.types.ResponseStatus;
+import org.apache.hadoop.ozone.recon.api.types.VolumeObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.FileSizeDistributionResponse;
+import org.apache.hadoop.ozone.recon.common.CommonUtils;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.scm.ReconNodeManager;
+import org.apache.hadoop.ozone.recon.scm.ReconStorageContainerManagerFacade;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.StorageContainerServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.spi.impl.StorageContainerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.tasks.NSSummaryTaskWithOBS;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import javax.ws.rs.core.Response;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.HashSet;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.apache.hadoop.hdds.protocol.MockDatanodeDetails.randomDatanodeDetails;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProvider;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for NSSummary REST APIs with OBS.
+ * Testing is done on a simple object store model with a flat hierarchy:
+ * vol
+ * / \
+ * bucket1 bucket2
+ * | |
+ * file1 file2
+ *
+ * This tests the Rest APIs for NSSummary in the context of OBS buckets,
+ * focusing on disk usage, quota usage, and file size distribution.
+ */
+public class TestNSSummaryEndpointWithOBS {
+ @TempDir
+ private Path temporaryFolder;
+
+ private ReconOMMetadataManager reconOMMetadataManager;
+ private NSSummaryEndpoint nsSummaryEndpoint;
+ private OzoneConfiguration conf;
+ private CommonUtils commonUtils;
+
+ private static final String TEST_PATH_UTILITY =
+ "/vol1/buck1/a/b/c/d/e/file1.txt";
+ private static final String PARENT_DIR = "vol1/buck1/a/b/c/d/e";
+ private static final String[] TEST_NAMES =
+ new String[]{"vol1", "buck1", "a", "b", "c", "d", "e", "file1.txt"};
+ private static final String TEST_KEY_NAMES = "a/b/c/d/e/file1.txt";
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String VOL_TWO = "vol2";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String BUCKET_THREE = "bucket3";
+ private static final String BUCKET_FOUR = "bucket4";
+ private static final String KEY_ONE = "file1";
+ private static final String KEY_TWO = "file2";
+ private static final String KEY_THREE = "file3";
+ private static final String KEY_FOUR = "file4";
+ private static final String KEY_FIVE = "file5";
+ private static final String KEY_SIX = "file6";
+ private static final String KEY_SEVEN = "file7";
+ private static final String KEY_EIGHT = "file8";
+ private static final String KEY_NINE = "file9";
+ private static final String KEY_TEN = "file10";
+ private static final String KEY_ELEVEN = "file11";
+ private static final String MULTI_BLOCK_KEY = "file7";
+ private static final String MULTI_BLOCK_FILE = "file7";
+
+ private static final String FILE_ONE = "file1";
+ private static final String FILE_TWO = "file2";
+ private static final String FILE_THREE = "file3";
+ private static final String FILE_FOUR = "file4";
+ private static final String FILE_FIVE = "file5";
+ private static final String FILE_SIX = "file6";
+ private static final String FILE_SEVEN = "file7";
+ private static final String FILE_EIGHT = "file8";
+ private static final String FILE_NINE = "file9";
+ private static final String FILE_TEN = "file10";
+ private static final String FILE_ELEVEN = "file11";
+ // objects IDs
+ private static final long PARENT_OBJECT_ID_ZERO = 0L;
+ private static final long VOL_OBJECT_ID = 0L;
+ private static final long BUCKET_ONE_OBJECT_ID = 1L;
+ private static final long BUCKET_TWO_OBJECT_ID = 2L;
+ private static final long KEY_ONE_OBJECT_ID = 3L;
+ private static final long KEY_TWO_OBJECT_ID = 5L;
+ private static final long KEY_FOUR_OBJECT_ID = 6L;
+ private static final long KEY_THREE_OBJECT_ID = 8L;
+ private static final long KEY_FIVE_OBJECT_ID = 9L;
+ private static final long KEY_SIX_OBJECT_ID = 10L;
+ private static final long MULTI_BLOCK_KEY_OBJECT_ID = 13L;
+ private static final long KEY_SEVEN_OBJECT_ID = 13L;
+ private static final long VOL_TWO_OBJECT_ID = 14L;
+ private static final long BUCKET_THREE_OBJECT_ID = 15L;
+ private static final long BUCKET_FOUR_OBJECT_ID = 16L;
+ private static final long KEY_EIGHT_OBJECT_ID = 17L;
+ private static final long KEY_NINE_OBJECT_ID = 19L;
+ private static final long KEY_TEN_OBJECT_ID = 20L;
+ private static final long KEY_ELEVEN_OBJECT_ID = 21L;
+
+ // container IDs
+ private static final long CONTAINER_ONE_ID = 1L;
+ private static final long CONTAINER_TWO_ID = 2L;
+ private static final long CONTAINER_THREE_ID = 3L;
+ private static final long CONTAINER_FOUR_ID = 4L;
+ private static final long CONTAINER_FIVE_ID = 5L;
+ private static final long CONTAINER_SIX_ID = 6L;
+
+ // replication factors
+ private static final int CONTAINER_ONE_REPLICA_COUNT = 3;
+ private static final int CONTAINER_TWO_REPLICA_COUNT = 2;
+ private static final int CONTAINER_THREE_REPLICA_COUNT = 4;
+ private static final int CONTAINER_FOUR_REPLICA_COUNT = 5;
+ private static final int CONTAINER_FIVE_REPLICA_COUNT = 2;
+ private static final int CONTAINER_SIX_REPLICA_COUNT = 3;
+
+ // block lengths
+ private static final long BLOCK_ONE_LENGTH = 1000L;
+ private static final long BLOCK_TWO_LENGTH = 2000L;
+ private static final long BLOCK_THREE_LENGTH = 3000L;
+ private static final long BLOCK_FOUR_LENGTH = 4000L;
+ private static final long BLOCK_FIVE_LENGTH = 5000L;
+ private static final long BLOCK_SIX_LENGTH = 6000L;
+
+ // data size in bytes
+ private static final long KEY_ONE_SIZE = 500L; // bin 0
+ private static final long KEY_TWO_SIZE = OzoneConsts.KB + 1; // bin 1
+ private static final long KEY_THREE_SIZE = 4 * OzoneConsts.KB + 1; // bin 3
+ private static final long KEY_FOUR_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_FIVE_SIZE = 100L; // bin 0
+ private static final long KEY_SIX_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_SEVEN_SIZE = 4 * OzoneConsts.KB + 1;
+ private static final long KEY_EIGHT_SIZE = OzoneConsts.KB + 1; // bin 1
+ private static final long KEY_NINE_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_TEN_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_ELEVEN_SIZE = OzoneConsts.KB + 1; // bin 1
+
+ private static final long FILE1_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_ONE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE2_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_TWO_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE3_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_THREE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE4_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_FOUR_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE5_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_FIVE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE6_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_SIX_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));;
+ private static final long FILE7_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_SEVEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));;
+ private static final long FILE8_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_EIGHT_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE9_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_NINE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE10_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_TEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE11_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_ELEVEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+
+ private static final long MULTI_BLOCK_KEY_SIZE_WITH_REPLICA
+ = FILE7_SIZE_WITH_REPLICA;
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_ROOT
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA
+ + FILE4_SIZE_WITH_REPLICA
+ + FILE5_SIZE_WITH_REPLICA
+ + FILE8_SIZE_WITH_REPLICA
+ + FILE9_SIZE_WITH_REPLICA
+ + FILE10_SIZE_WITH_REPLICA
+ + FILE11_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_VOL
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA
+ + FILE4_SIZE_WITH_REPLICA
+ + FILE5_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_BUCKET1
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_KEY
+ = FILE4_SIZE_WITH_REPLICA;
+
+ // quota in bytes
+ private static final long ROOT_QUOTA = 2 * (2 * OzoneConsts.MB);
+ private static final long VOL_QUOTA = 2 * OzoneConsts.MB;
+ private static final long VOL_TWO_QUOTA = 2 * OzoneConsts.MB;
+ private static final long BUCKET_ONE_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_TWO_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_THREE_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_FOUR_QUOTA = OzoneConsts.MB;
+
+ // mock client's path requests
+ private static final String TEST_USER = "TestUser";
+ private static final String ROOT_PATH = "/";
+ private static final String VOL_PATH = "/vol";
+ private static final String VOL_TWO_PATH = "/vol2";
+ private static final String BUCKET_ONE_PATH = "/vol/bucket1";
+ private static final String BUCKET_TWO_PATH = "/vol/bucket2";
+ private static final String KEY_PATH = "/vol/bucket2/file4";
+ private static final String MULTI_BLOCK_KEY_PATH = "/vol/bucket1/file7";
+ private static final String INVALID_PATH = "/vol/path/not/found";
Review Comment:
Done!
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestNSSummaryEndpointWithOBS.java:
##########
@@ -0,0 +1,1133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor.ONE;
+import static org.apache.hadoop.ozone.om.helpers.QuotaUtil.getReplicatedSize;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.StorageType;
+import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.container.ContainerReplica;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.handlers.BucketHandler;
+import org.apache.hadoop.ozone.recon.api.handlers.EntityHandler;
+import org.apache.hadoop.ozone.recon.api.types.BucketObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NamespaceSummaryResponse;
+import org.apache.hadoop.ozone.recon.api.types.QuotaUsageResponse;
+import org.apache.hadoop.ozone.recon.api.types.ResponseStatus;
+import org.apache.hadoop.ozone.recon.api.types.VolumeObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.FileSizeDistributionResponse;
+import org.apache.hadoop.ozone.recon.common.CommonUtils;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.scm.ReconNodeManager;
+import org.apache.hadoop.ozone.recon.scm.ReconStorageContainerManagerFacade;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.StorageContainerServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.spi.impl.StorageContainerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.tasks.NSSummaryTaskWithOBS;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import javax.ws.rs.core.Response;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.HashSet;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.apache.hadoop.hdds.protocol.MockDatanodeDetails.randomDatanodeDetails;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProvider;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for NSSummary REST APIs with OBS.
+ * Testing is done on a simple object store model with a flat hierarchy:
+ * vol
+ * / \
+ * bucket1 bucket2
+ * | |
+ * file1 file2
+ *
+ * This tests the Rest APIs for NSSummary in the context of OBS buckets,
+ * focusing on disk usage, quota usage, and file size distribution.
+ */
+public class TestNSSummaryEndpointWithOBS {
+ @TempDir
+ private Path temporaryFolder;
+
+ private ReconOMMetadataManager reconOMMetadataManager;
+ private NSSummaryEndpoint nsSummaryEndpoint;
+ private OzoneConfiguration conf;
+ private CommonUtils commonUtils;
+
+ private static final String TEST_PATH_UTILITY =
+ "/vol1/buck1/a/b/c/d/e/file1.txt";
+ private static final String PARENT_DIR = "vol1/buck1/a/b/c/d/e";
+ private static final String[] TEST_NAMES =
+ new String[]{"vol1", "buck1", "a", "b", "c", "d", "e", "file1.txt"};
+ private static final String TEST_KEY_NAMES = "a/b/c/d/e/file1.txt";
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String VOL_TWO = "vol2";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String BUCKET_THREE = "bucket3";
+ private static final String BUCKET_FOUR = "bucket4";
+ private static final String KEY_ONE = "file1";
+ private static final String KEY_TWO = "file2";
+ private static final String KEY_THREE = "file3";
+ private static final String KEY_FOUR = "file4";
+ private static final String KEY_FIVE = "file5";
+ private static final String KEY_SIX = "file6";
+ private static final String KEY_SEVEN = "file7";
+ private static final String KEY_EIGHT = "file8";
+ private static final String KEY_NINE = "file9";
+ private static final String KEY_TEN = "file10";
+ private static final String KEY_ELEVEN = "file11";
+ private static final String MULTI_BLOCK_KEY = "file7";
+ private static final String MULTI_BLOCK_FILE = "file7";
Review Comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1472757078
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/TestNSSummaryTaskWithOBS.java:
##########
@@ -0,0 +1,563 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.utils.db.RDBBatchOperation;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Nested;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Set;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProviderWithFSO;
+/**
+ * Unit test for NSSummaryTaskWithOBS.
+ */
+public final class TestNSSummaryTaskWithOBS implements Serializable {
+ private static ReconNamespaceSummaryManager reconNamespaceSummaryManager;
+ private static OMMetadataManager omMetadataManager;
+ private static ReconOMMetadataManager reconOMMetadataManager;
+ private static NSSummaryTaskWithOBS nSSummaryTaskWithOBS;
+ private static OzoneConfiguration omConfiguration;
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String KEY_ONE = "key1";
+ private static final String KEY_TWO = "key2";
+ private static final String KEY_THREE = "dir1/dir2/key3";
+ private static final String KEY_FOUR = "key4///////////";
+ private static final String KEY_FIVE = "//////////";
+ private static final String KEY_SIX = "key6";
+ private static final String KEY_SEVEN = "key7";
+
+ private static final String TEST_USER = "TestUser";
+
+ private static final long PARENT_OBJECT_ID_ZERO = 0L;
+ private static final long VOL_OBJECT_ID = 0L;
+ private static final long BUCKET_ONE_OBJECT_ID = 1L;
+ private static final long BUCKET_TWO_OBJECT_ID = 2L;
+ private static final long KEY_ONE_OBJECT_ID = 3L;
+ private static final long KEY_TWO_OBJECT_ID = 5L;
+ private static final long KEY_FOUR_OBJECT_ID = 6L;
+ private static final long KEY_THREE_OBJECT_ID = 8L;
+ private static final long KEY_FIVE_OBJECT_ID = 9L;
+ private static final long KEY_SIX_OBJECT_ID = 10L;
+ private static final long KEY_SEVEN_OBJECT_ID = 11L;
+
+
+ private static final long KEY_ONE_SIZE = 500L;
+ private static final long KEY_TWO_OLD_SIZE = 1025L;
+ private static final long KEY_TWO_UPDATE_SIZE = 1023L;
+ private static final long KEY_THREE_SIZE =
+ ReconConstants.MAX_FILE_SIZE_UPPER_BOUND - 100L;
+ private static final long KEY_FOUR_SIZE = 2050L;
+ private static final long KEY_FIVE_SIZE = 100L;
+ private static final long KEY_SIX_SIZE = 6000L;
+ private static final long KEY_SEVEN_SIZE = 7000L;
+
+ private TestNSSummaryTaskWithOBS() {
+ }
+
+ @BeforeAll
+ public static void setUp(@TempDir File tmpDir) throws Exception {
+ initializeNewOmMetadataManager(new File(tmpDir, "om"));
+ OzoneManagerServiceProviderImpl ozoneManagerServiceProvider =
+ getMockOzoneManagerServiceProviderWithFSO();
+ reconOMMetadataManager = getTestReconOmMetadataManager(omMetadataManager,
+ new File(tmpDir, "recon"));
+
+ ReconTestInjector reconTestInjector =
+ new ReconTestInjector.Builder(tmpDir)
+ .withReconOm(reconOMMetadataManager)
+ .withOmServiceProvider(ozoneManagerServiceProvider)
+ .withReconSqlDb()
+ .withContainerDB()
+ .build();
+ reconNamespaceSummaryManager =
+ reconTestInjector.getInstance(ReconNamespaceSummaryManager.class);
+
+ NSSummary nonExistentSummary =
+ reconNamespaceSummaryManager.getNSSummary(BUCKET_ONE_OBJECT_ID);
+ Assertions.assertNull(nonExistentSummary);
+
+ populateOMDB();
+
+ nSSummaryTaskWithOBS = new NSSummaryTaskWithOBS(
+ reconNamespaceSummaryManager,
+ reconOMMetadataManager, omConfiguration);
+ }
+
+ /**
+ * Nested class for testing NSSummaryTaskWithOBS reprocess.
+ */
+ @Nested
+ public class TestReprocess {
+
+ private NSSummary nsSummaryForBucket1;
+ private NSSummary nsSummaryForBucket2;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ // write a NSSummary prior to reprocess
+ // verify it got cleaned up after.
+ NSSummary staleNSSummary = new NSSummary();
+ RDBBatchOperation rdbBatchOperation = new RDBBatchOperation();
+ reconNamespaceSummaryManager.batchStoreNSSummaries(rdbBatchOperation, -1L,
+ staleNSSummary);
+ reconNamespaceSummaryManager.commitBatchOperation(rdbBatchOperation);
+
+ // Verify commit
+ Assertions.assertNotNull(reconNamespaceSummaryManager.getNSSummary(-1L));
+
+ // reinit Recon RocksDB's namespace CF.
+ reconNamespaceSummaryManager.clearNSSummaryTable();
+
+ nSSummaryTaskWithOBS.reprocessWithOBS(reconOMMetadataManager);
+ Assertions.assertNull(reconNamespaceSummaryManager.getNSSummary(-1L));
+
+ nsSummaryForBucket1 =
+ reconNamespaceSummaryManager.getNSSummary(BUCKET_ONE_OBJECT_ID);
+ nsSummaryForBucket2 =
+ reconNamespaceSummaryManager.getNSSummary(BUCKET_TWO_OBJECT_ID);
+ Assertions.assertNotNull(nsSummaryForBucket1);
+ Assertions.assertNotNull(nsSummaryForBucket2);
+ }
+
+ @Test
+ public void testReprocessNSSummaryNull() throws IOException {
+ Assertions.assertNull(reconNamespaceSummaryManager.getNSSummary(-1L));
+ }
+
+ @Test
+ public void testReprocessGetFiles() {
+ Assertions.assertEquals(3, nsSummaryForBucket1.getNumOfFiles());
+ Assertions.assertEquals(2, nsSummaryForBucket2.getNumOfFiles());
+
+ Assertions.assertEquals(KEY_ONE_SIZE + KEY_TWO_OLD_SIZE + KEY_THREE_SIZE,
+ nsSummaryForBucket1.getSizeOfFiles());
+ Assertions.assertEquals(KEY_FOUR_SIZE + KEY_FIVE_SIZE,
+ nsSummaryForBucket2.getSizeOfFiles());
+ }
+
+ @Test
+ public void testReprocessFileBucketSize() {
+ int[] fileDistBucket1 = nsSummaryForBucket1.getFileSizeBucket();
+ int[] fileDistBucket2 = nsSummaryForBucket2.getFileSizeBucket();
+ Assertions.assertEquals(ReconConstants.NUM_OF_FILE_SIZE_BINS,
+ fileDistBucket1.length);
+ Assertions.assertEquals(ReconConstants.NUM_OF_FILE_SIZE_BINS,
+ fileDistBucket2.length);
+
+ // Check for 1's and 0's in fileDistBucket1
+ int[] expectedIndexes1 = {0, 1, 40};
+ for (int index = 0; index < fileDistBucket1.length; index++) {
+ if (contains(expectedIndexes1, index)) {
+ Assertions.assertEquals(1, fileDistBucket1[index]);
+ } else {
+ Assertions.assertEquals(0, fileDistBucket1[index]);
+ }
+ }
+
+ // Check for 1's and 0's in fileDistBucket2
+ int[] expectedIndexes2 = {0, 2};
+ for (int index = 0; index < fileDistBucket2.length; index++) {
+ if (contains(expectedIndexes2, index)) {
+ Assertions.assertEquals(1, fileDistBucket2[index]);
+ } else {
+ Assertions.assertEquals(0, fileDistBucket2[index]);
+ }
+ }
+ }
+
+ @Test
+ public void testReprocessBucketDirs() {
+ // None of the buckets have any child dirs because OBS is flat namespace.
+ Set<Long> childDirBucketOne = nsSummaryForBucket1.getChildDir();
+ Set<Long> childDirBucketTwo = nsSummaryForBucket2.getChildDir();
+ Assertions.assertEquals(0, childDirBucketOne.size());
+ Assertions.assertEquals(0, childDirBucketTwo.size());
+ }
+
+ }
+
+ /**
+ * Nested class for testing NSSummaryTaskWithLegacy process.
+ */
+ @Nested
+ public class TestProcess {
+
+ private NSSummary nsSummaryForBucket1;
+ private NSSummary nsSummaryForBucket2;
+
+ private OMDBUpdateEvent keyEvent1;
+ private OMDBUpdateEvent keyEvent2;
+ private OMDBUpdateEvent keyEvent3;
+ private OMDBUpdateEvent keyEvent4;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ // reinit Recon RocksDB's namespace CF.
+ reconNamespaceSummaryManager.clearNSSummaryTable();
+ nSSummaryTaskWithOBS.reprocessWithOBS(reconOMMetadataManager);
+ nSSummaryTaskWithOBS.processWithOBS(processEventBatch());
+
+ nsSummaryForBucket1 =
+ reconNamespaceSummaryManager.getNSSummary(BUCKET_ONE_OBJECT_ID);
+ Assertions.assertNotNull(nsSummaryForBucket1);
+ nsSummaryForBucket2 =
+ reconNamespaceSummaryManager.getNSSummary(BUCKET_TWO_OBJECT_ID);
+ Assertions.assertNotNull(nsSummaryForBucket2);
+ }
+
+ private OMUpdateEventBatch processEventBatch() throws IOException {
+ // Test PUT Event.
+ // PUT Key6 in Bucket2.
+ String omPutKey =
+ OM_KEY_PREFIX + VOL
+ + OM_KEY_PREFIX + BUCKET_TWO +
+ OM_KEY_PREFIX + KEY_SIX;
+ OmKeyInfo omPutKeyInfo = buildOmKeyInfo(VOL, BUCKET_TWO, KEY_SIX,
+ KEY_SIX, KEY_SIX_OBJECT_ID, BUCKET_TWO_OBJECT_ID, KEY_SIX_SIZE);
+ keyEvent1 = new OMDBUpdateEvent.
+ OMUpdateEventBuilder<String, OmKeyInfo>()
+ .setKey(omPutKey)
+ .setValue(omPutKeyInfo)
+ .setTable(omMetadataManager.getKeyTable(getBucketLayout())
+ .getName())
+ .setAction(OMDBUpdateEvent.OMDBUpdateAction.PUT)
+ .build();
+ // PUT Key7 in Bucket1.
+ omPutKey =
+ OM_KEY_PREFIX + VOL
+ + OM_KEY_PREFIX + BUCKET_ONE +
+ OM_KEY_PREFIX + KEY_SEVEN;
+ omPutKeyInfo = buildOmKeyInfo(VOL, BUCKET_ONE, KEY_SEVEN,
+ KEY_SEVEN, KEY_SEVEN_OBJECT_ID, BUCKET_ONE_OBJECT_ID, KEY_SEVEN_SIZE);
+ keyEvent2 = new OMDBUpdateEvent.
+ OMUpdateEventBuilder<String, OmKeyInfo>()
+ .setKey(omPutKey)
+ .setValue(omPutKeyInfo)
+ .setTable(omMetadataManager.getKeyTable(getBucketLayout())
+ .getName())
+ .setAction(OMDBUpdateEvent.OMDBUpdateAction.PUT)
+ .build();
+
+ // Test DELETE Event.
+ // Delete Key1 in Bucket1.
+ String omDeleteKey =
+ OM_KEY_PREFIX + VOL
+ + OM_KEY_PREFIX + BUCKET_ONE +
+ OM_KEY_PREFIX + KEY_ONE;
+ OmKeyInfo omDeleteKeyInfo = buildOmKeyInfo(VOL, BUCKET_ONE, KEY_ONE,
+ KEY_ONE, KEY_ONE_OBJECT_ID, BUCKET_ONE_OBJECT_ID, KEY_ONE_SIZE);
+ keyEvent3 = new OMDBUpdateEvent.
+ OMUpdateEventBuilder<String, OmKeyInfo>()
+ .setKey(omDeleteKey)
+ .setTable(omMetadataManager.getKeyTable(getBucketLayout())
+ .getName())
+ .setValue(omDeleteKeyInfo)
+ .setAction(OMDBUpdateEvent.OMDBUpdateAction.DELETE)
+ .build();
+
+ // Test UPDATE Event.
+ // Resize Key2 in Bucket1.
+ String omResizeKey =
+ OM_KEY_PREFIX + VOL
+ + OM_KEY_PREFIX + BUCKET_ONE +
+ OM_KEY_PREFIX + KEY_TWO;
+ OmKeyInfo oldOmResizeKeyInfo =
+ buildOmKeyInfo(VOL, BUCKET_ONE, KEY_TWO, KEY_TWO, KEY_TWO_OBJECT_ID,
+ BUCKET_ONE_OBJECT_ID, KEY_TWO_OLD_SIZE);
+ OmKeyInfo newOmResizeKeyInfo =
+ buildOmKeyInfo(VOL, BUCKET_ONE, KEY_TWO, KEY_TWO, KEY_TWO_OBJECT_ID,
+ BUCKET_ONE_OBJECT_ID, KEY_TWO_OLD_SIZE + 100);
+ keyEvent4 = new OMDBUpdateEvent.
+ OMUpdateEventBuilder<String, OmKeyInfo>()
+ .setKey(omResizeKey)
+ .setOldValue(oldOmResizeKeyInfo)
+ .setValue(newOmResizeKeyInfo)
+ .setTable(omMetadataManager.getKeyTable(getBucketLayout())
+ .getName())
+ .setAction(OMDBUpdateEvent.OMDBUpdateAction.UPDATE)
+ .build();
+
+ OMUpdateEventBatch omUpdateEventBatch =
+ new OMUpdateEventBatch(new ArrayList<OMDBUpdateEvent>() {
+ {
+ add(keyEvent1);
+ add(keyEvent2);
+ add(keyEvent3);
+ add(keyEvent4);
+ }
+ });
Review Comment:
Use `Arrays.asList()`, similar to:
https://github.com/apache/ozone/blob/fb2caac8ba7bf646d246cc2bd3138bdf37167e66/hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/TestNSSummaryTaskWithFSO.java#L393-L396
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on PR #4245:
URL: https://github.com/apache/ozone/pull/4245#issuecomment-1843339512
> > The decision to reconsider the proposed approach for creating namespace summaries for OBS keys with prefix-based filtering was made after discussions that happened in the [document](https://docs.google.com/document/d/1g2NCR-kgcSfLFQRBhZfYsMZmaZ6edlA_26Pq0-qfzT8/edit) They expressed concerns about two key aspects:
> >
> > 1. **Performance Impact**: We are concerned that the introduction of synthetic paths as outlined in the proposal could potentially have a negative impact on Recon performance. The additional complexity of generating and managing these synthetic paths might consume more resources and potentially slow down the system.
> > 2. **Lack of Business Use Case**: There were questions the practical necessity of introducing this complexity for (OBS). In contrast to the Filesystem (FSO) layout, where directory tables exist, it was suggested that there might not be a strong business need to create synthetic paths and namespace summaries for individual OBS keys. As a result, they proposed focusing on generating bucket-level summaries for OBS keys as a more pragmatic approach.
> >
> > Following these discussions, we've chosen to proceed with generating disk usage statistics at the bucket level. We've decided to put the prefix-based filtering on hold for now. If it becomes necessary in the future, we'll seek input from the community. For more information, please refer to the [document](https://docs.google.com/document/d/1g2NCR-kgcSfLFQRBhZfYsMZmaZ6edlA_26Pq0-qfzT8/edit).
> > cc: @xBis7 @devmadhuu @adoroszlai @devmadhuu @swagle
>
> Thanks, @ArafatKhan2198 for starting this discussion and putting together ideas in a doc. +1 for bucket-level summaries for OBS.
>
> Are we supporting Legacy with filesystem.paths enabled? I would lean on the side of not adding support for this scenario.
Yes, we are currently providing support for keys from legacy buckets with fileSystemEnabled in NSSummary. However, we do not offer support for legacy buckets when the config is turned off. Nonetheless, extending support for legacy buckets (**with filesystem.paths disabled**) should not present a significant challenge, as the code is largely identical, and we can invoke the OBS bucket handlers in that specific scenario. In essence, an OBS bucket behaves much like a legacy bucket with filesystem.paths disabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1380330596
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTask.java:
##########
@@ -101,6 +105,11 @@ public Pair<String, Boolean> process(OMUpdateEventBatch events) {
} else {
LOG.error("processWithFSO failed.");
}
+ if (success) {
Review Comment:
Pls check if it is really necessary this dependency of success on one another ?
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTask.java:
##########
@@ -121,6 +130,8 @@ public Pair<String, Boolean> reprocess(OMMetadataManager omMetadataManager) {
.reprocessWithFSO(omMetadataManager));
tasks.add(() -> nsSummaryTaskWithLegacy
.reprocessWithLegacy(reconOMMetadataManager));
+ tasks.add(() -> nsSummaryTaskWithOBS
+ .reprocessWithOBS(reconOMMetadataManager));
List<Future<Boolean>> results;
ExecutorService executorService = Executors
Review Comment:
We can think of increasing the fixed number of threads to 3, if want to run in parallel, but this may back fire as well because resources in recon are limited, however it will be good if we can analyse and test the resource usage with minimum of 10 million add key events or test with limited number of events depending on availability of system resources to have an idea how Recon will perform. Else after finishing of previous 2 tasks, this OBS taks will run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on PR #4245:
URL: https://github.com/apache/ozone/pull/4245#issuecomment-1969602385
@dombizita Could you take a look now!
Some of the changes missed out in the previous commit for some reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "dombizita (via GitHub)" <gi...@apache.org>.
dombizita commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1506086973
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/TestNSSummaryTaskWithOBS.java:
##########
@@ -0,0 +1,548 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.utils.db.RDBBatchOperation;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Nested;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Set;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProviderWithFSO;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+/**
+ * Unit test for NSSummaryTaskWithOBS.
+ */
+public final class TestNSSummaryTaskWithOBS implements Serializable {
+ private static ReconNamespaceSummaryManager reconNamespaceSummaryManager;
+ private static OMMetadataManager omMetadataManager;
+ private static ReconOMMetadataManager reconOMMetadataManager;
+ private static NSSummaryTaskWithOBS nSSummaryTaskWithOBS;
+ private static OzoneConfiguration omConfiguration;
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String KEY_ONE = "key1";
+ private static final String KEY_TWO = "key2";
+ private static final String KEY_THREE = "dir1/dir2/key3";
+ private static final String KEY_FOUR = "key4///////////";
+ private static final String KEY_FIVE = "//////////";
+ private static final String KEY_SIX = "key6";
+ private static final String KEY_SEVEN = "key7";
+
+ private static final String TEST_USER = "TestUser";
+
+ private static final long PARENT_OBJECT_ID_ZERO = 0L;
+ private static final long VOL_OBJECT_ID = 0L;
+ private static final long BUCKET_ONE_OBJECT_ID = 1L;
+ private static final long BUCKET_TWO_OBJECT_ID = 2L;
+ private static final long KEY_ONE_OBJECT_ID = 3L;
+ private static final long KEY_TWO_OBJECT_ID = 5L;
+ private static final long KEY_FOUR_OBJECT_ID = 6L;
+ private static final long KEY_THREE_OBJECT_ID = 8L;
+ private static final long KEY_FIVE_OBJECT_ID = 9L;
+ private static final long KEY_SIX_OBJECT_ID = 10L;
+ private static final long KEY_SEVEN_OBJECT_ID = 11L;
+
+
+ private static final long KEY_ONE_SIZE = 500L;
+ private static final long KEY_TWO_OLD_SIZE = 1025L;
+ private static final long KEY_TWO_UPDATE_SIZE = 1023L;
+ private static final long KEY_THREE_SIZE =
+ ReconConstants.MAX_FILE_SIZE_UPPER_BOUND - 100L;
+ private static final long KEY_FOUR_SIZE = 2050L;
+ private static final long KEY_FIVE_SIZE = 100L;
+ private static final long KEY_SIX_SIZE = 6000L;
+ private static final long KEY_SEVEN_SIZE = 7000L;
+
+ private TestNSSummaryTaskWithOBS() {
+ }
+
+ @BeforeAll
+ public static void setUp(@TempDir File tmpDir) throws Exception {
+ initializeNewOmMetadataManager(new File(tmpDir, "om"));
+ OzoneManagerServiceProviderImpl ozoneManagerServiceProvider =
+ getMockOzoneManagerServiceProviderWithFSO();
+ reconOMMetadataManager = getTestReconOmMetadataManager(omMetadataManager,
+ new File(tmpDir, "recon"));
+
+ ReconTestInjector reconTestInjector =
+ new ReconTestInjector.Builder(tmpDir)
+ .withReconOm(reconOMMetadataManager)
+ .withOmServiceProvider(ozoneManagerServiceProvider)
+ .withReconSqlDb()
+ .withContainerDB()
+ .build();
+ reconNamespaceSummaryManager =
+ reconTestInjector.getInstance(ReconNamespaceSummaryManager.class);
+
+ NSSummary nonExistentSummary =
+ reconNamespaceSummaryManager.getNSSummary(BUCKET_ONE_OBJECT_ID);
+ assertNull(nonExistentSummary);
+
+ populateOMDB();
+
+ nSSummaryTaskWithOBS = new NSSummaryTaskWithOBS(
+ reconNamespaceSummaryManager,
+ reconOMMetadataManager, omConfiguration);
+ }
+
+ /**
+ * Nested class for testing NSSummaryTaskWithOBS reprocess.
+ */
+ @Nested
+ public class TestReprocess {
+
+ private NSSummary nsSummaryForBucket1;
+ private NSSummary nsSummaryForBucket2;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ // write a NSSummary prior to reprocess
+ // verify it got cleaned up after.
+ NSSummary staleNSSummary = new NSSummary();
+ RDBBatchOperation rdbBatchOperation = new RDBBatchOperation();
+ reconNamespaceSummaryManager.batchStoreNSSummaries(rdbBatchOperation, -1L,
+ staleNSSummary);
+ reconNamespaceSummaryManager.commitBatchOperation(rdbBatchOperation);
+
+ // Verify commit
+ assertNotNull(reconNamespaceSummaryManager.getNSSummary(-1L));
+
+ // reinit Recon RocksDB's namespace CF.
+ reconNamespaceSummaryManager.clearNSSummaryTable();
+
+ nSSummaryTaskWithOBS.reprocessWithOBS(reconOMMetadataManager);
+ assertNull(reconNamespaceSummaryManager.getNSSummary(-1L));
+
+ nsSummaryForBucket1 =
+ reconNamespaceSummaryManager.getNSSummary(BUCKET_ONE_OBJECT_ID);
+ nsSummaryForBucket2 =
+ reconNamespaceSummaryManager.getNSSummary(BUCKET_TWO_OBJECT_ID);
+ assertNotNull(nsSummaryForBucket1);
+ assertNotNull(nsSummaryForBucket2);
+ }
+
+ @Test
+ public void testReprocessNSSummaryNull() throws IOException {
+ assertNull(reconNamespaceSummaryManager.getNSSummary(-1L));
+ }
+
+ @Test
+ public void testReprocessGetFiles() {
+ assertEquals(3, nsSummaryForBucket1.getNumOfFiles());
+ assertEquals(2, nsSummaryForBucket2.getNumOfFiles());
+
+ assertEquals(KEY_ONE_SIZE + KEY_TWO_OLD_SIZE + KEY_THREE_SIZE,
+ nsSummaryForBucket1.getSizeOfFiles());
+ assertEquals(KEY_FOUR_SIZE + KEY_FIVE_SIZE,
+ nsSummaryForBucket2.getSizeOfFiles());
+ }
+
+ @Test
+ public void testReprocessFileBucketSize() {
+ int[] fileDistBucket1 = nsSummaryForBucket1.getFileSizeBucket();
+ int[] fileDistBucket2 = nsSummaryForBucket2.getFileSizeBucket();
+ assertEquals(ReconConstants.NUM_OF_FILE_SIZE_BINS,
+ fileDistBucket1.length);
+ assertEquals(ReconConstants.NUM_OF_FILE_SIZE_BINS,
+ fileDistBucket2.length);
+
+ // Check for 1's and 0's in fileDistBucket1
+ int[] expectedIndexes1 = {0, 1, 40};
+ for (int index = 0; index < fileDistBucket1.length; index++) {
+ if (contains(expectedIndexes1, index)) {
+ assertEquals(1, fileDistBucket1[index]);
+ } else {
+ assertEquals(0, fileDistBucket1[index]);
+ }
+ }
+
+ // Check for 1's and 0's in fileDistBucket2
+ int[] expectedIndexes2 = {0, 2};
+ for (int index = 0; index < fileDistBucket2.length; index++) {
+ if (contains(expectedIndexes2, index)) {
+ assertEquals(1, fileDistBucket2[index]);
+ } else {
+ assertEquals(0, fileDistBucket2[index]);
+ }
+ }
+ }
+
+ }
+
+ /**
+ * Nested class for testing NSSummaryTaskWithLegacy process.
Review Comment:
I don't think you changed this, can you check it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "dombizita (via GitHub)" <gi...@apache.org>.
dombizita commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1506084229
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestNSSummaryEndpointWithOBS.java:
##########
@@ -0,0 +1,1133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor.ONE;
+import static org.apache.hadoop.ozone.om.helpers.QuotaUtil.getReplicatedSize;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.StorageType;
+import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.container.ContainerReplica;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.handlers.BucketHandler;
+import org.apache.hadoop.ozone.recon.api.handlers.EntityHandler;
+import org.apache.hadoop.ozone.recon.api.types.BucketObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NamespaceSummaryResponse;
+import org.apache.hadoop.ozone.recon.api.types.QuotaUsageResponse;
+import org.apache.hadoop.ozone.recon.api.types.ResponseStatus;
+import org.apache.hadoop.ozone.recon.api.types.VolumeObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.FileSizeDistributionResponse;
+import org.apache.hadoop.ozone.recon.common.CommonUtils;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.scm.ReconNodeManager;
+import org.apache.hadoop.ozone.recon.scm.ReconStorageContainerManagerFacade;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.StorageContainerServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.spi.impl.StorageContainerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.tasks.NSSummaryTaskWithOBS;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import javax.ws.rs.core.Response;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.HashSet;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.apache.hadoop.hdds.protocol.MockDatanodeDetails.randomDatanodeDetails;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProvider;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for NSSummary REST APIs with OBS.
+ * Testing is done on a simple object store model with a flat hierarchy:
+ * vol
+ * / \
+ * bucket1 bucket2
+ * | |
+ * file1 file2
+ *
+ * This tests the Rest APIs for NSSummary in the context of OBS buckets,
+ * focusing on disk usage, quota usage, and file size distribution.
+ */
+public class TestNSSummaryEndpointWithOBS {
+ @TempDir
+ private Path temporaryFolder;
+
+ private ReconOMMetadataManager reconOMMetadataManager;
+ private NSSummaryEndpoint nsSummaryEndpoint;
+ private OzoneConfiguration conf;
+ private CommonUtils commonUtils;
+
+ private static final String TEST_PATH_UTILITY =
+ "/vol1/buck1/a/b/c/d/e/file1.txt";
+ private static final String PARENT_DIR = "vol1/buck1/a/b/c/d/e";
+ private static final String[] TEST_NAMES =
+ new String[]{"vol1", "buck1", "a", "b", "c", "d", "e", "file1.txt"};
+ private static final String TEST_KEY_NAMES = "a/b/c/d/e/file1.txt";
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String VOL_TWO = "vol2";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String BUCKET_THREE = "bucket3";
+ private static final String BUCKET_FOUR = "bucket4";
+ private static final String KEY_ONE = "file1";
+ private static final String KEY_TWO = "file2";
+ private static final String KEY_THREE = "file3";
+ private static final String KEY_FOUR = "file4";
+ private static final String KEY_FIVE = "file5";
+ private static final String KEY_SIX = "file6";
+ private static final String KEY_SEVEN = "file7";
+ private static final String KEY_EIGHT = "file8";
+ private static final String KEY_NINE = "file9";
+ private static final String KEY_TEN = "file10";
+ private static final String KEY_ELEVEN = "file11";
+ private static final String MULTI_BLOCK_KEY = "file7";
+ private static final String MULTI_BLOCK_FILE = "file7";
Review Comment:
I didn't say to remove it, I'd like to know why is this names `file7` and why don't we use the constant for that? Also you didn't remove it, you renamed it to `file3`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "swagle (via GitHub)" <gi...@apache.org>.
swagle commented on PR #4245:
URL: https://github.com/apache/ozone/pull/4245#issuecomment-1843319143
> The decision to reconsider the proposed approach for creating namespace summaries for OBS keys with prefix-based filtering was made after discussions that happened in the [document](https://docs.google.com/document/d/1g2NCR-kgcSfLFQRBhZfYsMZmaZ6edlA_26Pq0-qfzT8/edit) They expressed concerns about two key aspects:
>
> 1. **Performance Impact**: We are concerned that the introduction of synthetic paths as outlined in the proposal could potentially have a negative impact on Recon performance. The additional complexity of generating and managing these synthetic paths might consume more resources and potentially slow down the system.
> 2. **Lack of Business Use Case**: There were questions the practical necessity of introducing this complexity for (OBS). In contrast to the Filesystem (FSO) layout, where directory tables exist, it was suggested that there might not be a strong business need to create synthetic paths and namespace summaries for individual OBS keys. As a result, they proposed focusing on generating bucket-level summaries for OBS keys as a more pragmatic approach.
>
> Following these discussions, we've chosen to proceed with generating disk usage statistics at the bucket level. We've decided to put the prefix-based filtering on hold for now. If it becomes necessary in the future, we'll seek input from the community. For more information, please refer to the [document](https://docs.google.com/document/d/1g2NCR-kgcSfLFQRBhZfYsMZmaZ6edlA_26Pq0-qfzT8/edit).
>
> cc: @xBis7 @devmadhuu @adoroszlai @devmadhuu @swagle
Thanks, @ArafatKhan2198 for starting this discussion and putting together ideas in a doc.
+1 for bucket-level summaries for OBS.
Are we supporting Legacy with filesystem.paths enabled?
I would lean on the side of not adding support for this scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on PR #4245:
URL: https://github.com/apache/ozone/pull/4245#issuecomment-1790566926
@ArafatKhan2198 Thanks for persisting with this.
> I'm presently in the process of creating a design document outlining how we can implement prefix-based searching for OBS (Object Store) buckets.
Is the purpose of this patch to implement prefix-based support for OBS buckets? Or is this going to be in a following PR after the design is finalized?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1418470764
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling Legacy buckets.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ OBSBucketHandler.class);
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ public EntityType determineKeyPath(String keyName)
+ throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ iterator.seek(key);
+ if (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+ if (dbKey.equals(key)) {
+ return EntityType.KEY;
+ }
+ }
+ return EntityType.UNKNOWN;
+ }
+
+ /**
+ * This method handles disk usage of direct keys.
+ *
+ * @param parentId parent OBS bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * for each direct key's DU
+ * @param listFile if listFile is enabled, append key DU as a children
+ * keys
+ * @param duData the current DU data
+ * @param normalizedPath the normalized path request
+ * @return the total DU of all direct keys
+ * @throws IOException IOE
+ */
+ public long handleDirectKeys(long parentId, boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ String normalizedPath) throws IOException {
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+ long keyDataSizeWithReplica = 0L;
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ String seekPrefix = OM_KEY_PREFIX +
+ vol +
+ OM_KEY_PREFIX +
+ bucket +
+ OM_KEY_PREFIX;
+
+ NSSummary nsSummary = getReconNamespaceSummaryManager()
+ .getNSSummary(parentId);
+ // Handle the case of an empty bucket.
+ if (nsSummary == null) {
+ return 0;
+ }
+
+ iterator.seek(seekPrefix);
+
+ while (iterator.hasNext()) {
+ // KeyName : OmKeyInfo-Object
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+
+ // Exit loop if the key doesn't match the seekPrefix.
+ if (!dbKey.startsWith(seekPrefix)) {
+ break;
+ }
+
+ OmKeyInfo keyInfo = kv.getValue();
+ if (keyInfo != null) {
+ DUResponse.DiskUsage diskUsage = new DUResponse.DiskUsage();
+ String objectName = keyInfo.getKeyName();
+ diskUsage.setSubpath(objectName);
+ diskUsage.setKey(true);
+ diskUsage.setSize(keyInfo.getDataSize());
+
+ if (withReplica) {
+ long keyDU = keyInfo.getReplicatedSize();
+ keyDataSizeWithReplica += keyDU;
+ diskUsage.setSizeWithReplica(keyDU);
+ }
+ // List all the keys for the OBS bucket if requested.
+ if (listFile) {
+ duData.add(diskUsage);
+ }
+ }
+ }
+
+ return keyDataSizeWithReplica;
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ public long calculateDUUnderObject(long parentId)
+ throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ public long getDirObjectId(String[] names) throws IOException {
Review Comment:
Done!
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling Legacy buckets.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ OBSBucketHandler.class);
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ public EntityType determineKeyPath(String keyName)
+ throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ iterator.seek(key);
+ if (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+ if (dbKey.equals(key)) {
+ return EntityType.KEY;
+ }
+ }
+ return EntityType.UNKNOWN;
+ }
+
+ /**
+ * This method handles disk usage of direct keys.
+ *
+ * @param parentId parent OBS bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * for each direct key's DU
+ * @param listFile if listFile is enabled, append key DU as a children
+ * keys
+ * @param duData the current DU data
+ * @param normalizedPath the normalized path request
+ * @return the total DU of all direct keys
+ * @throws IOException IOE
+ */
+ public long handleDirectKeys(long parentId, boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ String normalizedPath) throws IOException {
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+ long keyDataSizeWithReplica = 0L;
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ String seekPrefix = OM_KEY_PREFIX +
+ vol +
+ OM_KEY_PREFIX +
+ bucket +
+ OM_KEY_PREFIX;
+
+ NSSummary nsSummary = getReconNamespaceSummaryManager()
+ .getNSSummary(parentId);
+ // Handle the case of an empty bucket.
+ if (nsSummary == null) {
+ return 0;
+ }
+
+ iterator.seek(seekPrefix);
+
+ while (iterator.hasNext()) {
+ // KeyName : OmKeyInfo-Object
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+
+ // Exit loop if the key doesn't match the seekPrefix.
+ if (!dbKey.startsWith(seekPrefix)) {
+ break;
+ }
+
+ OmKeyInfo keyInfo = kv.getValue();
+ if (keyInfo != null) {
+ DUResponse.DiskUsage diskUsage = new DUResponse.DiskUsage();
+ String objectName = keyInfo.getKeyName();
+ diskUsage.setSubpath(objectName);
+ diskUsage.setKey(true);
+ diskUsage.setSize(keyInfo.getDataSize());
+
+ if (withReplica) {
+ long keyDU = keyInfo.getReplicatedSize();
+ keyDataSizeWithReplica += keyDU;
+ diskUsage.setSizeWithReplica(keyDU);
+ }
+ // List all the keys for the OBS bucket if requested.
+ if (listFile) {
+ duData.add(diskUsage);
+ }
+ }
+ }
+
+ return keyDataSizeWithReplica;
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ public long calculateDUUnderObject(long parentId)
+ throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ public long getDirObjectId(String[] names) throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ public long getDirObjectId(String[] names, int cutoff) throws IOException {
+ return Long.parseLong(null);
+ }
+
+
+ public OmKeyInfo getKeyInfo(String[] names) throws IOException {
Review Comment:
Done!
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling Legacy buckets.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ OBSBucketHandler.class);
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ public EntityType determineKeyPath(String keyName)
+ throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ iterator.seek(key);
+ if (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+ if (dbKey.equals(key)) {
+ return EntityType.KEY;
+ }
+ }
+ return EntityType.UNKNOWN;
+ }
+
+ /**
+ * This method handles disk usage of direct keys.
+ *
+ * @param parentId parent OBS bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * for each direct key's DU
+ * @param listFile if listFile is enabled, append key DU as a children
+ * keys
+ * @param duData the current DU data
+ * @param normalizedPath the normalized path request
+ * @return the total DU of all direct keys
+ * @throws IOException IOE
+ */
+ public long handleDirectKeys(long parentId, boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ String normalizedPath) throws IOException {
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+ long keyDataSizeWithReplica = 0L;
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ String seekPrefix = OM_KEY_PREFIX +
+ vol +
+ OM_KEY_PREFIX +
+ bucket +
+ OM_KEY_PREFIX;
+
+ NSSummary nsSummary = getReconNamespaceSummaryManager()
+ .getNSSummary(parentId);
+ // Handle the case of an empty bucket.
+ if (nsSummary == null) {
+ return 0;
+ }
+
+ iterator.seek(seekPrefix);
+
+ while (iterator.hasNext()) {
+ // KeyName : OmKeyInfo-Object
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+
+ // Exit loop if the key doesn't match the seekPrefix.
+ if (!dbKey.startsWith(seekPrefix)) {
+ break;
+ }
+
+ OmKeyInfo keyInfo = kv.getValue();
+ if (keyInfo != null) {
+ DUResponse.DiskUsage diskUsage = new DUResponse.DiskUsage();
+ String objectName = keyInfo.getKeyName();
+ diskUsage.setSubpath(objectName);
+ diskUsage.setKey(true);
+ diskUsage.setSize(keyInfo.getDataSize());
+
+ if (withReplica) {
+ long keyDU = keyInfo.getReplicatedSize();
+ keyDataSizeWithReplica += keyDU;
+ diskUsage.setSizeWithReplica(keyDU);
+ }
+ // List all the keys for the OBS bucket if requested.
+ if (listFile) {
+ duData.add(diskUsage);
+ }
+ }
+ }
+
+ return keyDataSizeWithReplica;
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ public long calculateDUUnderObject(long parentId)
Review Comment:
Done!
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling Legacy buckets.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ OBSBucketHandler.class);
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ public EntityType determineKeyPath(String keyName)
+ throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ iterator.seek(key);
+ if (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+ if (dbKey.equals(key)) {
+ return EntityType.KEY;
+ }
+ }
+ return EntityType.UNKNOWN;
+ }
+
+ /**
+ * This method handles disk usage of direct keys.
+ *
+ * @param parentId parent OBS bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * for each direct key's DU
+ * @param listFile if listFile is enabled, append key DU as a children
+ * keys
+ * @param duData the current DU data
+ * @param normalizedPath the normalized path request
+ * @return the total DU of all direct keys
+ * @throws IOException IOE
+ */
+ public long handleDirectKeys(long parentId, boolean withReplica,
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl merged PR #4245:
URL: https://github.com/apache/ozone/pull/4245
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1469207665
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling OBS buckets NameSpaceSummaries.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ @Override
+ public EntityType determineKeyPath(String keyName)
+ throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ iterator.seek(key);
+ if (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+ if (dbKey.equals(key)) {
+ return EntityType.KEY;
+ }
+ }
+ return EntityType.UNKNOWN;
+ }
+
+ /**
+ * This method handles disk usage of direct keys.
+ *
+ * @param parentId parent OBS bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * for each direct key's DU
+ * @param listFile if listFile is enabled, append key DU as a children
+ * keys
+ * @param duData the current DU data
+ * @param normalizedPath the normalized path request
+ * @return the total DU of all direct keys
+ * @throws IOException IOE
+ */
+ @Override
+ public long handleDirectKeys(long parentId, boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ String normalizedPath) throws IOException {
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+ long keyDataSizeWithReplica = 0L;
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ String seekPrefix = OM_KEY_PREFIX +
+ vol +
+ OM_KEY_PREFIX +
+ bucket +
+ OM_KEY_PREFIX;
+
+ NSSummary nsSummary = getReconNamespaceSummaryManager()
+ .getNSSummary(parentId);
+ // Handle the case of an empty bucket.
+ if (nsSummary == null) {
+ return 0;
+ }
+
+ iterator.seek(seekPrefix);
+
+ while (iterator.hasNext()) {
+ // KeyName : OmKeyInfo-Object
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+
+ // Exit loop if the key doesn't match the seekPrefix.
+ if (!dbKey.startsWith(seekPrefix)) {
+ break;
+ }
+
+ OmKeyInfo keyInfo = kv.getValue();
+ if (keyInfo != null) {
+ DUResponse.DiskUsage diskUsage = new DUResponse.DiskUsage();
+ String objectName = keyInfo.getKeyName();
+ diskUsage.setSubpath(objectName);
+ diskUsage.setKey(true);
+ diskUsage.setSize(keyInfo.getDataSize());
+
+ if (withReplica) {
+ long keyDU = keyInfo.getReplicatedSize();
+ keyDataSizeWithReplica += keyDU;
+ diskUsage.setSizeWithReplica(keyDU);
+ }
+ // List all the keys for the OBS bucket if requested.
+ if (listFile) {
+ duData.add(diskUsage);
+ }
+ }
+ }
+
+ return keyDataSizeWithReplica;
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public long calculateDUUnderObject(long parentId)
+ throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public long getDirObjectId(String[] names) throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public long getDirObjectId(String[] names, int cutoff) throws IOException {
+ return Long.parseLong(null);
Review Comment:
Thank you for the suggestion. I have implemented the recommended changes.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling OBS buckets NameSpaceSummaries.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ @Override
+ public EntityType determineKeyPath(String keyName)
+ throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
Review Comment:
Thank you for the suggestion. I have implemented the recommended changes.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling OBS buckets NameSpaceSummaries.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ @Override
+ public EntityType determineKeyPath(String keyName)
+ throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ iterator.seek(key);
+ if (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+ if (dbKey.equals(key)) {
+ return EntityType.KEY;
+ }
+ }
+ return EntityType.UNKNOWN;
+ }
+
+ /**
+ * This method handles disk usage of direct keys.
+ *
+ * @param parentId parent OBS bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * for each direct key's DU
+ * @param listFile if listFile is enabled, append key DU as a children
+ * keys
+ * @param duData the current DU data
+ * @param normalizedPath the normalized path request
+ * @return the total DU of all direct keys
+ * @throws IOException IOE
+ */
+ @Override
+ public long handleDirectKeys(long parentId, boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ String normalizedPath) throws IOException {
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+ long keyDataSizeWithReplica = 0L;
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ String seekPrefix = OM_KEY_PREFIX +
+ vol +
+ OM_KEY_PREFIX +
+ bucket +
+ OM_KEY_PREFIX;
+
+ NSSummary nsSummary = getReconNamespaceSummaryManager()
+ .getNSSummary(parentId);
+ // Handle the case of an empty bucket.
+ if (nsSummary == null) {
+ return 0;
+ }
Review Comment:
Thank you for the suggestion. I have implemented the recommended changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1469208385
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTask.java:
##########
@@ -137,17 +151,23 @@ public Pair<String, Boolean> reprocess(OMMetadataManager omMetadataManager) {
}
}
} catch (InterruptedException ex) {
- LOG.error("Error while reprocessing NSSummary " +
- "table in Recon DB. ", ex);
+ LOG.error("Error while reprocessing NSSummary table in Recon DB.", ex);
return new ImmutablePair<>(getTaskName(), false);
} catch (ExecutionException ex2) {
- LOG.error("Error while reprocessing NSSummary " +
- "table in Recon DB. ", ex2);
+ LOG.error("Error while reprocessing NSSummary table in Recon DB.", ex2);
return new ImmutablePair<>(getTaskName(), false);
} finally {
executorService.shutdown();
+
+ long endTime = System.nanoTime(); // Record end time
+ long durationInMillis = (endTime - startTime) / 1_000_000; // Convert to milliseconds
Review Comment:
Thank you for the suggestion. I have implemented the recommended changes.
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/TestNSSummaryTaskWithOBS.java:
##########
@@ -0,0 +1,556 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.RDBBatchOperation;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.NSSummaryEndpoint;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Nested;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import javax.ws.rs.core.Response;
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.*;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeDirToOm;
+import static org.mockito.Mockito.mock;
+
+public class TestNSSummaryTaskWithOBS {
+ private static ReconNamespaceSummaryManager reconNamespaceSummaryManager;
+ private static OMMetadataManager omMetadataManager;
+ private static ReconOMMetadataManager reconOMMetadataManager;
+ private static NSSummaryTaskWithOBS nSSummaryTaskWithOBS;
+ private static OzoneConfiguration omConfiguration;
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String KEY_ONE = "key1";
+ private static final String KEY_TWO = "key2";
+ private static final String KEY_THREE = "dir1/dir2/key3";
+ private static final String KEY_FOUR = "key4///////////";
+ private static final String KEY_FIVE = "//////////";
+ private static final String KEY_SIX = "key6";
+ private static final String KEY_SEVEN = "key7";
+
+ private static final String TEST_USER = "TestUser";
+
+ private static final long PARENT_OBJECT_ID_ZERO = 0L;
+ private static final long VOL_OBJECT_ID = 0L;
+ private static final long BUCKET_ONE_OBJECT_ID = 1L;
+ private static final long BUCKET_TWO_OBJECT_ID = 2L;
+ private static final long KEY_ONE_OBJECT_ID = 3L;
+ private static final long KEY_TWO_OBJECT_ID = 5L;
+ private static final long KEY_FOUR_OBJECT_ID = 6L;
+ private static final long KEY_THREE_OBJECT_ID = 8L;
+ private static final long KEY_FIVE_OBJECT_ID = 9L;
+ private static final long KEY_SIX_OBJECT_ID = 10L;
+ private static final long KEY_SEVEN_OBJECT_ID = 11L;
+
+
+ private static final long KEY_ONE_SIZE = 500L;
+ private static final long KEY_TWO_OLD_SIZE = 1025L;
+ private static final long KEY_TWO_UPDATE_SIZE = 1023L;
+ private static final long KEY_THREE_SIZE =
+ ReconConstants.MAX_FILE_SIZE_UPPER_BOUND - 100L;
+ private static final long KEY_FOUR_SIZE = 2050L;
+ private static final long KEY_FIVE_SIZE = 100L;
+ private static final long KEY_SIX_SIZE = 6000L;
+ private static final long KEY_SEVEN_SIZE = 7000L;
+
+ private static Set<Long> bucketOneAns = new HashSet<>();
+ private static Set<Long> bucketTwoAns = new HashSet<>();
+ private static Set<Long> dirOneAns = new HashSet<>();
+
+ private TestNSSummaryTaskWithOBS() {
+ }
+
+ @BeforeAll
+ public static void setUp(@TempDir File tmpDir) throws Exception {
+ initializeNewOmMetadataManager(new File(tmpDir, "om"));
+ OzoneManagerServiceProviderImpl ozoneManagerServiceProvider =
+ getMockOzoneManagerServiceProvider();
+ reconOMMetadataManager = getTestReconOmMetadataManager(omMetadataManager,
+ new File(tmpDir, "recon"));
+
+ ReconTestInjector reconTestInjector =
+ new ReconTestInjector.Builder(tmpDir)
+ .withReconOm(reconOMMetadataManager)
+ .withOmServiceProvider(ozoneManagerServiceProvider)
+ .withReconSqlDb()
+ .withContainerDB()
+ .build();
+ reconNamespaceSummaryManager =
+ reconTestInjector.getInstance(ReconNamespaceSummaryManager.class);
+
+ NSSummary nonExistentSummary =
+ reconNamespaceSummaryManager.getNSSummary(BUCKET_ONE_OBJECT_ID);
+ Assertions.assertNull(nonExistentSummary);
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on PR #4245:
URL: https://github.com/apache/ozone/pull/4245#issuecomment-1792492760
@GeorgeJahad @xBis7 @devmadhuu @ArafatKhan2198 I have looked over discussion and current state of feature, few of suggestions for going ahead,
1. Current implementation for FSO, UI shows all files, directories and same gets queried. This does not work if have large number of directory Or files at same level, Pie-Chart will be mess, and data query itself will fail. So need support query with pagination at different level of directory and files.
@ArafatKhan2198 Please raise separate bug to handle this.
2. We need refactor code to have
-- Legacy and OBS : as same handler as both case are OBS only
-- Legacy with filesystem enable: this needs separate handler
-- FSO: need separate handler
Legacy with FSO is not supported, this support can be provided with this PR.
and for OBS, as plain query of data (not a FSO simulation for OBS)
3. Need another JIRA to support "OBS support with FSO simulation"
-- This needs enable using flag at backend
-- frontend: can provide flag to query flat keys OR FSO simulated keys
-- Backend: Need have pre-processing as separate NSSummary table to have data in hierarchy -- a separate NSSummary handler and query handler
Detailed design of this needs to be done.
@ArafatKhan2198 IMO, we can break as above and handle this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1506348802
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestNSSummaryEndpointWithOBS.java:
##########
@@ -0,0 +1,1133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor.ONE;
+import static org.apache.hadoop.ozone.om.helpers.QuotaUtil.getReplicatedSize;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.StorageType;
+import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.container.ContainerReplica;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.handlers.BucketHandler;
+import org.apache.hadoop.ozone.recon.api.handlers.EntityHandler;
+import org.apache.hadoop.ozone.recon.api.types.BucketObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NamespaceSummaryResponse;
+import org.apache.hadoop.ozone.recon.api.types.QuotaUsageResponse;
+import org.apache.hadoop.ozone.recon.api.types.ResponseStatus;
+import org.apache.hadoop.ozone.recon.api.types.VolumeObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.FileSizeDistributionResponse;
+import org.apache.hadoop.ozone.recon.common.CommonUtils;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.scm.ReconNodeManager;
+import org.apache.hadoop.ozone.recon.scm.ReconStorageContainerManagerFacade;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.StorageContainerServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.spi.impl.StorageContainerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.tasks.NSSummaryTaskWithOBS;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import javax.ws.rs.core.Response;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.HashSet;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.apache.hadoop.hdds.protocol.MockDatanodeDetails.randomDatanodeDetails;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProvider;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for NSSummary REST APIs with OBS.
+ * Testing is done on a simple object store model with a flat hierarchy:
+ * vol
+ * / \
+ * bucket1 bucket2
+ * | |
+ * file1 file2
+ *
+ * This tests the Rest APIs for NSSummary in the context of OBS buckets,
+ * focusing on disk usage, quota usage, and file size distribution.
+ */
+public class TestNSSummaryEndpointWithOBS {
+ @TempDir
+ private Path temporaryFolder;
+
+ private ReconOMMetadataManager reconOMMetadataManager;
+ private NSSummaryEndpoint nsSummaryEndpoint;
+ private OzoneConfiguration conf;
+ private CommonUtils commonUtils;
+
+ private static final String TEST_PATH_UTILITY =
+ "/vol1/buck1/a/b/c/d/e/file1.txt";
+ private static final String PARENT_DIR = "vol1/buck1/a/b/c/d/e";
+ private static final String[] TEST_NAMES =
+ new String[]{"vol1", "buck1", "a", "b", "c", "d", "e", "file1.txt"};
+ private static final String TEST_KEY_NAMES = "a/b/c/d/e/file1.txt";
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String VOL_TWO = "vol2";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String BUCKET_THREE = "bucket3";
+ private static final String BUCKET_FOUR = "bucket4";
+ private static final String KEY_ONE = "file1";
+ private static final String KEY_TWO = "file2";
+ private static final String KEY_THREE = "file3";
+ private static final String KEY_FOUR = "file4";
+ private static final String KEY_FIVE = "file5";
+ private static final String KEY_SIX = "file6";
+ private static final String KEY_SEVEN = "file7";
+ private static final String KEY_EIGHT = "file8";
+ private static final String KEY_NINE = "file9";
+ private static final String KEY_TEN = "file10";
+ private static final String KEY_ELEVEN = "file11";
+ private static final String MULTI_BLOCK_KEY = "file7";
+ private static final String MULTI_BLOCK_FILE = "file7";
Review Comment:
Could you check again.
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/TestNSSummaryTaskWithOBS.java:
##########
@@ -0,0 +1,548 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.utils.db.RDBBatchOperation;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Nested;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Set;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProviderWithFSO;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+/**
+ * Unit test for NSSummaryTaskWithOBS.
+ */
+public final class TestNSSummaryTaskWithOBS implements Serializable {
+ private static ReconNamespaceSummaryManager reconNamespaceSummaryManager;
+ private static OMMetadataManager omMetadataManager;
+ private static ReconOMMetadataManager reconOMMetadataManager;
+ private static NSSummaryTaskWithOBS nSSummaryTaskWithOBS;
+ private static OzoneConfiguration omConfiguration;
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String KEY_ONE = "key1";
+ private static final String KEY_TWO = "key2";
+ private static final String KEY_THREE = "dir1/dir2/key3";
+ private static final String KEY_FOUR = "key4///////////";
+ private static final String KEY_FIVE = "//////////";
+ private static final String KEY_SIX = "key6";
+ private static final String KEY_SEVEN = "key7";
+
+ private static final String TEST_USER = "TestUser";
+
+ private static final long PARENT_OBJECT_ID_ZERO = 0L;
+ private static final long VOL_OBJECT_ID = 0L;
+ private static final long BUCKET_ONE_OBJECT_ID = 1L;
+ private static final long BUCKET_TWO_OBJECT_ID = 2L;
+ private static final long KEY_ONE_OBJECT_ID = 3L;
+ private static final long KEY_TWO_OBJECT_ID = 5L;
+ private static final long KEY_FOUR_OBJECT_ID = 6L;
+ private static final long KEY_THREE_OBJECT_ID = 8L;
+ private static final long KEY_FIVE_OBJECT_ID = 9L;
+ private static final long KEY_SIX_OBJECT_ID = 10L;
+ private static final long KEY_SEVEN_OBJECT_ID = 11L;
+
+
+ private static final long KEY_ONE_SIZE = 500L;
+ private static final long KEY_TWO_OLD_SIZE = 1025L;
+ private static final long KEY_TWO_UPDATE_SIZE = 1023L;
+ private static final long KEY_THREE_SIZE =
+ ReconConstants.MAX_FILE_SIZE_UPPER_BOUND - 100L;
+ private static final long KEY_FOUR_SIZE = 2050L;
+ private static final long KEY_FIVE_SIZE = 100L;
+ private static final long KEY_SIX_SIZE = 6000L;
+ private static final long KEY_SEVEN_SIZE = 7000L;
+
+ private TestNSSummaryTaskWithOBS() {
+ }
+
+ @BeforeAll
+ public static void setUp(@TempDir File tmpDir) throws Exception {
+ initializeNewOmMetadataManager(new File(tmpDir, "om"));
+ OzoneManagerServiceProviderImpl ozoneManagerServiceProvider =
+ getMockOzoneManagerServiceProviderWithFSO();
+ reconOMMetadataManager = getTestReconOmMetadataManager(omMetadataManager,
+ new File(tmpDir, "recon"));
+
+ ReconTestInjector reconTestInjector =
+ new ReconTestInjector.Builder(tmpDir)
+ .withReconOm(reconOMMetadataManager)
+ .withOmServiceProvider(ozoneManagerServiceProvider)
+ .withReconSqlDb()
+ .withContainerDB()
+ .build();
+ reconNamespaceSummaryManager =
+ reconTestInjector.getInstance(ReconNamespaceSummaryManager.class);
+
+ NSSummary nonExistentSummary =
+ reconNamespaceSummaryManager.getNSSummary(BUCKET_ONE_OBJECT_ID);
+ assertNull(nonExistentSummary);
+
+ populateOMDB();
+
+ nSSummaryTaskWithOBS = new NSSummaryTaskWithOBS(
+ reconNamespaceSummaryManager,
+ reconOMMetadataManager, omConfiguration);
+ }
+
+ /**
+ * Nested class for testing NSSummaryTaskWithOBS reprocess.
+ */
+ @Nested
+ public class TestReprocess {
+
+ private NSSummary nsSummaryForBucket1;
+ private NSSummary nsSummaryForBucket2;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ // write a NSSummary prior to reprocess
+ // verify it got cleaned up after.
+ NSSummary staleNSSummary = new NSSummary();
+ RDBBatchOperation rdbBatchOperation = new RDBBatchOperation();
+ reconNamespaceSummaryManager.batchStoreNSSummaries(rdbBatchOperation, -1L,
+ staleNSSummary);
+ reconNamespaceSummaryManager.commitBatchOperation(rdbBatchOperation);
+
+ // Verify commit
+ assertNotNull(reconNamespaceSummaryManager.getNSSummary(-1L));
+
+ // reinit Recon RocksDB's namespace CF.
+ reconNamespaceSummaryManager.clearNSSummaryTable();
+
+ nSSummaryTaskWithOBS.reprocessWithOBS(reconOMMetadataManager);
+ assertNull(reconNamespaceSummaryManager.getNSSummary(-1L));
+
+ nsSummaryForBucket1 =
+ reconNamespaceSummaryManager.getNSSummary(BUCKET_ONE_OBJECT_ID);
+ nsSummaryForBucket2 =
+ reconNamespaceSummaryManager.getNSSummary(BUCKET_TWO_OBJECT_ID);
+ assertNotNull(nsSummaryForBucket1);
+ assertNotNull(nsSummaryForBucket2);
+ }
+
+ @Test
+ public void testReprocessNSSummaryNull() throws IOException {
+ assertNull(reconNamespaceSummaryManager.getNSSummary(-1L));
+ }
+
+ @Test
+ public void testReprocessGetFiles() {
+ assertEquals(3, nsSummaryForBucket1.getNumOfFiles());
+ assertEquals(2, nsSummaryForBucket2.getNumOfFiles());
+
+ assertEquals(KEY_ONE_SIZE + KEY_TWO_OLD_SIZE + KEY_THREE_SIZE,
+ nsSummaryForBucket1.getSizeOfFiles());
+ assertEquals(KEY_FOUR_SIZE + KEY_FIVE_SIZE,
+ nsSummaryForBucket2.getSizeOfFiles());
+ }
+
+ @Test
+ public void testReprocessFileBucketSize() {
+ int[] fileDistBucket1 = nsSummaryForBucket1.getFileSizeBucket();
+ int[] fileDistBucket2 = nsSummaryForBucket2.getFileSizeBucket();
+ assertEquals(ReconConstants.NUM_OF_FILE_SIZE_BINS,
+ fileDistBucket1.length);
+ assertEquals(ReconConstants.NUM_OF_FILE_SIZE_BINS,
+ fileDistBucket2.length);
+
+ // Check for 1's and 0's in fileDistBucket1
+ int[] expectedIndexes1 = {0, 1, 40};
+ for (int index = 0; index < fileDistBucket1.length; index++) {
+ if (contains(expectedIndexes1, index)) {
+ assertEquals(1, fileDistBucket1[index]);
+ } else {
+ assertEquals(0, fileDistBucket1[index]);
+ }
+ }
+
+ // Check for 1's and 0's in fileDistBucket2
+ int[] expectedIndexes2 = {0, 2};
+ for (int index = 0; index < fileDistBucket2.length; index++) {
+ if (contains(expectedIndexes2, index)) {
+ assertEquals(1, fileDistBucket2[index]);
+ } else {
+ assertEquals(0, fileDistBucket2[index]);
+ }
+ }
+ }
+
+ }
+
+ /**
+ * Nested class for testing NSSummaryTaskWithLegacy process.
Review Comment:
I have made the changes for this!
Not sure why its not showing it as outdated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1506349306
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestNSSummaryEndpointWithOBS.java:
##########
@@ -0,0 +1,1133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor.ONE;
+import static org.apache.hadoop.ozone.om.helpers.QuotaUtil.getReplicatedSize;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.StorageType;
+import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.container.ContainerReplica;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.handlers.BucketHandler;
+import org.apache.hadoop.ozone.recon.api.handlers.EntityHandler;
+import org.apache.hadoop.ozone.recon.api.types.BucketObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NamespaceSummaryResponse;
+import org.apache.hadoop.ozone.recon.api.types.QuotaUsageResponse;
+import org.apache.hadoop.ozone.recon.api.types.ResponseStatus;
+import org.apache.hadoop.ozone.recon.api.types.VolumeObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.FileSizeDistributionResponse;
+import org.apache.hadoop.ozone.recon.common.CommonUtils;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.scm.ReconNodeManager;
+import org.apache.hadoop.ozone.recon.scm.ReconStorageContainerManagerFacade;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.StorageContainerServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.spi.impl.StorageContainerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.tasks.NSSummaryTaskWithOBS;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import javax.ws.rs.core.Response;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.HashSet;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.apache.hadoop.hdds.protocol.MockDatanodeDetails.randomDatanodeDetails;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProvider;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for NSSummary REST APIs with OBS.
+ * Testing is done on a simple object store model with a flat hierarchy:
+ * vol
+ * / \
+ * bucket1 bucket2
+ * | |
+ * file1 file2
+ *
+ * This tests the Rest APIs for NSSummary in the context of OBS buckets,
+ * focusing on disk usage, quota usage, and file size distribution.
+ */
+public class TestNSSummaryEndpointWithOBS {
+ @TempDir
+ private Path temporaryFolder;
+
+ private ReconOMMetadataManager reconOMMetadataManager;
+ private NSSummaryEndpoint nsSummaryEndpoint;
+ private OzoneConfiguration conf;
+ private CommonUtils commonUtils;
+
+ private static final String TEST_PATH_UTILITY =
+ "/vol1/buck1/a/b/c/d/e/file1.txt";
+ private static final String PARENT_DIR = "vol1/buck1/a/b/c/d/e";
+ private static final String[] TEST_NAMES =
+ new String[]{"vol1", "buck1", "a", "b", "c", "d", "e", "file1.txt"};
+ private static final String TEST_KEY_NAMES = "a/b/c/d/e/file1.txt";
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String VOL_TWO = "vol2";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String BUCKET_THREE = "bucket3";
+ private static final String BUCKET_FOUR = "bucket4";
+ private static final String KEY_ONE = "file1";
+ private static final String KEY_TWO = "file2";
+ private static final String KEY_THREE = "file3";
+ private static final String KEY_FOUR = "file4";
+ private static final String KEY_FIVE = "file5";
+ private static final String KEY_SIX = "file6";
+ private static final String KEY_SEVEN = "file7";
+ private static final String KEY_EIGHT = "file8";
+ private static final String KEY_NINE = "file9";
+ private static final String KEY_TEN = "file10";
+ private static final String KEY_ELEVEN = "file11";
+ private static final String MULTI_BLOCK_KEY = "file7";
+ private static final String MULTI_BLOCK_FILE = "file7";
+
+ private static final String FILE_ONE = "file1";
+ private static final String FILE_TWO = "file2";
+ private static final String FILE_THREE = "file3";
+ private static final String FILE_FOUR = "file4";
+ private static final String FILE_FIVE = "file5";
+ private static final String FILE_SIX = "file6";
+ private static final String FILE_SEVEN = "file7";
+ private static final String FILE_EIGHT = "file8";
+ private static final String FILE_NINE = "file9";
+ private static final String FILE_TEN = "file10";
+ private static final String FILE_ELEVEN = "file11";
+ // objects IDs
+ private static final long PARENT_OBJECT_ID_ZERO = 0L;
+ private static final long VOL_OBJECT_ID = 0L;
+ private static final long BUCKET_ONE_OBJECT_ID = 1L;
+ private static final long BUCKET_TWO_OBJECT_ID = 2L;
+ private static final long KEY_ONE_OBJECT_ID = 3L;
+ private static final long KEY_TWO_OBJECT_ID = 5L;
+ private static final long KEY_FOUR_OBJECT_ID = 6L;
+ private static final long KEY_THREE_OBJECT_ID = 8L;
+ private static final long KEY_FIVE_OBJECT_ID = 9L;
+ private static final long KEY_SIX_OBJECT_ID = 10L;
+ private static final long MULTI_BLOCK_KEY_OBJECT_ID = 13L;
+ private static final long KEY_SEVEN_OBJECT_ID = 13L;
+ private static final long VOL_TWO_OBJECT_ID = 14L;
+ private static final long BUCKET_THREE_OBJECT_ID = 15L;
+ private static final long BUCKET_FOUR_OBJECT_ID = 16L;
+ private static final long KEY_EIGHT_OBJECT_ID = 17L;
+ private static final long KEY_NINE_OBJECT_ID = 19L;
+ private static final long KEY_TEN_OBJECT_ID = 20L;
+ private static final long KEY_ELEVEN_OBJECT_ID = 21L;
+
+ // container IDs
+ private static final long CONTAINER_ONE_ID = 1L;
+ private static final long CONTAINER_TWO_ID = 2L;
+ private static final long CONTAINER_THREE_ID = 3L;
+ private static final long CONTAINER_FOUR_ID = 4L;
+ private static final long CONTAINER_FIVE_ID = 5L;
+ private static final long CONTAINER_SIX_ID = 6L;
+
+ // replication factors
+ private static final int CONTAINER_ONE_REPLICA_COUNT = 3;
+ private static final int CONTAINER_TWO_REPLICA_COUNT = 2;
+ private static final int CONTAINER_THREE_REPLICA_COUNT = 4;
+ private static final int CONTAINER_FOUR_REPLICA_COUNT = 5;
+ private static final int CONTAINER_FIVE_REPLICA_COUNT = 2;
+ private static final int CONTAINER_SIX_REPLICA_COUNT = 3;
+
+ // block lengths
+ private static final long BLOCK_ONE_LENGTH = 1000L;
+ private static final long BLOCK_TWO_LENGTH = 2000L;
+ private static final long BLOCK_THREE_LENGTH = 3000L;
+ private static final long BLOCK_FOUR_LENGTH = 4000L;
+ private static final long BLOCK_FIVE_LENGTH = 5000L;
+ private static final long BLOCK_SIX_LENGTH = 6000L;
+
+ // data size in bytes
+ private static final long KEY_ONE_SIZE = 500L; // bin 0
+ private static final long KEY_TWO_SIZE = OzoneConsts.KB + 1; // bin 1
+ private static final long KEY_THREE_SIZE = 4 * OzoneConsts.KB + 1; // bin 3
+ private static final long KEY_FOUR_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_FIVE_SIZE = 100L; // bin 0
+ private static final long KEY_SIX_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_SEVEN_SIZE = 4 * OzoneConsts.KB + 1;
+ private static final long KEY_EIGHT_SIZE = OzoneConsts.KB + 1; // bin 1
+ private static final long KEY_NINE_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_TEN_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_ELEVEN_SIZE = OzoneConsts.KB + 1; // bin 1
+
+ private static final long FILE1_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_ONE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE2_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_TWO_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE3_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_THREE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE4_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_FOUR_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE5_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_FIVE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE6_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_SIX_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));;
+ private static final long FILE7_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_SEVEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));;
+ private static final long FILE8_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_EIGHT_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE9_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_NINE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE10_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_TEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE11_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_ELEVEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+
+ private static final long MULTI_BLOCK_KEY_SIZE_WITH_REPLICA
+ = FILE7_SIZE_WITH_REPLICA;
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_ROOT
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA
+ + FILE4_SIZE_WITH_REPLICA
+ + FILE5_SIZE_WITH_REPLICA
+ + FILE8_SIZE_WITH_REPLICA
+ + FILE9_SIZE_WITH_REPLICA
+ + FILE10_SIZE_WITH_REPLICA
+ + FILE11_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_VOL
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA
+ + FILE4_SIZE_WITH_REPLICA
+ + FILE5_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_BUCKET1
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_KEY
+ = FILE4_SIZE_WITH_REPLICA;
+
+ // quota in bytes
+ private static final long ROOT_QUOTA = 2 * (2 * OzoneConsts.MB);
+ private static final long VOL_QUOTA = 2 * OzoneConsts.MB;
+ private static final long VOL_TWO_QUOTA = 2 * OzoneConsts.MB;
+ private static final long BUCKET_ONE_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_TWO_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_THREE_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_FOUR_QUOTA = OzoneConsts.MB;
+
+ // mock client's path requests
+ private static final String TEST_USER = "TestUser";
+ private static final String ROOT_PATH = "/";
+ private static final String VOL_PATH = "/vol";
+ private static final String VOL_TWO_PATH = "/vol2";
+ private static final String BUCKET_ONE_PATH = "/vol/bucket1";
+ private static final String BUCKET_TWO_PATH = "/vol/bucket2";
+ private static final String KEY_PATH = "/vol/bucket2/file4";
+ private static final String MULTI_BLOCK_KEY_PATH = "/vol/bucket1/file7";
+ private static final String INVALID_PATH = "/vol/path/not/found";
Review Comment:
Yes I missed to commit this one!
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1505885153
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTaskWithOBS.java:
##########
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.WithParentObjectId;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+
+import static org.apache.hadoop.ozone.om.OmMetadataManagerImpl.KEY_TABLE;
+
+
+/**
+ * Class for handling OBS specific tasks.
+ */
+public class NSSummaryTaskWithOBS extends NSSummaryTaskDbEventHandler {
+
+ private static final BucketLayout BUCKET_LAYOUT = BucketLayout.OBJECT_STORE;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(NSSummaryTaskWithOBS.class);
+
+
+ public NSSummaryTaskWithOBS(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager reconOMMetadataManager,
+ OzoneConfiguration ozoneConfiguration) {
+ super(reconNamespaceSummaryManager,
+ reconOMMetadataManager, ozoneConfiguration);
+ }
+
+
+ public boolean reprocessWithOBS(OMMetadataManager omMetadataManager) {
+ Map<Long, NSSummary> nsSummaryMap = new HashMap<>();
+
+ try {
+ Table<String, OmKeyInfo> keyTable =
+ omMetadataManager.getKeyTable(BUCKET_LAYOUT);
+
+ try (TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ keyTableIter = keyTable.iterator()) {
+
+ while (keyTableIter.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = keyTableIter.next();
+ OmKeyInfo keyInfo = kv.getValue();
+
+ // KeyTable entries belong to both Legacy and OBS buckets.
+ // Check bucket layout and if it's anything other than OBS,
+ // continue to the next iteration.
+ String volumeName = keyInfo.getVolumeName();
+ String bucketName = keyInfo.getBucketName();
+ String bucketDBKey = omMetadataManager
+ .getBucketKey(volumeName, bucketName);
+ // Get bucket info from bucket table
+ OmBucketInfo omBucketInfo = omMetadataManager
+ .getBucketTable().getSkipCache(bucketDBKey);
+
+ if (omBucketInfo.getBucketLayout() != BUCKET_LAYOUT) {
+ continue;
+ }
+
+ setKeyParentID(keyInfo);
+
+ handlePutKeyEvent(keyInfo, nsSummaryMap);
+ if (!checkAndCallFlushToDB(nsSummaryMap)) {
+ return false;
+ }
+ }
+ }
+ } catch (IOException ioEx) {
+ LOG.error("Unable to reprocess Namespace Summary data in Recon DB. ",
+ ioEx);
+ return false;
+ }
+
+ // flush and commit left out entries at end
+ if (!flushAndCommitNSToDB(nsSummaryMap)) {
+ return false;
+ }
+ LOG.info("Completed a reprocess run of NSSummaryTaskWithOBS");
+ return true;
+ }
+
+ public boolean processWithOBS(OMUpdateEventBatch events) {
+ Iterator<OMDBUpdateEvent> eventIterator = events.getIterator();
+ Map<Long, NSSummary> nsSummaryMap = new HashMap<>();
+
+ while (eventIterator.hasNext()) {
+ OMDBUpdateEvent<String, ? extends WithParentObjectId> omdbUpdateEvent =
+ eventIterator.next();
+ OMDBUpdateEvent.OMDBUpdateAction action = omdbUpdateEvent.getAction();
+
+ // We only process updates on OM's KeyTable
+ String table = omdbUpdateEvent.getTable();
+ boolean updateOnKeyTable = table.equals(KEY_TABLE);
+ if (!updateOnKeyTable) {
+ continue;
+ }
+
+ String updatedKey = omdbUpdateEvent.getKey();
+
+ try {
+ OMDBUpdateEvent<String, ?> keyTableUpdateEvent = omdbUpdateEvent;
+ Object value = keyTableUpdateEvent.getValue();
+ Object oldValue = keyTableUpdateEvent.getOldValue();
+ if (!(value instanceof OmKeyInfo)) {
+ LOG.warn("Unexpected value type {} for key {}. Skipping processing.",
+ value.getClass().getName(), updatedKey);
Review Comment:
Hope there is no NPE thrown. value.getClass.getName
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1418470923
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling Legacy buckets.
Review Comment:
Done!
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/EntityHandler.java:
##########
@@ -46,6 +48,10 @@
*/
public abstract class EntityHandler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1418479573
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTask.java:
##########
@@ -101,6 +105,11 @@ public Pair<String, Boolean> process(OMUpdateEventBatch events) {
} else {
LOG.error("processWithFSO failed.");
}
+ if (success) {
Review Comment:
Yes I have removed the dependency on each-other.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] GeorgeJahad commented on pull request #4245: HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets.
Posted by "GeorgeJahad (via GitHub)" <gi...@apache.org>.
GeorgeJahad commented on PR #4245:
URL: https://github.com/apache/ozone/pull/4245#issuecomment-1430505298
Hey @ArafatKhan2198 @smengcl @dombizita:
Last year, when @xBis7 and I implemented this for legacy buckets, we were going to do object store buckets as well.
We didn't because @avijayanhwx (the engineer who originally created this epic), told us it would be good to support "object store prefixes".
It looks like this PR still doesn't support them. Instead it just assigns all object store keys to the corresponding bucket. I'm fine with that, but just want to be sure that everyone understands it wasn't the initial intention.
FYI: this is how @avijayanhwx defined the prefixes:
*"What I mean as object store prefix is not dissimilar to a directory in legacy FS buckets, but there is added complexity in Recon to track them since they will not be present in the OM metadata. For example, in a pure OS bucket, the rocksdb entry would be /vol1/bucket1/application1/instance1/file1. The prefixes ( /vol1/bucket1/application1 & /vol1/bucket1/application1/instance1) are not created as keys in RocksDB. We wanted to explore how to capture them and calculate summaries at prefix levels. Feel free to think about the need to do it as well as how we could design it. Thanks for taking this up."*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] umamaheswararao commented on pull request #4245: HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets.
Posted by "umamaheswararao (via GitHub)" <gi...@apache.org>.
umamaheswararao commented on PR #4245:
URL: https://github.com/apache/ozone/pull/4245#issuecomment-1419428068
@GeorgeJahad
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on pull request #4245: HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets.
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on PR #4245:
URL: https://github.com/apache/ozone/pull/4245#issuecomment-1430582708
I agree with @GeorgeJahad, there are so many unit tests for all the other bucket types. Shouldn't there be unit tests for OBS as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1469208094
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling OBS buckets NameSpaceSummaries.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ @Override
+ public EntityType determineKeyPath(String keyName)
+ throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ iterator.seek(key);
+ if (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+ if (dbKey.equals(key)) {
+ return EntityType.KEY;
+ }
+ }
+ return EntityType.UNKNOWN;
+ }
+
+ /**
+ * This method handles disk usage of direct keys.
+ *
+ * @param parentId parent OBS bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * for each direct key's DU
+ * @param listFile if listFile is enabled, append key DU as a children
+ * keys
+ * @param duData the current DU data
+ * @param normalizedPath the normalized path request
+ * @return the total DU of all direct keys
+ * @throws IOException IOE
+ */
+ @Override
+ public long handleDirectKeys(long parentId, boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ String normalizedPath) throws IOException {
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+ long keyDataSizeWithReplica = 0L;
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ String seekPrefix = OM_KEY_PREFIX +
+ vol +
+ OM_KEY_PREFIX +
+ bucket +
+ OM_KEY_PREFIX;
+
+ NSSummary nsSummary = getReconNamespaceSummaryManager()
+ .getNSSummary(parentId);
+ // Handle the case of an empty bucket.
+ if (nsSummary == null) {
+ return 0;
+ }
+
+ iterator.seek(seekPrefix);
+
+ while (iterator.hasNext()) {
+ // KeyName : OmKeyInfo-Object
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+
+ // Exit loop if the key doesn't match the seekPrefix.
+ if (!dbKey.startsWith(seekPrefix)) {
+ break;
+ }
+
+ OmKeyInfo keyInfo = kv.getValue();
+ if (keyInfo != null) {
+ DUResponse.DiskUsage diskUsage = new DUResponse.DiskUsage();
+ String objectName = keyInfo.getKeyName();
+ diskUsage.setSubpath(objectName);
+ diskUsage.setKey(true);
+ diskUsage.setSize(keyInfo.getDataSize());
+
+ if (withReplica) {
+ long keyDU = keyInfo.getReplicatedSize();
+ keyDataSizeWithReplica += keyDU;
+ diskUsage.setSizeWithReplica(keyDU);
+ }
+ // List all the keys for the OBS bucket if requested.
+ if (listFile) {
+ duData.add(diskUsage);
+ }
+ }
+ }
+
+ return keyDataSizeWithReplica;
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public long calculateDUUnderObject(long parentId)
+ throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public long getDirObjectId(String[] names) throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public long getDirObjectId(String[] names, int cutoff) throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Returns the keyInfo object from the KEY table.
+ * @return OmKeyInfo
+ */
+ @Override
+ public OmKeyInfo getKeyInfo(String[] names) throws IOException {
+ String ozoneKey = OM_KEY_PREFIX;
+ ozoneKey += String.join(OM_KEY_PREFIX, names);
+
+ OmKeyInfo keyInfo = getKeyTable().getSkipCache(ozoneKey);
+ return keyInfo;
Review Comment:
Thank you for the suggestion. I have implemented the recommended changes.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.recon.api.handlers;
+
+
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Class for handling OBS buckets NameSpaceSummaries.
+ */
+public class OBSBucketHandler extends BucketHandler {
+
+ private final String vol;
+ private final String bucket;
+ private final OmBucketInfo omBucketInfo;
+
+ public OBSBucketHandler(
+ ReconNamespaceSummaryManager reconNamespaceSummaryManager,
+ ReconOMMetadataManager omMetadataManager,
+ OzoneStorageContainerManager reconSCM,
+ OmBucketInfo bucketInfo) {
+ super(reconNamespaceSummaryManager, omMetadataManager,
+ reconSCM);
+ this.omBucketInfo = bucketInfo;
+ this.vol = omBucketInfo.getVolumeName();
+ this.bucket = omBucketInfo.getBucketName();
+ }
+
+ /**
+ * Helper function to check if a path is a key, or invalid.
+ *
+ * @param keyName key name
+ * @return KEY, or UNKNOWN
+ * @throws IOException
+ */
+ @Override
+ public EntityType determineKeyPath(String keyName)
+ throws IOException {
+ String key = OM_KEY_PREFIX + vol +
+ OM_KEY_PREFIX + bucket +
+ OM_KEY_PREFIX + keyName;
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ iterator.seek(key);
+ if (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+ if (dbKey.equals(key)) {
+ return EntityType.KEY;
+ }
+ }
+ return EntityType.UNKNOWN;
+ }
+
+ /**
+ * This method handles disk usage of direct keys.
+ *
+ * @param parentId parent OBS bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * for each direct key's DU
+ * @param listFile if listFile is enabled, append key DU as a children
+ * keys
+ * @param duData the current DU data
+ * @param normalizedPath the normalized path request
+ * @return the total DU of all direct keys
+ * @throws IOException IOE
+ */
+ @Override
+ public long handleDirectKeys(long parentId, boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ String normalizedPath) throws IOException {
+
+ Table<String, OmKeyInfo> keyTable = getKeyTable();
+ long keyDataSizeWithReplica = 0L;
+
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
+ iterator = keyTable.iterator();
+
+ String seekPrefix = OM_KEY_PREFIX +
+ vol +
+ OM_KEY_PREFIX +
+ bucket +
+ OM_KEY_PREFIX;
+
+ NSSummary nsSummary = getReconNamespaceSummaryManager()
+ .getNSSummary(parentId);
+ // Handle the case of an empty bucket.
+ if (nsSummary == null) {
+ return 0;
+ }
+
+ iterator.seek(seekPrefix);
+
+ while (iterator.hasNext()) {
+ // KeyName : OmKeyInfo-Object
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+
+ // Exit loop if the key doesn't match the seekPrefix.
+ if (!dbKey.startsWith(seekPrefix)) {
+ break;
+ }
+
+ OmKeyInfo keyInfo = kv.getValue();
+ if (keyInfo != null) {
+ DUResponse.DiskUsage diskUsage = new DUResponse.DiskUsage();
+ String objectName = keyInfo.getKeyName();
+ diskUsage.setSubpath(objectName);
+ diskUsage.setKey(true);
+ diskUsage.setSize(keyInfo.getDataSize());
+
+ if (withReplica) {
+ long keyDU = keyInfo.getReplicatedSize();
+ keyDataSizeWithReplica += keyDU;
+ diskUsage.setSizeWithReplica(keyDU);
+ }
+ // List all the keys for the OBS bucket if requested.
+ if (listFile) {
+ duData.add(diskUsage);
+ }
+ }
+ }
+
+ return keyDataSizeWithReplica;
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public long calculateDUUnderObject(long parentId)
+ throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public long getDirObjectId(String[] names) throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public long getDirObjectId(String[] names, int cutoff) throws IOException {
+ return Long.parseLong(null);
+ }
+
+ /**
+ * Returns the keyInfo object from the KEY table.
+ * @return OmKeyInfo
+ */
+ @Override
+ public OmKeyInfo getKeyInfo(String[] names) throws IOException {
+ String ozoneKey = OM_KEY_PREFIX;
+ ozoneKey += String.join(OM_KEY_PREFIX, names);
+
+ OmKeyInfo keyInfo = getKeyTable().getSkipCache(ozoneKey);
+ return keyInfo;
+ }
+
+ /**
+ * Object stores do not support directories, hence return null.
+ *
+ * @return null
+ */
+ @Override
+ public OmDirectoryInfo getDirInfo(String[] names) throws IOException {
+ return null;
+ }
+
+ public Table<String, OmKeyInfo> getKeyTable() {
+ Table keyTable =
+ getOmMetadataManager().getKeyTable(getBucketLayout());
+ return keyTable;
Review Comment:
Thank you for the suggestion. I have implemented the recommended changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on PR #4245:
URL: https://github.com/apache/ozone/pull/4245#issuecomment-1790152640
> Hey @ArafatKhan2198 @smengcl @dombizita:
>
> Last year, when @xBis7 and I implemented this for legacy buckets, we were going to do object store buckets as well.
>
> We didn't because @avijayanhwx (the engineer who originally created this epic), told us it would be good to support "object store prefixes", (and we didn't have time to do the extra work.)
>
> It looks like this PR still doesn't support them. Instead it just assigns all object store keys to the corresponding bucket. I'm fine with that, but just want to be sure that everyone understands it wasn't the initial intention.
>
> FYI: this is how @avijayanhwx defined the prefixes, (from an email he sent me):
>
> _"What I mean as object store prefix is not dissimilar to a directory in legacy FS buckets, but there is added complexity in Recon to track them since they will not be present in the OM metadata. For example, in a pure OS bucket, the rocksdb entry would be /vol1/bucket1/application1/instance1/file1. The prefixes ( /vol1/bucket1/application1 & /vol1/bucket1/application1/instance1) are not created as keys in RocksDB. We wanted to explore how to capture them and calculate summaries at prefix levels. Feel free to think about the need to do it as well as how we could design it. Thanks for taking this up."_
@GeorgeJahad @xBis7
That's a great suggestion! thanks for explaining it so well. I'm presently in the process of creating a design document outlining how we can implement prefix-based searching for OBS (Object Store) buckets. You can monitor the progress of this initiative on Jira using the following link: https://issues.apache.org/jira/browse/HDDS-9535.
Our primary objective at the moment is to ensure that our customers don't encounter any confusion when using Recon with various bucket types. Implementing prefix-based filtering will likely be a crucial improvement, and I'll be working on it soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1506351430
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestNSSummaryEndpointWithOBS.java:
##########
@@ -0,0 +1,1133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor.ONE;
+import static org.apache.hadoop.ozone.om.helpers.QuotaUtil.getReplicatedSize;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.StorageType;
+import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.container.ContainerReplica;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.handlers.BucketHandler;
+import org.apache.hadoop.ozone.recon.api.handlers.EntityHandler;
+import org.apache.hadoop.ozone.recon.api.types.BucketObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NamespaceSummaryResponse;
+import org.apache.hadoop.ozone.recon.api.types.QuotaUsageResponse;
+import org.apache.hadoop.ozone.recon.api.types.ResponseStatus;
+import org.apache.hadoop.ozone.recon.api.types.VolumeObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.FileSizeDistributionResponse;
+import org.apache.hadoop.ozone.recon.common.CommonUtils;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.scm.ReconNodeManager;
+import org.apache.hadoop.ozone.recon.scm.ReconStorageContainerManagerFacade;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.StorageContainerServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.spi.impl.StorageContainerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.tasks.NSSummaryTaskWithOBS;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import javax.ws.rs.core.Response;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.HashSet;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.apache.hadoop.hdds.protocol.MockDatanodeDetails.randomDatanodeDetails;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProvider;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for NSSummary REST APIs with OBS.
+ * Testing is done on a simple object store model with a flat hierarchy:
+ * vol
+ * / \
+ * bucket1 bucket2
+ * | |
+ * file1 file2
+ *
+ * This tests the Rest APIs for NSSummary in the context of OBS buckets,
+ * focusing on disk usage, quota usage, and file size distribution.
+ */
+public class TestNSSummaryEndpointWithOBS {
+ @TempDir
+ private Path temporaryFolder;
+
+ private ReconOMMetadataManager reconOMMetadataManager;
+ private NSSummaryEndpoint nsSummaryEndpoint;
+ private OzoneConfiguration conf;
+ private CommonUtils commonUtils;
+
+ private static final String TEST_PATH_UTILITY =
+ "/vol1/buck1/a/b/c/d/e/file1.txt";
+ private static final String PARENT_DIR = "vol1/buck1/a/b/c/d/e";
+ private static final String[] TEST_NAMES =
+ new String[]{"vol1", "buck1", "a", "b", "c", "d", "e", "file1.txt"};
+ private static final String TEST_KEY_NAMES = "a/b/c/d/e/file1.txt";
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String VOL_TWO = "vol2";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String BUCKET_THREE = "bucket3";
+ private static final String BUCKET_FOUR = "bucket4";
+ private static final String KEY_ONE = "file1";
+ private static final String KEY_TWO = "file2";
+ private static final String KEY_THREE = "file3";
+ private static final String KEY_FOUR = "file4";
+ private static final String KEY_FIVE = "file5";
+ private static final String KEY_SIX = "file6";
+ private static final String KEY_SEVEN = "file7";
+ private static final String KEY_EIGHT = "file8";
+ private static final String KEY_NINE = "file9";
+ private static final String KEY_TEN = "file10";
+ private static final String KEY_ELEVEN = "file11";
+ private static final String MULTI_BLOCK_KEY = "file7";
+ private static final String MULTI_BLOCK_FILE = "file7";
+
+ private static final String FILE_ONE = "file1";
+ private static final String FILE_TWO = "file2";
+ private static final String FILE_THREE = "file3";
+ private static final String FILE_FOUR = "file4";
+ private static final String FILE_FIVE = "file5";
+ private static final String FILE_SIX = "file6";
+ private static final String FILE_SEVEN = "file7";
+ private static final String FILE_EIGHT = "file8";
+ private static final String FILE_NINE = "file9";
+ private static final String FILE_TEN = "file10";
+ private static final String FILE_ELEVEN = "file11";
Review Comment:
I've made the removals, but there was a bit of confusion. Now, for OBS buckets, we don't have directories, only files. The term 'Key' encompasses the entire path, while 'file' refers to the file itself. Therefore, it's more appropriate to use 'files' for this class, as OBS employs a flat hierarchy without deeper paths.
Could you please check again.
@dombizita
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1504175474
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/TestNSSummaryTaskWithOBS.java:
##########
@@ -0,0 +1,548 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.utils.db.RDBBatchOperation;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Nested;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Set;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProviderWithFSO;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+/**
+ * Unit test for NSSummaryTaskWithOBS.
+ */
+public final class TestNSSummaryTaskWithOBS implements Serializable {
+ private static ReconNamespaceSummaryManager reconNamespaceSummaryManager;
+ private static OMMetadataManager omMetadataManager;
+ private static ReconOMMetadataManager reconOMMetadataManager;
+ private static NSSummaryTaskWithOBS nSSummaryTaskWithOBS;
+ private static OzoneConfiguration omConfiguration;
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String KEY_ONE = "key1";
+ private static final String KEY_TWO = "key2";
+ private static final String KEY_THREE = "dir1/dir2/key3";
+ private static final String KEY_FOUR = "key4///////////";
+ private static final String KEY_FIVE = "//////////";
+ private static final String KEY_SIX = "key6";
+ private static final String KEY_SEVEN = "key7";
+
+ private static final String TEST_USER = "TestUser";
+
+ private static final long PARENT_OBJECT_ID_ZERO = 0L;
+ private static final long VOL_OBJECT_ID = 0L;
+ private static final long BUCKET_ONE_OBJECT_ID = 1L;
+ private static final long BUCKET_TWO_OBJECT_ID = 2L;
+ private static final long KEY_ONE_OBJECT_ID = 3L;
+ private static final long KEY_TWO_OBJECT_ID = 5L;
+ private static final long KEY_FOUR_OBJECT_ID = 6L;
+ private static final long KEY_THREE_OBJECT_ID = 8L;
+ private static final long KEY_FIVE_OBJECT_ID = 9L;
+ private static final long KEY_SIX_OBJECT_ID = 10L;
+ private static final long KEY_SEVEN_OBJECT_ID = 11L;
+
+
+ private static final long KEY_ONE_SIZE = 500L;
+ private static final long KEY_TWO_OLD_SIZE = 1025L;
+ private static final long KEY_TWO_UPDATE_SIZE = 1023L;
+ private static final long KEY_THREE_SIZE =
+ ReconConstants.MAX_FILE_SIZE_UPPER_BOUND - 100L;
+ private static final long KEY_FOUR_SIZE = 2050L;
+ private static final long KEY_FIVE_SIZE = 100L;
+ private static final long KEY_SIX_SIZE = 6000L;
+ private static final long KEY_SEVEN_SIZE = 7000L;
+
+ private TestNSSummaryTaskWithOBS() {
+ }
+
+ @BeforeAll
+ public static void setUp(@TempDir File tmpDir) throws Exception {
+ initializeNewOmMetadataManager(new File(tmpDir, "om"));
+ OzoneManagerServiceProviderImpl ozoneManagerServiceProvider =
+ getMockOzoneManagerServiceProviderWithFSO();
+ reconOMMetadataManager = getTestReconOmMetadataManager(omMetadataManager,
+ new File(tmpDir, "recon"));
+
+ ReconTestInjector reconTestInjector =
+ new ReconTestInjector.Builder(tmpDir)
+ .withReconOm(reconOMMetadataManager)
+ .withOmServiceProvider(ozoneManagerServiceProvider)
+ .withReconSqlDb()
+ .withContainerDB()
+ .build();
+ reconNamespaceSummaryManager =
+ reconTestInjector.getInstance(ReconNamespaceSummaryManager.class);
+
+ NSSummary nonExistentSummary =
+ reconNamespaceSummaryManager.getNSSummary(BUCKET_ONE_OBJECT_ID);
+ assertNull(nonExistentSummary);
+
+ populateOMDB();
+
+ nSSummaryTaskWithOBS = new NSSummaryTaskWithOBS(
+ reconNamespaceSummaryManager,
+ reconOMMetadataManager, omConfiguration);
+ }
+
+ /**
+ * Nested class for testing NSSummaryTaskWithOBS reprocess.
+ */
+ @Nested
+ public class TestReprocess {
+
+ private NSSummary nsSummaryForBucket1;
+ private NSSummary nsSummaryForBucket2;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ // write a NSSummary prior to reprocess
+ // verify it got cleaned up after.
+ NSSummary staleNSSummary = new NSSummary();
+ RDBBatchOperation rdbBatchOperation = new RDBBatchOperation();
+ reconNamespaceSummaryManager.batchStoreNSSummaries(rdbBatchOperation, -1L,
+ staleNSSummary);
+ reconNamespaceSummaryManager.commitBatchOperation(rdbBatchOperation);
+
+ // Verify commit
+ assertNotNull(reconNamespaceSummaryManager.getNSSummary(-1L));
+
+ // reinit Recon RocksDB's namespace CF.
+ reconNamespaceSummaryManager.clearNSSummaryTable();
+
+ nSSummaryTaskWithOBS.reprocessWithOBS(reconOMMetadataManager);
+ assertNull(reconNamespaceSummaryManager.getNSSummary(-1L));
+
+ nsSummaryForBucket1 =
+ reconNamespaceSummaryManager.getNSSummary(BUCKET_ONE_OBJECT_ID);
+ nsSummaryForBucket2 =
+ reconNamespaceSummaryManager.getNSSummary(BUCKET_TWO_OBJECT_ID);
+ assertNotNull(nsSummaryForBucket1);
+ assertNotNull(nsSummaryForBucket2);
+ }
+
+ @Test
+ public void testReprocessNSSummaryNull() throws IOException {
+ assertNull(reconNamespaceSummaryManager.getNSSummary(-1L));
+ }
+
+ @Test
+ public void testReprocessGetFiles() {
+ assertEquals(3, nsSummaryForBucket1.getNumOfFiles());
+ assertEquals(2, nsSummaryForBucket2.getNumOfFiles());
+
+ assertEquals(KEY_ONE_SIZE + KEY_TWO_OLD_SIZE + KEY_THREE_SIZE,
+ nsSummaryForBucket1.getSizeOfFiles());
+ assertEquals(KEY_FOUR_SIZE + KEY_FIVE_SIZE,
+ nsSummaryForBucket2.getSizeOfFiles());
+ }
+
+ @Test
+ public void testReprocessFileBucketSize() {
+ int[] fileDistBucket1 = nsSummaryForBucket1.getFileSizeBucket();
+ int[] fileDistBucket2 = nsSummaryForBucket2.getFileSizeBucket();
+ assertEquals(ReconConstants.NUM_OF_FILE_SIZE_BINS,
+ fileDistBucket1.length);
+ assertEquals(ReconConstants.NUM_OF_FILE_SIZE_BINS,
+ fileDistBucket2.length);
+
+ // Check for 1's and 0's in fileDistBucket1
+ int[] expectedIndexes1 = {0, 1, 40};
+ for (int index = 0; index < fileDistBucket1.length; index++) {
+ if (contains(expectedIndexes1, index)) {
+ assertEquals(1, fileDistBucket1[index]);
+ } else {
+ assertEquals(0, fileDistBucket1[index]);
+ }
+ }
+
+ // Check for 1's and 0's in fileDistBucket2
+ int[] expectedIndexes2 = {0, 2};
+ for (int index = 0; index < fileDistBucket2.length; index++) {
+ if (contains(expectedIndexes2, index)) {
+ assertEquals(1, fileDistBucket2[index]);
+ } else {
+ assertEquals(0, fileDistBucket2[index]);
+ }
+ }
+ }
+
+ }
+
+ /**
+ * Nested class for testing NSSummaryTaskWithLegacy process.
Review Comment:
Changed thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "dombizita (via GitHub)" <gi...@apache.org>.
dombizita commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1506082369
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestNSSummaryEndpointWithOBS.java:
##########
@@ -0,0 +1,1133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor.ONE;
+import static org.apache.hadoop.ozone.om.helpers.QuotaUtil.getReplicatedSize;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.StorageType;
+import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.container.ContainerReplica;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.handlers.BucketHandler;
+import org.apache.hadoop.ozone.recon.api.handlers.EntityHandler;
+import org.apache.hadoop.ozone.recon.api.types.BucketObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NamespaceSummaryResponse;
+import org.apache.hadoop.ozone.recon.api.types.QuotaUsageResponse;
+import org.apache.hadoop.ozone.recon.api.types.ResponseStatus;
+import org.apache.hadoop.ozone.recon.api.types.VolumeObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.FileSizeDistributionResponse;
+import org.apache.hadoop.ozone.recon.common.CommonUtils;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.scm.ReconNodeManager;
+import org.apache.hadoop.ozone.recon.scm.ReconStorageContainerManagerFacade;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.StorageContainerServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.spi.impl.StorageContainerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.tasks.NSSummaryTaskWithOBS;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import javax.ws.rs.core.Response;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.HashSet;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.apache.hadoop.hdds.protocol.MockDatanodeDetails.randomDatanodeDetails;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProvider;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for NSSummary REST APIs with OBS.
+ * Testing is done on a simple object store model with a flat hierarchy:
+ * vol
+ * / \
+ * bucket1 bucket2
+ * | |
+ * file1 file2
+ *
+ * This tests the Rest APIs for NSSummary in the context of OBS buckets,
+ * focusing on disk usage, quota usage, and file size distribution.
+ */
+public class TestNSSummaryEndpointWithOBS {
+ @TempDir
+ private Path temporaryFolder;
+
+ private ReconOMMetadataManager reconOMMetadataManager;
+ private NSSummaryEndpoint nsSummaryEndpoint;
+ private OzoneConfiguration conf;
+ private CommonUtils commonUtils;
+
+ private static final String TEST_PATH_UTILITY =
+ "/vol1/buck1/a/b/c/d/e/file1.txt";
+ private static final String PARENT_DIR = "vol1/buck1/a/b/c/d/e";
+ private static final String[] TEST_NAMES =
+ new String[]{"vol1", "buck1", "a", "b", "c", "d", "e", "file1.txt"};
+ private static final String TEST_KEY_NAMES = "a/b/c/d/e/file1.txt";
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String VOL_TWO = "vol2";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String BUCKET_THREE = "bucket3";
+ private static final String BUCKET_FOUR = "bucket4";
+ private static final String KEY_ONE = "file1";
+ private static final String KEY_TWO = "file2";
+ private static final String KEY_THREE = "file3";
+ private static final String KEY_FOUR = "file4";
+ private static final String KEY_FIVE = "file5";
+ private static final String KEY_SIX = "file6";
+ private static final String KEY_SEVEN = "file7";
+ private static final String KEY_EIGHT = "file8";
+ private static final String KEY_NINE = "file9";
+ private static final String KEY_TEN = "file10";
+ private static final String KEY_ELEVEN = "file11";
+ private static final String MULTI_BLOCK_KEY = "file7";
+ private static final String MULTI_BLOCK_FILE = "file7";
+
+ private static final String FILE_ONE = "file1";
+ private static final String FILE_TWO = "file2";
+ private static final String FILE_THREE = "file3";
+ private static final String FILE_FOUR = "file4";
+ private static final String FILE_FIVE = "file5";
+ private static final String FILE_SIX = "file6";
+ private static final String FILE_SEVEN = "file7";
+ private static final String FILE_EIGHT = "file8";
+ private static final String FILE_NINE = "file9";
+ private static final String FILE_TEN = "file10";
+ private static final String FILE_ELEVEN = "file11";
+ // objects IDs
+ private static final long PARENT_OBJECT_ID_ZERO = 0L;
+ private static final long VOL_OBJECT_ID = 0L;
+ private static final long BUCKET_ONE_OBJECT_ID = 1L;
+ private static final long BUCKET_TWO_OBJECT_ID = 2L;
+ private static final long KEY_ONE_OBJECT_ID = 3L;
+ private static final long KEY_TWO_OBJECT_ID = 5L;
+ private static final long KEY_FOUR_OBJECT_ID = 6L;
+ private static final long KEY_THREE_OBJECT_ID = 8L;
+ private static final long KEY_FIVE_OBJECT_ID = 9L;
+ private static final long KEY_SIX_OBJECT_ID = 10L;
+ private static final long MULTI_BLOCK_KEY_OBJECT_ID = 13L;
+ private static final long KEY_SEVEN_OBJECT_ID = 13L;
+ private static final long VOL_TWO_OBJECT_ID = 14L;
+ private static final long BUCKET_THREE_OBJECT_ID = 15L;
+ private static final long BUCKET_FOUR_OBJECT_ID = 16L;
+ private static final long KEY_EIGHT_OBJECT_ID = 17L;
+ private static final long KEY_NINE_OBJECT_ID = 19L;
+ private static final long KEY_TEN_OBJECT_ID = 20L;
+ private static final long KEY_ELEVEN_OBJECT_ID = 21L;
+
+ // container IDs
+ private static final long CONTAINER_ONE_ID = 1L;
+ private static final long CONTAINER_TWO_ID = 2L;
+ private static final long CONTAINER_THREE_ID = 3L;
+ private static final long CONTAINER_FOUR_ID = 4L;
+ private static final long CONTAINER_FIVE_ID = 5L;
+ private static final long CONTAINER_SIX_ID = 6L;
+
+ // replication factors
+ private static final int CONTAINER_ONE_REPLICA_COUNT = 3;
+ private static final int CONTAINER_TWO_REPLICA_COUNT = 2;
+ private static final int CONTAINER_THREE_REPLICA_COUNT = 4;
+ private static final int CONTAINER_FOUR_REPLICA_COUNT = 5;
+ private static final int CONTAINER_FIVE_REPLICA_COUNT = 2;
+ private static final int CONTAINER_SIX_REPLICA_COUNT = 3;
+
+ // block lengths
+ private static final long BLOCK_ONE_LENGTH = 1000L;
+ private static final long BLOCK_TWO_LENGTH = 2000L;
+ private static final long BLOCK_THREE_LENGTH = 3000L;
+ private static final long BLOCK_FOUR_LENGTH = 4000L;
+ private static final long BLOCK_FIVE_LENGTH = 5000L;
+ private static final long BLOCK_SIX_LENGTH = 6000L;
+
+ // data size in bytes
+ private static final long KEY_ONE_SIZE = 500L; // bin 0
+ private static final long KEY_TWO_SIZE = OzoneConsts.KB + 1; // bin 1
+ private static final long KEY_THREE_SIZE = 4 * OzoneConsts.KB + 1; // bin 3
+ private static final long KEY_FOUR_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_FIVE_SIZE = 100L; // bin 0
+ private static final long KEY_SIX_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_SEVEN_SIZE = 4 * OzoneConsts.KB + 1;
+ private static final long KEY_EIGHT_SIZE = OzoneConsts.KB + 1; // bin 1
+ private static final long KEY_NINE_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_TEN_SIZE = 2 * OzoneConsts.KB + 1; // bin 2
+ private static final long KEY_ELEVEN_SIZE = OzoneConsts.KB + 1; // bin 1
+
+ private static final long FILE1_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_ONE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE2_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_TWO_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE3_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_THREE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE4_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_FOUR_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE5_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_FIVE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE6_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_SIX_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));;
+ private static final long FILE7_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_SEVEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));;
+ private static final long FILE8_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_EIGHT_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE9_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_NINE_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE10_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_TEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+ private static final long FILE11_SIZE_WITH_REPLICA =
+ getReplicatedSize(KEY_ELEVEN_SIZE,
+ StandaloneReplicationConfig.getInstance(ONE));
+
+ private static final long MULTI_BLOCK_KEY_SIZE_WITH_REPLICA
+ = FILE7_SIZE_WITH_REPLICA;
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_ROOT
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA
+ + FILE4_SIZE_WITH_REPLICA
+ + FILE5_SIZE_WITH_REPLICA
+ + FILE8_SIZE_WITH_REPLICA
+ + FILE9_SIZE_WITH_REPLICA
+ + FILE10_SIZE_WITH_REPLICA
+ + FILE11_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_VOL
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA
+ + FILE4_SIZE_WITH_REPLICA
+ + FILE5_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_BUCKET1
+ = FILE1_SIZE_WITH_REPLICA
+ + FILE2_SIZE_WITH_REPLICA
+ + FILE3_SIZE_WITH_REPLICA;
+
+ private static final long
+ MULTI_BLOCK_TOTAL_SIZE_WITH_REPLICA_UNDER_KEY
+ = FILE4_SIZE_WITH_REPLICA;
+
+ // quota in bytes
+ private static final long ROOT_QUOTA = 2 * (2 * OzoneConsts.MB);
+ private static final long VOL_QUOTA = 2 * OzoneConsts.MB;
+ private static final long VOL_TWO_QUOTA = 2 * OzoneConsts.MB;
+ private static final long BUCKET_ONE_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_TWO_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_THREE_QUOTA = OzoneConsts.MB;
+ private static final long BUCKET_FOUR_QUOTA = OzoneConsts.MB;
+
+ // mock client's path requests
+ private static final String TEST_USER = "TestUser";
+ private static final String ROOT_PATH = "/";
+ private static final String VOL_PATH = "/vol";
+ private static final String VOL_TWO_PATH = "/vol2";
+ private static final String BUCKET_ONE_PATH = "/vol/bucket1";
+ private static final String BUCKET_TWO_PATH = "/vol/bucket2";
+ private static final String KEY_PATH = "/vol/bucket2/file4";
+ private static final String MULTI_BLOCK_KEY_PATH = "/vol/bucket1/file7";
+ private static final String INVALID_PATH = "/vol/path/not/found";
Review Comment:
I don't see that you did the changes, can you check it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-7810. Support namespace summaries (du, dist & counts) for OBJECT_STORE buckets. [ozone]
Posted by "dombizita (via GitHub)" <gi...@apache.org>.
dombizita commented on code in PR #4245:
URL: https://github.com/apache/ozone/pull/4245#discussion_r1506086195
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestNSSummaryEndpointWithOBS.java:
##########
@@ -0,0 +1,1133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.api;
+
+import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor.ONE;
+import static org.apache.hadoop.ozone.om.helpers.QuotaUtil.getReplicatedSize;
+
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.StorageType;
+import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos;
+import org.apache.hadoop.hdds.scm.container.ContainerID;
+import org.apache.hadoop.hdds.scm.container.ContainerManager;
+import org.apache.hadoop.hdds.scm.container.ContainerNotFoundException;
+import org.apache.hadoop.hdds.scm.container.ContainerReplica;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMNodeStat;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
+import org.apache.hadoop.ozone.recon.ReconConstants;
+import org.apache.hadoop.ozone.recon.ReconTestInjector;
+import org.apache.hadoop.ozone.recon.api.handlers.BucketHandler;
+import org.apache.hadoop.ozone.recon.api.handlers.EntityHandler;
+import org.apache.hadoop.ozone.recon.api.types.BucketObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.DUResponse;
+import org.apache.hadoop.ozone.recon.api.types.EntityType;
+import org.apache.hadoop.ozone.recon.api.types.NamespaceSummaryResponse;
+import org.apache.hadoop.ozone.recon.api.types.QuotaUsageResponse;
+import org.apache.hadoop.ozone.recon.api.types.ResponseStatus;
+import org.apache.hadoop.ozone.recon.api.types.VolumeObjectDBInfo;
+import org.apache.hadoop.ozone.recon.api.types.FileSizeDistributionResponse;
+import org.apache.hadoop.ozone.recon.common.CommonUtils;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.scm.ReconNodeManager;
+import org.apache.hadoop.ozone.recon.scm.ReconStorageContainerManagerFacade;
+import org.apache.hadoop.ozone.recon.spi.ReconNamespaceSummaryManager;
+import org.apache.hadoop.ozone.recon.spi.StorageContainerServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.spi.impl.StorageContainerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.tasks.NSSummaryTaskWithOBS;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import javax.ws.rs.core.Response;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Set;
+import java.util.HashSet;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.apache.hadoop.hdds.protocol.MockDatanodeDetails.randomDatanodeDetails;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.writeKeyToOm;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getMockOzoneManagerServiceProvider;
+import static org.apache.hadoop.ozone.recon.OMMetadataManagerTestUtils.getTestReconOmMetadataManager;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Test for NSSummary REST APIs with OBS.
+ * Testing is done on a simple object store model with a flat hierarchy:
+ * vol
+ * / \
+ * bucket1 bucket2
+ * | |
+ * file1 file2
+ *
+ * This tests the Rest APIs for NSSummary in the context of OBS buckets,
+ * focusing on disk usage, quota usage, and file size distribution.
+ */
+public class TestNSSummaryEndpointWithOBS {
+ @TempDir
+ private Path temporaryFolder;
+
+ private ReconOMMetadataManager reconOMMetadataManager;
+ private NSSummaryEndpoint nsSummaryEndpoint;
+ private OzoneConfiguration conf;
+ private CommonUtils commonUtils;
+
+ private static final String TEST_PATH_UTILITY =
+ "/vol1/buck1/a/b/c/d/e/file1.txt";
+ private static final String PARENT_DIR = "vol1/buck1/a/b/c/d/e";
+ private static final String[] TEST_NAMES =
+ new String[]{"vol1", "buck1", "a", "b", "c", "d", "e", "file1.txt"};
+ private static final String TEST_KEY_NAMES = "a/b/c/d/e/file1.txt";
+
+ // Object names
+ private static final String VOL = "vol";
+ private static final String VOL_TWO = "vol2";
+ private static final String BUCKET_ONE = "bucket1";
+ private static final String BUCKET_TWO = "bucket2";
+ private static final String BUCKET_THREE = "bucket3";
+ private static final String BUCKET_FOUR = "bucket4";
+ private static final String KEY_ONE = "file1";
+ private static final String KEY_TWO = "file2";
+ private static final String KEY_THREE = "file3";
+ private static final String KEY_FOUR = "file4";
+ private static final String KEY_FIVE = "file5";
+ private static final String KEY_SIX = "file6";
+ private static final String KEY_SEVEN = "file7";
+ private static final String KEY_EIGHT = "file8";
+ private static final String KEY_NINE = "file9";
+ private static final String KEY_TEN = "file10";
+ private static final String KEY_ELEVEN = "file11";
+ private static final String MULTI_BLOCK_KEY = "file7";
+ private static final String MULTI_BLOCK_FILE = "file7";
+
+ private static final String FILE_ONE = "file1";
+ private static final String FILE_TWO = "file2";
+ private static final String FILE_THREE = "file3";
+ private static final String FILE_FOUR = "file4";
+ private static final String FILE_FIVE = "file5";
+ private static final String FILE_SIX = "file6";
+ private static final String FILE_SEVEN = "file7";
+ private static final String FILE_EIGHT = "file8";
+ private static final String FILE_NINE = "file9";
+ private static final String FILE_TEN = "file10";
+ private static final String FILE_ELEVEN = "file11";
Review Comment:
I don't think this is going with a convention, I really don't why do we have these, you are also not using it in a way that would explain them. Don't you think we could remove them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org