You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ArafatKhan2198 (via GitHub)" <gi...@apache.org> on 2023/10/31 05:58:21 UTC
[PR] HDDS-9534 Support namespace summaries (du, dist & counts) for LEGACY buckets with file system disabled [ozone]
ArafatKhan2198 opened a new pull request, #5517:
URL: https://github.com/apache/ozone/pull/5517
## What changes were proposed in this pull request?
Currently, only FSO (File System Optimised) buckets and Legacy buckets (which support file structure hierarchy) are supported by namespace summaries. Therefore, this change task aims to provide support for both object store and filesystem structures as well for Legacy buckets in particular.
Please describe your PR in detail:
- Modification of LegacyBucketHandler to enable the handling of direct keys for OBS (Object Storage) layout Legacy buckets.
- Segregation of parent ID allocation in NSSummaryTaskWithLegacy to accommodate different scenarios.
- Introduction of a new field in NSSummary to validate whether the entity originates from an object store layout bucket or a file system layout bucket.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-9534
## How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9534 Support namespace summaries (du, dist & counts) for LEGACY buckets with file system disabled [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5517:
URL: https://github.com/apache/ozone/pull/5517#discussion_r1538687423
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/EntityHandler.java:
##########
@@ -256,7 +280,52 @@ public static String[] parseRequestPath(String path) {
return names;
}
- private static String normalizePath(String path) {
+ /**
+ * Splits an object store path into volume, bucket, and key name components.
+ *
+ * This method parses a path of the format "/volumeName/bucketName/keyName",
+ * including paths with additional '/' characters within the key name. It's
+ * designed for object store paths where the first three '/' characters
+ * separate the root, volume and bucket names from the key name.
+ *
+ * @param path The object store path to parse, starting with a slash.
+ * @return A String array with three elements: volume name, bucket name, and
+ * key name, or {null} if the path format is invalid.
+ */
+ public static String[] parseObjectStorePath(String path) {
Review Comment:
Done!
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java:
##########
@@ -743,6 +743,46 @@ public static String normalizeKey(String keyName,
return keyName;
}
+ /**
+ * Normalizes a given path up to the bucket level.
+ *
+ * This method takes a path as input and normalises uptil the bucket level.
+ * It handles empty, removes leading slashes, and splits the path into
+ * segments. It then extracts the volume and bucket names, forming a
+ * normalized path with a single slash. Finally, any remaining segments are
+ * joined as the key name, returning the complete standardized path.
+ *
+ * @param path The path string to be normalized.
+ * @return The normalized path string.
+ */
+ public static String normalizePathUptilBucket(String path) {
Review Comment:
No this normalisation method will not change the objectName structure. I have added a unit test to verify it in the latest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9534. Support namespace summaries (du, dist & counts) for LEGACY buckets with file system disabled [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5517:
URL: https://github.com/apache/ozone/pull/5517#discussion_r1544577830
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java:
##########
@@ -743,6 +743,46 @@ public static String normalizeKey(String keyName,
return keyName;
}
+ /**
+ * Normalizes a given path up to the bucket level.
+ *
+ * This method takes a path as input and normalises uptil the bucket level.
+ * It handles empty, removes leading slashes, and splits the path into
+ * segments. It then extracts the volume and bucket names, forming a
+ * normalized path with a single slash. Finally, any remaining segments are
+ * joined as the key name, returning the complete standardized path.
+ *
+ * @param path The path string to be normalized.
+ * @return The normalized path string.
+ */
+ public static String normalizePathUptilBucket(String path) {
Review Comment:
Done!
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java:
##########
@@ -743,6 +743,46 @@ public static String normalizeKey(String keyName,
return keyName;
}
+ /**
+ * Normalizes a given path up to the bucket level.
+ *
+ * This method takes a path as input and normalises uptil the bucket level.
+ * It handles empty, removes leading slashes, and splits the path into
+ * segments. It then extracts the volume and bucket names, forming a
+ * normalized path with a single slash. Finally, any remaining segments are
+ * joined as the key name, returning the complete standardized path.
+ *
+ * @param path The path string to be normalized.
+ * @return The normalized path string.
+ */
+ public static String normalizePathUptilBucket(String path) {
+ if (path == null || path.isEmpty()) {
+ return "/"; // Handle empty path
Review Comment:
Done!
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java:
##########
@@ -743,6 +743,46 @@ public static String normalizeKey(String keyName,
return keyName;
}
+ /**
+ * Normalizes a given path up to the bucket level.
+ *
+ * This method takes a path as input and normalises uptil the bucket level.
+ * It handles empty, removes leading slashes, and splits the path into
+ * segments. It then extracts the volume and bucket names, forming a
+ * normalized path with a single slash. Finally, any remaining segments are
+ * joined as the key name, returning the complete standardized path.
+ *
+ * @param path The path string to be normalized.
+ * @return The normalized path string.
+ */
+ public static String normalizePathUptilBucket(String path) {
+ if (path == null || path.isEmpty()) {
+ return "/"; // Handle empty path
+ }
+
+ // Remove leading slashes
+ path = path.replaceAll("^/*", "");
+
+ String[] segments = path.split("/", -1);
+
+ String volumeName = segments[0];
+ String bucketName = segments.length > 1 ? segments[1] : "";
+
+ // Combine volume and bucket.
+ StringBuilder normalizedPath = new StringBuilder(volumeName);
+ if (!bucketName.isEmpty()) {
+ normalizedPath.append("/").append(bucketName);
Review Comment:
Done!
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java:
##########
@@ -743,6 +743,46 @@ public static String normalizeKey(String keyName,
return keyName;
}
+ /**
+ * Normalizes a given path up to the bucket level.
+ *
+ * This method takes a path as input and normalises uptil the bucket level.
+ * It handles empty, removes leading slashes, and splits the path into
+ * segments. It then extracts the volume and bucket names, forming a
+ * normalized path with a single slash. Finally, any remaining segments are
+ * joined as the key name, returning the complete standardized path.
+ *
+ * @param path The path string to be normalized.
+ * @return The normalized path string.
+ */
+ public static String normalizePathUptilBucket(String path) {
+ if (path == null || path.isEmpty()) {
+ return "/"; // Handle empty path
+ }
+
+ // Remove leading slashes
+ path = path.replaceAll("^/*", "");
+
+ String[] segments = path.split("/", -1);
+
+ String volumeName = segments[0];
+ String bucketName = segments.length > 1 ? segments[1] : "";
+
+ // Combine volume and bucket.
+ StringBuilder normalizedPath = new StringBuilder(volumeName);
+ if (!bucketName.isEmpty()) {
+ normalizedPath.append("/").append(bucketName);
+ }
+
+ // Add remaining segments as the key
+ if (segments.length > 2) {
+ normalizedPath.append("/").append(
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9534 Support namespace summaries (du, dist & counts) for LEGACY buckets with file system disabled [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on code in PR #5517:
URL: https://github.com/apache/ozone/pull/5517#discussion_r1381661011
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTaskWithLegacy.java:
##########
@@ -206,6 +131,103 @@ public boolean processWithLegacy(OMUpdateEventBatch events) {
return true;
}
+ private void processWithFileSystemLayout(OmKeyInfo updatedKeyInfo,
+ OmKeyInfo oldKeyInfo,
+ OMDBUpdateEvent.OMDBUpdateAction action,
+ Map<Long, NSSummary> nsSummaryMap)
+ throws IOException {
+ setParentDirectoryId(updatedKeyInfo);
+
+ if (!updatedKeyInfo.getKeyName().endsWith(OM_KEY_PREFIX)) {
+ switch (action) {
+ case PUT:
+ handlePutKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+
+ case DELETE:
+ handleDeleteKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+
+ case UPDATE:
+ if (oldKeyInfo != null) {
+ setParentDirectoryId(oldKeyInfo);
+ handleDeleteKeyEvent(oldKeyInfo, nsSummaryMap);
+ } else {
+ LOG.warn("Update event does not have the old keyInfo for {}.",
+ updatedKeyInfo.getKeyName());
+ }
+ handlePutKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+ }
+ } else {
+ OmDirectoryInfo updatedDirectoryInfo = new OmDirectoryInfo.Builder()
+ .setName(updatedKeyInfo.getKeyName())
+ .setObjectID(updatedKeyInfo.getObjectID())
+ .setParentObjectID(updatedKeyInfo.getParentObjectID())
+ .build();
+
+ OmDirectoryInfo oldDirectoryInfo = null;
+
+ if (oldKeyInfo != null) {
+ oldDirectoryInfo =
+ new OmDirectoryInfo.Builder()
+ .setName(oldKeyInfo.getKeyName())
+ .setObjectID(oldKeyInfo.getObjectID())
+ .setParentObjectID(oldKeyInfo.getParentObjectID())
+ .build();
+ }
+
+ switch (action) {
+ case PUT:
+ handlePutDirEvent(updatedDirectoryInfo, nsSummaryMap);
+ break;
+
+ case DELETE:
+ handleDeleteDirEvent(updatedDirectoryInfo, nsSummaryMap);
+ break;
+
+ case UPDATE:
+ if (oldDirectoryInfo != null) {
+ handleDeleteDirEvent(oldDirectoryInfo, nsSummaryMap);
+ } else {
+ LOG.warn("Update event does not have the old dirInfo for {}.",
+ updatedKeyInfo.getKeyName());
+ }
+ handlePutDirEvent(updatedDirectoryInfo, nsSummaryMap);
+ break;
+ }
+ }
+ }
+
+ private void processWithObjectStoreLayout(OmKeyInfo updatedKeyInfo,
Review Comment:
@ArafatKhan2198 thanks for working on this patch. I think we can refactor and reuse this OBS handling code with LEGACY of FS path enabled scenario when key doesn't end with / prefix (actual key case), so that below code duplication can be avoided.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9534. Support namespace summaries (du, dist & counts) for LEGACY buckets with file system disabled [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on PR #5517:
URL: https://github.com/apache/ozone/pull/5517#issuecomment-2027272717
@ArafatKhan2198 , you can ignore If (LOG.isDebugEnabled()) comments, as current logger library being used is already having check of `If (LOG.isDebugEnabled()) `, so you can fix rest of other comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9534 Support namespace summaries (du, dist & counts) for LEGACY buckets with file system disabled [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5517:
URL: https://github.com/apache/ozone/pull/5517#discussion_r1542880135
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTaskWithLegacy.java:
##########
@@ -275,7 +305,7 @@ public boolean reprocessWithLegacy(OMMetadataManager omMetadataManager) {
* @param keyInfo
* @throws IOException
*/
- private void setKeyParentID(OmKeyInfo keyInfo) throws IOException {
+ private void setParentDirectoryId(OmKeyInfo keyInfo) throws IOException {
Review Comment:
Reverted the change!
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9534. Support namespace summaries (du, dist & counts) for LEGACY buckets with file system disabled [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on PR #5517:
URL: https://github.com/apache/ozone/pull/5517#issuecomment-2027486349
Thanks @dombizita @sumitagrawl for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9534 Support namespace summaries (du, dist & counts) for LEGACY buckets with file system disabled [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #5517:
URL: https://github.com/apache/ozone/pull/5517#discussion_r1535255931
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java:
##########
@@ -743,6 +743,46 @@ public static String normalizeKey(String keyName,
return keyName;
}
+ /**
+ * Normalizes a given path up to the bucket level.
+ *
+ * This method takes a path as input and normalises uptil the bucket level.
+ * It handles empty, removes leading slashes, and splits the path into
+ * segments. It then extracts the volume and bucket names, forming a
+ * normalized path with a single slash. Finally, any remaining segments are
+ * joined as the key name, returning the complete standardized path.
+ *
+ * @param path The path string to be normalized.
+ * @return The normalized path string.
+ */
+ public static String normalizePathUptilBucket(String path) {
Review Comment:
This is called only for OBS, please check that it should not change file having "//" or other pattern, as OBS do not have normalization. While re-construction, it should get same value.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/EntityHandler.java:
##########
@@ -256,7 +280,52 @@ public static String[] parseRequestPath(String path) {
return names;
}
- private static String normalizePath(String path) {
+ /**
+ * Splits an object store path into volume, bucket, and key name components.
+ *
+ * This method parses a path of the format "/volumeName/bucketName/keyName",
+ * including paths with additional '/' characters within the key name. It's
+ * designed for object store paths where the first three '/' characters
+ * separate the root, volume and bucket names from the key name.
+ *
+ * @param path The object store path to parse, starting with a slash.
+ * @return A String array with three elements: volume name, bucket name, and
+ * key name, or {null} if the path format is invalid.
+ */
+ public static String[] parseObjectStorePath(String path) {
Review Comment:
this nees to be private
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9534. Support namespace summaries (du, dist & counts) for LEGACY buckets with file system disabled [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on code in PR #5517:
URL: https://github.com/apache/ozone/pull/5517#discussion_r1544348873
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/EntityHandler.java:
##########
@@ -256,7 +280,52 @@ public static String[] parseRequestPath(String path) {
return names;
}
- private static String normalizePath(String path) {
+ /**
+ * Splits an object store path into volume, bucket, and key name components.
+ *
+ * This method parses a path of the format "/volumeName/bucketName/keyName",
+ * including paths with additional '/' characters within the key name. It's
+ * designed for object store paths where the first three '/' characters
+ * separate the root, volume and bucket names from the key name.
+ *
+ * @param path The object store path to parse, starting with a slash.
+ * @return A String array with three elements: volume name, bucket name, and
+ * key name, or {null} if the path format is invalid.
+ */
+ public static String[] parseObjectStorePath(String path) {
+ // Removing the leading slash for correct splitting
+ path = path.substring(1);
+
+ // Splitting the modified path by "/", limiting to 3 parts
+ String[] parts = path.split("/", 3);
Review Comment:
```suggestion
String[] parts = path.split(OM_KEY_PREFIX, 3);
```
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java:
##########
@@ -743,6 +743,46 @@ public static String normalizeKey(String keyName,
return keyName;
}
+ /**
+ * Normalizes a given path up to the bucket level.
+ *
+ * This method takes a path as input and normalises uptil the bucket level.
+ * It handles empty, removes leading slashes, and splits the path into
+ * segments. It then extracts the volume and bucket names, forming a
+ * normalized path with a single slash. Finally, any remaining segments are
+ * joined as the key name, returning the complete standardized path.
+ *
+ * @param path The path string to be normalized.
+ * @return The normalized path string.
+ */
+ public static String normalizePathUptilBucket(String path) {
Review Comment:
Can we rename as `uptil` is not very familiar word to use in coding.
```suggestion
public static String normalizePathUptoBucket(String path) {
```
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTaskWithLegacy.java:
##########
@@ -206,12 +131,118 @@ public boolean processWithLegacy(OMUpdateEventBatch events) {
return true;
}
+ private void processWithFileSystemLayout(OmKeyInfo updatedKeyInfo,
+ OmKeyInfo oldKeyInfo,
+ OMDBUpdateEvent.OMDBUpdateAction action,
+ Map<Long, NSSummary> nsSummaryMap)
+ throws IOException {
+ setKeyParentID(updatedKeyInfo);
+
+ if (!updatedKeyInfo.getKeyName().endsWith(OM_KEY_PREFIX)) {
+ switch (action) {
+ case PUT:
+ handlePutKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+
+ case DELETE:
+ handleDeleteKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+
+ case UPDATE:
+ if (oldKeyInfo != null) {
+ setKeyParentID(oldKeyInfo);
+ handleDeleteKeyEvent(oldKeyInfo, nsSummaryMap);
+ } else {
+ LOG.warn("Update event does not have the old keyInfo for {}.",
+ updatedKeyInfo.getKeyName());
+ }
+ handlePutKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+
+ default:
+ LOG.debug("Skipping DB update event fir Key: {}", action);
Review Comment:
```suggestion
if (LOG.isDebugEnabled()) {
LOG.debug("Skipping DB update event fir Key: {}", action);
}
```
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTaskWithLegacy.java:
##########
@@ -206,12 +131,118 @@ public boolean processWithLegacy(OMUpdateEventBatch events) {
return true;
}
+ private void processWithFileSystemLayout(OmKeyInfo updatedKeyInfo,
+ OmKeyInfo oldKeyInfo,
+ OMDBUpdateEvent.OMDBUpdateAction action,
+ Map<Long, NSSummary> nsSummaryMap)
+ throws IOException {
+ setKeyParentID(updatedKeyInfo);
+
+ if (!updatedKeyInfo.getKeyName().endsWith(OM_KEY_PREFIX)) {
+ switch (action) {
+ case PUT:
+ handlePutKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+
+ case DELETE:
+ handleDeleteKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+
+ case UPDATE:
+ if (oldKeyInfo != null) {
+ setKeyParentID(oldKeyInfo);
+ handleDeleteKeyEvent(oldKeyInfo, nsSummaryMap);
+ } else {
+ LOG.warn("Update event does not have the old keyInfo for {}.",
+ updatedKeyInfo.getKeyName());
+ }
+ handlePutKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+
+ default:
+ LOG.debug("Skipping DB update event fir Key: {}", action);
Review Comment:
```suggestion
LOG.debug("Skipping DB update event for Key: {}", action);
```
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/EntityHandler.java:
##########
@@ -256,7 +280,52 @@ public static String[] parseRequestPath(String path) {
return names;
}
- private static String normalizePath(String path) {
+ /**
+ * Splits an object store path into volume, bucket, and key name components.
+ *
+ * This method parses a path of the format "/volumeName/bucketName/keyName",
+ * including paths with additional '/' characters within the key name. It's
+ * designed for object store paths where the first three '/' characters
+ * separate the root, volume and bucket names from the key name.
+ *
+ * @param path The object store path to parse, starting with a slash.
+ * @return A String array with three elements: volume name, bucket name, and
+ * key name, or {null} if the path format is invalid.
+ */
+ public static String[] parseObjectStorePath(String path) {
+ // Removing the leading slash for correct splitting
+ path = path.substring(1);
+
+ // Splitting the modified path by "/", limiting to 3 parts
+ String[] parts = path.split("/", 3);
+
+ // Checking if we correctly obtained 3 parts after removing the leading slash
+ if (parts.length <= 3) {
+ return parts;
+ } else {
+ return null;
+ }
+ }
+
+ /**
+ * Normalizes a given path based on the specified bucket layout.
+ *
+ * This method adjusts the path according to the bucket layout.
+ * For {OBJECT_STORE Layout}, it normalizes the path up to the bucket level
+ * using OmUtils.normalizePathUptilBucket. For other layouts, it
Review Comment:
```suggestion
* using OmUtils.normalizePathUptoBucket. For other layouts, it
```
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java:
##########
@@ -743,6 +743,46 @@ public static String normalizeKey(String keyName,
return keyName;
}
+ /**
+ * Normalizes a given path up to the bucket level.
+ *
+ * This method takes a path as input and normalises uptil the bucket level.
+ * It handles empty, removes leading slashes, and splits the path into
+ * segments. It then extracts the volume and bucket names, forming a
+ * normalized path with a single slash. Finally, any remaining segments are
+ * joined as the key name, returning the complete standardized path.
+ *
+ * @param path The path string to be normalized.
+ * @return The normalized path string.
+ */
+ public static String normalizePathUptilBucket(String path) {
+ if (path == null || path.isEmpty()) {
+ return "/"; // Handle empty path
Review Comment:
```suggestion
return OM_KEY_PREFIX; // Handle empty path
```
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTaskWithLegacy.java:
##########
@@ -206,12 +131,118 @@ public boolean processWithLegacy(OMUpdateEventBatch events) {
return true;
}
+ private void processWithFileSystemLayout(OmKeyInfo updatedKeyInfo,
+ OmKeyInfo oldKeyInfo,
+ OMDBUpdateEvent.OMDBUpdateAction action,
+ Map<Long, NSSummary> nsSummaryMap)
+ throws IOException {
+ setKeyParentID(updatedKeyInfo);
+
+ if (!updatedKeyInfo.getKeyName().endsWith(OM_KEY_PREFIX)) {
+ switch (action) {
+ case PUT:
+ handlePutKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+
+ case DELETE:
+ handleDeleteKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+
+ case UPDATE:
+ if (oldKeyInfo != null) {
+ setKeyParentID(oldKeyInfo);
+ handleDeleteKeyEvent(oldKeyInfo, nsSummaryMap);
+ } else {
+ LOG.warn("Update event does not have the old keyInfo for {}.",
+ updatedKeyInfo.getKeyName());
+ }
+ handlePutKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+
+ default:
+ LOG.debug("Skipping DB update event fir Key: {}", action);
+ }
+ } else {
+ OmDirectoryInfo updatedDirectoryInfo = new OmDirectoryInfo.Builder()
+ .setName(updatedKeyInfo.getKeyName())
+ .setObjectID(updatedKeyInfo.getObjectID())
+ .setParentObjectID(updatedKeyInfo.getParentObjectID())
+ .build();
+
+ OmDirectoryInfo oldDirectoryInfo = null;
+
+ if (oldKeyInfo != null) {
+ oldDirectoryInfo =
+ new OmDirectoryInfo.Builder()
+ .setName(oldKeyInfo.getKeyName())
+ .setObjectID(oldKeyInfo.getObjectID())
+ .setParentObjectID(oldKeyInfo.getParentObjectID())
+ .build();
+ }
+
+ switch (action) {
+ case PUT:
+ handlePutDirEvent(updatedDirectoryInfo, nsSummaryMap);
+ break;
+
+ case DELETE:
+ handleDeleteDirEvent(updatedDirectoryInfo, nsSummaryMap);
+ break;
+
+ case UPDATE:
+ if (oldDirectoryInfo != null) {
+ handleDeleteDirEvent(oldDirectoryInfo, nsSummaryMap);
+ } else {
+ LOG.warn("Update event does not have the old dirInfo for {}.",
+ updatedKeyInfo.getKeyName());
+ }
+ handlePutDirEvent(updatedDirectoryInfo, nsSummaryMap);
+ break;
+
+ default:
+ LOG.debug("Skipping DB update event for Directory: {}", action);
+ }
+ }
+ }
+
+ private void processWithObjectStoreLayout(OmKeyInfo updatedKeyInfo,
+ OmKeyInfo oldKeyInfo,
+ OMDBUpdateEvent.OMDBUpdateAction action,
+ Map<Long, NSSummary> nsSummaryMap)
+ throws IOException {
+ setParentBucketId(updatedKeyInfo);
+
+ switch (action) {
+ case PUT:
+ handlePutKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+
+ case DELETE:
+ handleDeleteKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+
+ case UPDATE:
+ if (oldKeyInfo != null) {
+ setParentBucketId(oldKeyInfo);
+ handleDeleteKeyEvent(oldKeyInfo, nsSummaryMap);
+ } else {
+ LOG.warn("Update event does not have the old keyInfo for {}.",
+ updatedKeyInfo.getKeyName());
+ }
+ handlePutKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+
+ default:
+ LOG.debug("Skipping DB update event for Key: {}", action);
Review Comment:
```suggestion
if(LOG.isDebugEnabled()) {
LOG.debug("Skipping DB update event for Key: {}", action);
}
```
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java:
##########
@@ -743,6 +743,46 @@ public static String normalizeKey(String keyName,
return keyName;
}
+ /**
+ * Normalizes a given path up to the bucket level.
+ *
+ * This method takes a path as input and normalises uptil the bucket level.
+ * It handles empty, removes leading slashes, and splits the path into
+ * segments. It then extracts the volume and bucket names, forming a
+ * normalized path with a single slash. Finally, any remaining segments are
+ * joined as the key name, returning the complete standardized path.
+ *
+ * @param path The path string to be normalized.
+ * @return The normalized path string.
+ */
+ public static String normalizePathUptilBucket(String path) {
+ if (path == null || path.isEmpty()) {
+ return "/"; // Handle empty path
+ }
+
+ // Remove leading slashes
+ path = path.replaceAll("^/*", "");
+
+ String[] segments = path.split("/", -1);
+
+ String volumeName = segments[0];
+ String bucketName = segments.length > 1 ? segments[1] : "";
+
+ // Combine volume and bucket.
+ StringBuilder normalizedPath = new StringBuilder(volumeName);
+ if (!bucketName.isEmpty()) {
+ normalizedPath.append("/").append(bucketName);
+ }
+
+ // Add remaining segments as the key
+ if (segments.length > 2) {
+ normalizedPath.append("/").append(
+ String.join("/", Arrays.copyOfRange(segments, 2, segments.length)));
Review Comment:
```suggestion
String.join(OM_KEY_PREFIX, Arrays.copyOfRange(segments, 2, segments.length)));
```
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTaskWithLegacy.java:
##########
@@ -206,12 +131,118 @@ public boolean processWithLegacy(OMUpdateEventBatch events) {
return true;
}
+ private void processWithFileSystemLayout(OmKeyInfo updatedKeyInfo,
+ OmKeyInfo oldKeyInfo,
+ OMDBUpdateEvent.OMDBUpdateAction action,
+ Map<Long, NSSummary> nsSummaryMap)
+ throws IOException {
+ setKeyParentID(updatedKeyInfo);
+
+ if (!updatedKeyInfo.getKeyName().endsWith(OM_KEY_PREFIX)) {
+ switch (action) {
+ case PUT:
+ handlePutKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+
+ case DELETE:
+ handleDeleteKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+
+ case UPDATE:
+ if (oldKeyInfo != null) {
+ setKeyParentID(oldKeyInfo);
+ handleDeleteKeyEvent(oldKeyInfo, nsSummaryMap);
+ } else {
+ LOG.warn("Update event does not have the old keyInfo for {}.",
+ updatedKeyInfo.getKeyName());
+ }
+ handlePutKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+
+ default:
+ LOG.debug("Skipping DB update event fir Key: {}", action);
+ }
+ } else {
+ OmDirectoryInfo updatedDirectoryInfo = new OmDirectoryInfo.Builder()
+ .setName(updatedKeyInfo.getKeyName())
+ .setObjectID(updatedKeyInfo.getObjectID())
+ .setParentObjectID(updatedKeyInfo.getParentObjectID())
+ .build();
+
+ OmDirectoryInfo oldDirectoryInfo = null;
+
+ if (oldKeyInfo != null) {
+ oldDirectoryInfo =
+ new OmDirectoryInfo.Builder()
+ .setName(oldKeyInfo.getKeyName())
+ .setObjectID(oldKeyInfo.getObjectID())
+ .setParentObjectID(oldKeyInfo.getParentObjectID())
+ .build();
+ }
+
+ switch (action) {
+ case PUT:
+ handlePutDirEvent(updatedDirectoryInfo, nsSummaryMap);
+ break;
+
+ case DELETE:
+ handleDeleteDirEvent(updatedDirectoryInfo, nsSummaryMap);
+ break;
+
+ case UPDATE:
+ if (oldDirectoryInfo != null) {
+ handleDeleteDirEvent(oldDirectoryInfo, nsSummaryMap);
+ } else {
+ LOG.warn("Update event does not have the old dirInfo for {}.",
+ updatedKeyInfo.getKeyName());
+ }
+ handlePutDirEvent(updatedDirectoryInfo, nsSummaryMap);
+ break;
+
+ default:
+ LOG.debug("Skipping DB update event for Directory: {}", action);
Review Comment:
```suggestion
if(LOG.isDebugEnabled()) {
LOG.debug("Skipping DB update event for Directory: {}", action);
}
```
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java:
##########
@@ -743,6 +743,46 @@ public static String normalizeKey(String keyName,
return keyName;
}
+ /**
+ * Normalizes a given path up to the bucket level.
+ *
+ * This method takes a path as input and normalises uptil the bucket level.
+ * It handles empty, removes leading slashes, and splits the path into
+ * segments. It then extracts the volume and bucket names, forming a
+ * normalized path with a single slash. Finally, any remaining segments are
+ * joined as the key name, returning the complete standardized path.
+ *
+ * @param path The path string to be normalized.
+ * @return The normalized path string.
+ */
+ public static String normalizePathUptilBucket(String path) {
+ if (path == null || path.isEmpty()) {
+ return "/"; // Handle empty path
+ }
+
+ // Remove leading slashes
+ path = path.replaceAll("^/*", "");
+
+ String[] segments = path.split("/", -1);
+
+ String volumeName = segments[0];
+ String bucketName = segments.length > 1 ? segments[1] : "";
+
+ // Combine volume and bucket.
+ StringBuilder normalizedPath = new StringBuilder(volumeName);
+ if (!bucketName.isEmpty()) {
+ normalizedPath.append("/").append(bucketName);
+ }
+
+ // Add remaining segments as the key
+ if (segments.length > 2) {
+ normalizedPath.append("/").append(
Review Comment:
```suggestion
normalizedPath.append(OM_KEY_PREFIX).append(
```
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java:
##########
@@ -743,6 +743,46 @@ public static String normalizeKey(String keyName,
return keyName;
}
+ /**
+ * Normalizes a given path up to the bucket level.
+ *
+ * This method takes a path as input and normalises uptil the bucket level.
+ * It handles empty, removes leading slashes, and splits the path into
+ * segments. It then extracts the volume and bucket names, forming a
+ * normalized path with a single slash. Finally, any remaining segments are
+ * joined as the key name, returning the complete standardized path.
+ *
+ * @param path The path string to be normalized.
+ * @return The normalized path string.
+ */
+ public static String normalizePathUptilBucket(String path) {
+ if (path == null || path.isEmpty()) {
+ return "/"; // Handle empty path
+ }
+
+ // Remove leading slashes
+ path = path.replaceAll("^/*", "");
+
+ String[] segments = path.split("/", -1);
+
+ String volumeName = segments[0];
+ String bucketName = segments.length > 1 ? segments[1] : "";
+
+ // Combine volume and bucket.
+ StringBuilder normalizedPath = new StringBuilder(volumeName);
+ if (!bucketName.isEmpty()) {
+ normalizedPath.append("/").append(bucketName);
Review Comment:
```suggestion
normalizedPath.append(OM_KEY_PREFIX).append(bucketName);
```
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTaskWithLegacy.java:
##########
@@ -300,17 +330,53 @@ private void setKeyParentID(OmKeyInfo keyInfo) throws IOException {
"NSSummaryTaskWithLegacy is null");
}
} else {
- String bucketKey = getReconOMMetadataManager()
- .getBucketKey(keyInfo.getVolumeName(), keyInfo.getBucketName());
- OmBucketInfo parentBucketInfo =
- getReconOMMetadataManager().getBucketTable().getSkipCache(bucketKey);
+ setParentBucketId(keyInfo);
+ }
+ }
- if (parentBucketInfo != null) {
- keyInfo.setParentObjectID(parentBucketInfo.getObjectID());
- } else {
- throw new IOException("ParentKeyInfo for " +
- "NSSummaryTaskWithLegacy is null");
- }
+ /**
+ * Set the parent object ID for a bucket.
+ *@paramkeyInfo
+ *@throwsIOException
+ */
+ private void setParentBucketId(OmKeyInfo keyInfo)
+ throws IOException {
+ String bucketKey = getReconOMMetadataManager()
+ .getBucketKey(keyInfo.getVolumeName(), keyInfo.getBucketName());
+ OmBucketInfo parentBucketInfo =
+ getReconOMMetadataManager().getBucketTable().getSkipCache(bucketKey);
+
+ if (parentBucketInfo != null) {
+ keyInfo.setParentObjectID(parentBucketInfo.getObjectID());
+ } else {
+ throw new IOException("ParentKeyInfo for " +
+ "NSSummaryTaskWithLegacy is null");
}
}
+
+ /**
+ * Check if the bucket layout is LEGACY.
+ * @param metadataManager
+ * @param keyInfo
+ * @return
+ */
+ private boolean isBucketLayoutValid(ReconOMMetadataManager metadataManager,
+ OmKeyInfo keyInfo)
+ throws IOException {
+ String volumeName = keyInfo.getVolumeName();
+ String bucketName = keyInfo.getBucketName();
+ String bucketDBKey = metadataManager.getBucketKey(volumeName, bucketName);
+ OmBucketInfo omBucketInfo =
+ metadataManager.getBucketTable().getSkipCache(bucketDBKey);
+
+ if (omBucketInfo.getBucketLayout() != LEGACY_BUCKET_LAYOUT) {
+ LOG.debug(
Review Comment:
```suggestion
if(LOG.isDebugEnabled()) {
LOG.debug(
}
```
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/EntityHandler.java:
##########
@@ -256,7 +280,52 @@ public static String[] parseRequestPath(String path) {
return names;
}
- private static String normalizePath(String path) {
+ /**
+ * Splits an object store path into volume, bucket, and key name components.
+ *
+ * This method parses a path of the format "/volumeName/bucketName/keyName",
+ * including paths with additional '/' characters within the key name. It's
+ * designed for object store paths where the first three '/' characters
+ * separate the root, volume and bucket names from the key name.
+ *
+ * @param path The object store path to parse, starting with a slash.
+ * @return A String array with three elements: volume name, bucket name, and
+ * key name, or {null} if the path format is invalid.
+ */
+ public static String[] parseObjectStorePath(String path) {
+ // Removing the leading slash for correct splitting
+ path = path.substring(1);
+
+ // Splitting the modified path by "/", limiting to 3 parts
+ String[] parts = path.split("/", 3);
+
+ // Checking if we correctly obtained 3 parts after removing the leading slash
+ if (parts.length <= 3) {
+ return parts;
+ } else {
+ return null;
+ }
+ }
+
+ /**
+ * Normalizes a given path based on the specified bucket layout.
+ *
+ * This method adjusts the path according to the bucket layout.
+ * For {OBJECT_STORE Layout}, it normalizes the path up to the bucket level
+ * using OmUtils.normalizePathUptilBucket. For other layouts, it
+ * normalizes the entire path, including the key, using
+ * OmUtils.normalizeKey, and does not preserve any trailing slashes.
+ * The normalized path will always be prefixed with OM_KEY_PREFIX to ensure it
+ * is consistent with the expected format for object storage paths in Ozone.
+ *
+ * @param path
+ * @param bucketLayout
+ * @return A normalized path
+ */
+ private static String normalizePath(String path, BucketLayout bucketLayout) {
+ if (bucketLayout == BucketLayout.OBJECT_STORE) {
+ return OM_KEY_PREFIX + OmUtils.normalizePathUptilBucket(path);
Review Comment:
```suggestion
return OM_KEY_PREFIX + OmUtils.normalizePathUptoBucket(path);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9534 Support namespace summaries (du, dist & counts) for LEGACY buckets with file system disabled [ozone]
Posted by "dombizita (via GitHub)" <gi...@apache.org>.
dombizita commented on code in PR #5517:
URL: https://github.com/apache/ozone/pull/5517#discussion_r1542847371
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTaskWithLegacy.java:
##########
@@ -275,7 +305,7 @@ public boolean reprocessWithLegacy(OMMetadataManager omMetadataManager) {
* @param keyInfo
* @throws IOException
*/
- private void setKeyParentID(OmKeyInfo keyInfo) throws IOException {
+ private void setParentDirectoryId(OmKeyInfo keyInfo) throws IOException {
Review Comment:
I thinks `setKeyParentID` makes more sense, why did you change it to `setParentDirectoryId`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9534. Support namespace summaries (du, dist & counts) for LEGACY buckets with file system disabled [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #5517:
URL: https://github.com/apache/ozone/pull/5517#discussion_r1544535601
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTaskWithLegacy.java:
##########
@@ -206,12 +131,118 @@ public boolean processWithLegacy(OMUpdateEventBatch events) {
return true;
}
+ private void processWithFileSystemLayout(OmKeyInfo updatedKeyInfo,
+ OmKeyInfo oldKeyInfo,
+ OMDBUpdateEvent.OMDBUpdateAction action,
+ Map<Long, NSSummary> nsSummaryMap)
+ throws IOException {
+ setKeyParentID(updatedKeyInfo);
+
+ if (!updatedKeyInfo.getKeyName().endsWith(OM_KEY_PREFIX)) {
+ switch (action) {
+ case PUT:
+ handlePutKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+
+ case DELETE:
+ handleDeleteKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+
+ case UPDATE:
+ if (oldKeyInfo != null) {
+ setKeyParentID(oldKeyInfo);
+ handleDeleteKeyEvent(oldKeyInfo, nsSummaryMap);
+ } else {
+ LOG.warn("Update event does not have the old keyInfo for {}.",
+ updatedKeyInfo.getKeyName());
+ }
+ handlePutKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+
+ default:
+ LOG.debug("Skipping DB update event fir Key: {}", action);
+ }
+ } else {
+ OmDirectoryInfo updatedDirectoryInfo = new OmDirectoryInfo.Builder()
+ .setName(updatedKeyInfo.getKeyName())
+ .setObjectID(updatedKeyInfo.getObjectID())
+ .setParentObjectID(updatedKeyInfo.getParentObjectID())
+ .build();
+
+ OmDirectoryInfo oldDirectoryInfo = null;
+
+ if (oldKeyInfo != null) {
+ oldDirectoryInfo =
+ new OmDirectoryInfo.Builder()
+ .setName(oldKeyInfo.getKeyName())
+ .setObjectID(oldKeyInfo.getObjectID())
+ .setParentObjectID(oldKeyInfo.getParentObjectID())
+ .build();
+ }
+
+ switch (action) {
+ case PUT:
+ handlePutDirEvent(updatedDirectoryInfo, nsSummaryMap);
+ break;
+
+ case DELETE:
+ handleDeleteDirEvent(updatedDirectoryInfo, nsSummaryMap);
+ break;
+
+ case UPDATE:
+ if (oldDirectoryInfo != null) {
+ handleDeleteDirEvent(oldDirectoryInfo, nsSummaryMap);
+ } else {
+ LOG.warn("Update event does not have the old dirInfo for {}.",
+ updatedKeyInfo.getKeyName());
+ }
+ handlePutDirEvent(updatedDirectoryInfo, nsSummaryMap);
+ break;
+
+ default:
+ LOG.debug("Skipping DB update event for Directory: {}", action);
Review Comment:
IMO, isDebugEnabled() check is not required, as debug() itself have first check as enabled and there is no expression evaluation, so this will not cause any performance impact.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9534. Support namespace summaries (du, dist & counts) for LEGACY buckets with file system disabled [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5517:
URL: https://github.com/apache/ozone/pull/5517#discussion_r1544578352
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTaskWithLegacy.java:
##########
@@ -206,12 +131,118 @@ public boolean processWithLegacy(OMUpdateEventBatch events) {
return true;
}
+ private void processWithFileSystemLayout(OmKeyInfo updatedKeyInfo,
+ OmKeyInfo oldKeyInfo,
+ OMDBUpdateEvent.OMDBUpdateAction action,
+ Map<Long, NSSummary> nsSummaryMap)
+ throws IOException {
+ setKeyParentID(updatedKeyInfo);
+
+ if (!updatedKeyInfo.getKeyName().endsWith(OM_KEY_PREFIX)) {
+ switch (action) {
+ case PUT:
+ handlePutKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+
+ case DELETE:
+ handleDeleteKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+
+ case UPDATE:
+ if (oldKeyInfo != null) {
+ setKeyParentID(oldKeyInfo);
+ handleDeleteKeyEvent(oldKeyInfo, nsSummaryMap);
+ } else {
+ LOG.warn("Update event does not have the old keyInfo for {}.",
+ updatedKeyInfo.getKeyName());
+ }
+ handlePutKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+
+ default:
+ LOG.debug("Skipping DB update event fir Key: {}", action);
Review Comment:
Done!
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTaskWithLegacy.java:
##########
@@ -206,12 +131,118 @@ public boolean processWithLegacy(OMUpdateEventBatch events) {
return true;
}
+ private void processWithFileSystemLayout(OmKeyInfo updatedKeyInfo,
+ OmKeyInfo oldKeyInfo,
+ OMDBUpdateEvent.OMDBUpdateAction action,
+ Map<Long, NSSummary> nsSummaryMap)
+ throws IOException {
+ setKeyParentID(updatedKeyInfo);
+
+ if (!updatedKeyInfo.getKeyName().endsWith(OM_KEY_PREFIX)) {
+ switch (action) {
+ case PUT:
+ handlePutKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+
+ case DELETE:
+ handleDeleteKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+
+ case UPDATE:
+ if (oldKeyInfo != null) {
+ setKeyParentID(oldKeyInfo);
+ handleDeleteKeyEvent(oldKeyInfo, nsSummaryMap);
+ } else {
+ LOG.warn("Update event does not have the old keyInfo for {}.",
+ updatedKeyInfo.getKeyName());
+ }
+ handlePutKeyEvent(updatedKeyInfo, nsSummaryMap);
+ break;
+
+ default:
+ LOG.debug("Skipping DB update event fir Key: {}", action);
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9534. Support namespace summaries (du, dist & counts) for LEGACY buckets with file system disabled [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu merged PR #5517:
URL: https://github.com/apache/ozone/pull/5517
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9534. Support namespace summaries (du, dist & counts) for LEGACY buckets with file system disabled [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5517:
URL: https://github.com/apache/ozone/pull/5517#discussion_r1544578194
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/EntityHandler.java:
##########
@@ -256,7 +280,52 @@ public static String[] parseRequestPath(String path) {
return names;
}
- private static String normalizePath(String path) {
+ /**
+ * Splits an object store path into volume, bucket, and key name components.
+ *
+ * This method parses a path of the format "/volumeName/bucketName/keyName",
+ * including paths with additional '/' characters within the key name. It's
+ * designed for object store paths where the first three '/' characters
+ * separate the root, volume and bucket names from the key name.
+ *
+ * @param path The object store path to parse, starting with a slash.
+ * @return A String array with three elements: volume name, bucket name, and
+ * key name, or {null} if the path format is invalid.
+ */
+ public static String[] parseObjectStorePath(String path) {
+ // Removing the leading slash for correct splitting
+ path = path.substring(1);
+
+ // Splitting the modified path by "/", limiting to 3 parts
+ String[] parts = path.split("/", 3);
Review Comment:
Done!
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/EntityHandler.java:
##########
@@ -256,7 +280,52 @@ public static String[] parseRequestPath(String path) {
return names;
}
- private static String normalizePath(String path) {
+ /**
+ * Splits an object store path into volume, bucket, and key name components.
+ *
+ * This method parses a path of the format "/volumeName/bucketName/keyName",
+ * including paths with additional '/' characters within the key name. It's
+ * designed for object store paths where the first three '/' characters
+ * separate the root, volume and bucket names from the key name.
+ *
+ * @param path The object store path to parse, starting with a slash.
+ * @return A String array with three elements: volume name, bucket name, and
+ * key name, or {null} if the path format is invalid.
+ */
+ public static String[] parseObjectStorePath(String path) {
+ // Removing the leading slash for correct splitting
+ path = path.substring(1);
+
+ // Splitting the modified path by "/", limiting to 3 parts
+ String[] parts = path.split("/", 3);
+
+ // Checking if we correctly obtained 3 parts after removing the leading slash
+ if (parts.length <= 3) {
+ return parts;
+ } else {
+ return null;
+ }
+ }
+
+ /**
+ * Normalizes a given path based on the specified bucket layout.
+ *
+ * This method adjusts the path according to the bucket layout.
+ * For {OBJECT_STORE Layout}, it normalizes the path up to the bucket level
+ * using OmUtils.normalizePathUptilBucket. For other layouts, it
Review Comment:
Done!
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/EntityHandler.java:
##########
@@ -256,7 +280,52 @@ public static String[] parseRequestPath(String path) {
return names;
}
- private static String normalizePath(String path) {
+ /**
+ * Splits an object store path into volume, bucket, and key name components.
+ *
+ * This method parses a path of the format "/volumeName/bucketName/keyName",
+ * including paths with additional '/' characters within the key name. It's
+ * designed for object store paths where the first three '/' characters
+ * separate the root, volume and bucket names from the key name.
+ *
+ * @param path The object store path to parse, starting with a slash.
+ * @return A String array with three elements: volume name, bucket name, and
+ * key name, or {null} if the path format is invalid.
+ */
+ public static String[] parseObjectStorePath(String path) {
+ // Removing the leading slash for correct splitting
+ path = path.substring(1);
+
+ // Splitting the modified path by "/", limiting to 3 parts
+ String[] parts = path.split("/", 3);
+
+ // Checking if we correctly obtained 3 parts after removing the leading slash
+ if (parts.length <= 3) {
+ return parts;
+ } else {
+ return null;
+ }
+ }
+
+ /**
+ * Normalizes a given path based on the specified bucket layout.
+ *
+ * This method adjusts the path according to the bucket layout.
+ * For {OBJECT_STORE Layout}, it normalizes the path up to the bucket level
+ * using OmUtils.normalizePathUptilBucket. For other layouts, it
+ * normalizes the entire path, including the key, using
+ * OmUtils.normalizeKey, and does not preserve any trailing slashes.
+ * The normalized path will always be prefixed with OM_KEY_PREFIX to ensure it
+ * is consistent with the expected format for object storage paths in Ozone.
+ *
+ * @param path
+ * @param bucketLayout
+ * @return A normalized path
+ */
+ private static String normalizePath(String path, BucketLayout bucketLayout) {
+ if (bucketLayout == BucketLayout.OBJECT_STORE) {
+ return OM_KEY_PREFIX + OmUtils.normalizePathUptilBucket(path);
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9534 Support namespace summaries (du, dist & counts) for LEGACY buckets with file system disabled [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #5517:
URL: https://github.com/apache/ozone/pull/5517#discussion_r1448424630
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/LegacyBucketHandler.java:
##########
@@ -267,6 +344,35 @@ public long handleDirectKeys(long parentId, boolean withReplica,
return keyDataSizeWithReplica;
}
+ /**
+ * This method handles disk usage calculation for legacy buckets.
+ * @param keyInfo the key info
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * @param listFile if listFile is enabled, append key DU as a subpath
+ */
+ private DUResponse.DiskUsage createDiskUsage(OmKeyInfo keyInfo,
+ boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData) {
+ DUResponse.DiskUsage diskUsage = new DUResponse.DiskUsage();
+ String objectName = keyInfo.getKeyName();
+ diskUsage.setSubpath(objectName);
Review Comment:
earlier used buildSubPath, plz check using objectName do same logic
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTaskWithLegacy.java:
##########
@@ -300,17 +321,53 @@ private void setKeyParentID(OmKeyInfo keyInfo) throws IOException {
"NSSummaryTaskWithLegacy is null");
}
} else {
- String bucketKey = getReconOMMetadataManager()
- .getBucketKey(keyInfo.getVolumeName(), keyInfo.getBucketName());
- OmBucketInfo parentBucketInfo =
- getReconOMMetadataManager().getBucketTable().getSkipCache(bucketKey);
+ setParentBucketId(keyInfo);
+ }
+ }
- if (parentBucketInfo != null) {
- keyInfo.setParentObjectID(parentBucketInfo.getObjectID());
- } else {
- throw new IOException("ParentKeyInfo for " +
- "NSSummaryTaskWithLegacy is null");
- }
+ /**
+ * Set the parent object ID for a bucket.
+ *@paramkeyInfo
+ *@throwsIOException
+ */
+ private void setParentBucketId(OmKeyInfo keyInfo)
+ throws IOException {
+ String bucketKey = getReconOMMetadataManager()
+ .getBucketKey(keyInfo.getVolumeName(), keyInfo.getBucketName());
+ OmBucketInfo parentBucketInfo =
+ getReconOMMetadataManager().getBucketTable().getSkipCache(bucketKey);
+
+ if (parentBucketInfo != null) {
+ keyInfo.setParentObjectID(parentBucketInfo.getObjectID());
+ } else {
+ throw new IOException("ParentKeyInfo for " +
+ "NSSummaryTaskWithLegacy is null");
+ }
+ }
+
+ /**
+ * Check if the bucket layout is LEGACY.
+ * @param metadataManager
+ * @param keyInfo
+ * @return
+ */
+ private boolean isBucketLayoutValid(ReconOMMetadataManager metadataManager,
+ OmKeyInfo keyInfo)
+ throws IOException {
+ String volumeName = keyInfo.getVolumeName();
+ String bucketName = keyInfo.getBucketName();
+ String bucketDBKey = metadataManager.getBucketKey(volumeName, bucketName);
+ OmBucketInfo omBucketInfo =
+ metadataManager.getBucketTable().getSkipCache(bucketDBKey);
+
+ if (omBucketInfo.getBucketLayout() != BUCKET_LAYOUT) {
Review Comment:
rename BUCKET_LAYOUT as LEGACY_ BUCKET_LAYOUT
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/LegacyBucketHandler.java:
##########
@@ -212,13 +220,93 @@ public long handleDirectKeys(long parentId, boolean withReplica,
return 0;
}
+ if (enableFileSystemPaths) {
+ keyDataSizeWithReplica +=
+ handleDirectKeysForFSOLayout(parentId, withReplica, listFile, duData,
+ keyTable, seekPrefix, nsSummary);
+ } else {
+ keyDataSizeWithReplica +=
+ handleDirectKeysForOBSLayout(withReplica, listFile, duData, keyTable,
+ seekPrefix);
+ }
+
+ return keyDataSizeWithReplica;
+ }
+
+ /**
+ * This method handles disk usage of direct keys for OBS layout.
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * @param listFile if listFile is enabled, append key DU as a subpath
+ * @param duData the current DU data
+ * @param keyTable the key table
+ * @param seekPrefix the seek prefix used to position the iterator
+ * @return the total DU of all direct keys
+ * @throws IOException
+ */
+ public long handleDirectKeysForOBSLayout(boolean withReplica,
Review Comment:
this can also rename similarly as **LegacyOBS
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/LegacyBucketHandler.java:
##########
@@ -212,13 +220,93 @@ public long handleDirectKeys(long parentId, boolean withReplica,
return 0;
}
+ if (enableFileSystemPaths) {
+ keyDataSizeWithReplica +=
+ handleDirectKeysForFSOLayout(parentId, withReplica, listFile, duData,
+ keyTable, seekPrefix, nsSummary);
+ } else {
+ keyDataSizeWithReplica +=
+ handleDirectKeysForOBSLayout(withReplica, listFile, duData, keyTable,
+ seekPrefix);
+ }
+
+ return keyDataSizeWithReplica;
+ }
+
+ /**
+ * This method handles disk usage of direct keys for OBS layout.
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * @param listFile if listFile is enabled, append key DU as a subpath
+ * @param duData the current DU data
+ * @param keyTable the key table
+ * @param seekPrefix the seek prefix used to position the iterator
+ * @return the total DU of all direct keys
+ * @throws IOException
+ */
+ public long handleDirectKeysForOBSLayout(boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ Table<String, OmKeyInfo> keyTable,
+ String seekPrefix)
+ throws IOException {
+ long keyDataSizeWithReplica = 0L;
+
+
+ try (
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>> iterator = keyTable.iterator()) {
+ iterator.seek(seekPrefix);
+
+ while (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+
+ if (!dbKey.startsWith(seekPrefix)) {
+ break;
+ }
+
+ OmKeyInfo keyInfo = kv.getValue();
+ if (keyInfo != null) {
+ createDiskUsage(keyInfo, withReplica, listFile, duData);
+ if (withReplica) {
+ long keyDU = keyInfo.getReplicatedSize();
+ keyDataSizeWithReplica += keyDU;
+ }
+ }
+ }
+ }
+
+ return keyDataSizeWithReplica;
+ }
+
+ /**
+ * This method handles disk usage of direct keys for OBS layout.
+ * @param parentId parent directory/bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * @param listFile if listFile is enabled, append key DU as a subpath
+ * @param duData the current DU data
+ * @param keyTable the key table
+ * @param seekPrefix the seek prefix used to position the iterator
+ * @param nsSummary of the parent directory/bucket
+ * @return the total DU of all direct keys
+ * @throws IOException
+ */
+ public long handleDirectKeysForFSOLayout(long parentId, boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ Table<String, OmKeyInfo> keyTable,
+ String seekPrefix,
+ NSSummary nsSummary)
+ throws IOException {
+ long keyDataSizeWithReplica = 0L;
+
if (omBucketInfo.getObjectID() != parentId) {
String dirName = nsSummary.getDirName();
seekPrefix += dirName;
}
String[] seekKeys = seekPrefix.split(OM_KEY_PREFIX);
- try (TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
- iterator = keyTable.iterator()) {
+ try (
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
Review Comment:
We are getting all keys in directory, right? not a recursive listing?
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/LegacyBucketHandler.java:
##########
@@ -212,13 +220,93 @@ public long handleDirectKeys(long parentId, boolean withReplica,
return 0;
}
+ if (enableFileSystemPaths) {
+ keyDataSizeWithReplica +=
+ handleDirectKeysForFSOLayout(parentId, withReplica, listFile, duData,
+ keyTable, seekPrefix, nsSummary);
+ } else {
+ keyDataSizeWithReplica +=
+ handleDirectKeysForOBSLayout(withReplica, listFile, duData, keyTable,
+ seekPrefix);
+ }
+
+ return keyDataSizeWithReplica;
+ }
+
+ /**
+ * This method handles disk usage of direct keys for OBS layout.
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * @param listFile if listFile is enabled, append key DU as a subpath
+ * @param duData the current DU data
+ * @param keyTable the key table
+ * @param seekPrefix the seek prefix used to position the iterator
+ * @return the total DU of all direct keys
+ * @throws IOException
+ */
+ public long handleDirectKeysForOBSLayout(boolean withReplica,
+ boolean listFile,
+ List<DUResponse.DiskUsage> duData,
+ Table<String, OmKeyInfo> keyTable,
+ String seekPrefix)
+ throws IOException {
+ long keyDataSizeWithReplica = 0L;
+
+
+ try (
+ TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>> iterator = keyTable.iterator()) {
+ iterator.seek(seekPrefix);
+
+ while (iterator.hasNext()) {
+ Table.KeyValue<String, OmKeyInfo> kv = iterator.next();
+ String dbKey = kv.getKey();
+
+ if (!dbKey.startsWith(seekPrefix)) {
+ break;
+ }
+
+ OmKeyInfo keyInfo = kv.getValue();
+ if (keyInfo != null) {
+ createDiskUsage(keyInfo, withReplica, listFile, duData);
+ if (withReplica) {
+ long keyDU = keyInfo.getReplicatedSize();
+ keyDataSizeWithReplica += keyDU;
+ }
+ }
+ }
+ }
+
+ return keyDataSizeWithReplica;
+ }
+
+ /**
+ * This method handles disk usage of direct keys for OBS layout.
+ * @param parentId parent directory/bucket
+ * @param withReplica if withReplica is enabled, set sizeWithReplica
+ * @param listFile if listFile is enabled, append key DU as a subpath
+ * @param duData the current DU data
+ * @param keyTable the key table
+ * @param seekPrefix the seek prefix used to position the iterator
+ * @param nsSummary of the parent directory/bucket
+ * @return the total DU of all direct keys
+ * @throws IOException
+ */
+ public long handleDirectKeysForFSOLayout(long parentId, boolean withReplica,
Review Comment:
This is not FSO, rename handleDirectKeysForLegacyFS, i.e. legacy file system and update comment also
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9534 Support namespace summaries (du, dist & counts) for LEGACY buckets with file system disabled [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on PR #5517:
URL: https://github.com/apache/ozone/pull/5517#issuecomment-1790153635
@devmadhuu @adoroszlai @GeorgeJahad Could you please take a look !!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9534 Support namespace summaries (du, dist & counts) for LEGACY buckets with file system disabled [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on PR #5517:
URL: https://github.com/apache/ozone/pull/5517#issuecomment-1895385315
Thanks for the comments @sumitagrawl as informed I will have to merge this one first https://github.com/apache/ozone/pull/4245 and then move on to incorporate the comments posted by you on this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org