You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ArafatKhan2198 (via GitHub)" <gi...@apache.org> on 2023/07/10 06:47:10 UTC
[GitHub] [ozone] ArafatKhan2198 opened a new pull request, #5037: HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories.
ArafatKhan2198 opened a new pull request, #5037:
URL: https://github.com/apache/ozone/pull/5037
## What changes were proposed in this pull request?
To obtain the number of `deletePending directories` and the amount of data associated with them, we need to implement the logic using the DU space calculation. This requires modifying the existing DU interface.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8627
## How was this patch tested?
Unit Testing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on PR #5037:
URL: https://github.com/apache/ozone/pull/5037#issuecomment-1841100668
@devmadhuu @adoroszlai @ashishkumar50
Could you please review this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1415695507
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableHandler.java:
##########
@@ -0,0 +1,93 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+/**
+ * Interface for handling PUT, DELETE and UPDATE events for size-related
+ * tables for OM Insights.
+ */
+public interface OmTableHandler {
+
+ /**
+ * Handles a PUT event for size-related tables by updating both the data
+ * sizes and their corresponding record counts in the tables.
+ *
+ * @param event The PUT event to be processed.
+ * @param tableName Table name associated with the event.
+ * @param sizeRelatedTables Tables Requiring Size Calculation.
+ * @param objectCountMap A map storing object counts.
+ * @param unreplicatedSizeCountMap A map storing unReplicated size counts.
+ * @param replicatedSizeCountMap A map storing replicated size counts.
+ * @throws IOException If an I/O error occurs during processing.
+ */
+ void handlePutEvent(OMDBUpdateEvent<String, Object> event,
Review Comment:
Thanks for pointing it out I have removed and logged it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1406107915
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OpenFileTableHandler.java:
##########
@@ -0,0 +1,144 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+
+/**
+ * Manages records in the OpenFile Table, updating counts and sizes of
+ * open files in the backend.
+ */
+public class OpenFileTableHandler implements OmTableHandler {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OpenFileTableHandler.class);
+
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeCountMap,
Review Comment:
`replicatedSizeCountMap` and `unreplicatedSizeCountMap` variable name convention seems confusing. We are storing only size not count here. Pls rename and correct in all classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1415698155
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OpenFileTableHandler.java:
##########
@@ -0,0 +1,144 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+
+/**
+ * Manages records in the OpenFile Table, updating counts and sizes of
+ * open files in the backend.
+ */
+public class OpenFileTableHandler implements OmTableHandler {
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ArafatKhan2198 commented on a diff in pull request #5037: HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories.
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1258726020
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -271,14 +327,15 @@ private void handlePutEvent(OMDBUpdateEvent<String, Object> event,
Collection<String> sizeRelatedTables,
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ArafatKhan2198 commented on a diff in pull request #5037: HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories.
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1258456882
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -179,6 +200,40 @@ private Triple<Long, Long, Long> getTableSizeAndCount(
return Triple.of(count, unReplicatedSize, replicatedSize);
}
+ /**
+ * Fetches the size of a deleted directory identified by the given path.
+ * The size is obtained from the NSSummary table using the directory objectID.
+ * The path is expected to be in the format :-
+ * "volumeId/bucketId/parentId/dirName/dirObjectId".
+ *
+ * @param path The path of the deleted directory.
+ * @return The size of the deleted directory.
+ * @throws IOException If an I/O error occurs while retrieving the size.
+ */
+ public long fetchSizeForDeletedDirectory(String path)
+ throws IOException {
+ if (path == null || path.isEmpty()) {
+ return 0L;
+ }
+ String[] parts = path.split("/");
+ String directoryObjectId = parts.length >= 6 ? parts[5] : "";
+ /* DB key in DeletedDirectoryTable =>
+ "volumeId/bucketId/parentId/dirName/dirObjectId" */
+ try {
+ long convertedValue = Long.parseLong(directoryObjectId);
Review Comment:
Thanks for the comment! I have made the changes!
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -179,6 +200,40 @@ private Triple<Long, Long, Long> getTableSizeAndCount(
return Triple.of(count, unReplicatedSize, replicatedSize);
}
+ /**
+ * Fetches the size of a deleted directory identified by the given path.
+ * The size is obtained from the NSSummary table using the directory objectID.
+ * The path is expected to be in the format :-
+ * "volumeId/bucketId/parentId/dirName/dirObjectId".
+ *
+ * @param path The path of the deleted directory.
+ * @return The size of the deleted directory.
+ * @throws IOException If an I/O error occurs while retrieving the size.
+ */
+ public long fetchSizeForDeletedDirectory(String path)
+ throws IOException {
+ if (path == null || path.isEmpty()) {
Review Comment:
Thanks for the comment! I have made the changes!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1414309070
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OpenKeysInsightHandler.java:
##########
@@ -0,0 +1,59 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+
+import java.io.IOException;
+import java.util.HashMap;
+
+public class OpenKeysInsightHandler {
Review Comment:
Thanks I have added it !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1415770727
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/DeletedTableHandler.java:
##########
@@ -0,0 +1,176 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.RepeatedOmKeyInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+/**
+ * Manages records in the Deleted Table, updating counts and sizes of
+ * pending Key Deletions in the backend.
+ */
+public class DeletedTableHandler implements OmTableHandler {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DeletedTableHandler.class);
+
+ /**
+ * Invoked by the process method to add information on those keys that have
+ * been backlogged in the backend for deletion.
+ */
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeCountMap,
+ HashMap<String, Long> replicatedSizeCountMap)
+ throws IOException {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo =
+ (RepeatedOmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey,
+ (k, count) -> count + repeatedOmKeyInfo.getOmKeyInfoList().size());
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size + result.getLeft());
+ replicatedSizeCountMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size + result.getRight());
+ } else {
+ LOG.warn("Put event does not have the Key Info for {}.",
+ event.getKey());
+ }
+
+ }
+
+ /**
+ * Invoked by the process method to remove information on those keys that have
+ * been successfully deleted from the backend.
+ */
+ @Override
+ public void handleDeleteEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeCountMap,
+ HashMap<String, Long> replicatedSizeCountMap)
+ throws IOException {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo =
+ (RepeatedOmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey, (k, count) ->
+ count > 0 ? count - repeatedOmKeyInfo.getOmKeyInfoList().size() : 0L);
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size > result.getLeft() ? size - result.getLeft() : 0L);
+ replicatedSizeCountMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size > result.getRight() ? size - result.getRight() :
+ 0L);
+ } else {
+ LOG.warn("Delete event does not have the Key Info for {}.",
+ event.getKey());
+ }
+ }
+
+ /**
+ * Invoked by the process method to update the statistics on the keys
+ * pending to be deleted.
+ */
+ @Override
+ public void handleUpdateEvent(OMDBUpdateEvent<String, Object> event,
Review Comment:
The original idea at the time was to add an `update` event handler for deleted tables because the size of the `keyInfoList` could change when new keys with similar names were deleted. This change would then impact the overall count and size of the deleted keys.
But since we are no longer storing duplicate keys in `repeatedOmKeyInfo` we do not require any update event method for DeletedKeysInsightHandler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ArafatKhan2198 commented on pull request #5037: HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories.
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on PR #5037:
URL: https://github.com/apache/ozone/pull/5037#issuecomment-1628345833
@devmadhuu @dombizita @ashishkumar50 can you please take a look !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on a diff in pull request #5037: HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories.
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1258421449
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -271,14 +327,15 @@ private void handlePutEvent(OMDBUpdateEvent<String, Object> event,
Collection<String> sizeRelatedTables,
Review Comment:
We don't need to pass this argument as "getTablesToCalculateSize" is public method.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -179,6 +200,40 @@ private Triple<Long, Long, Long> getTableSizeAndCount(
return Triple.of(count, unReplicatedSize, replicatedSize);
}
+ /**
+ * Fetches the size of a deleted directory identified by the given path.
+ * The size is obtained from the NSSummary table using the directory objectID.
+ * The path is expected to be in the format :-
+ * "volumeId/bucketId/parentId/dirName/dirObjectId".
+ *
+ * @param path The path of the deleted directory.
+ * @return The size of the deleted directory.
+ * @throws IOException If an I/O error occurs while retrieving the size.
+ */
+ public long fetchSizeForDeletedDirectory(String path)
+ throws IOException {
+ if (path == null || path.isEmpty()) {
+ return 0L;
+ }
+ String[] parts = path.split("/");
+ String directoryObjectId = parts.length >= 6 ? parts[5] : "";
Review Comment:
Can you add a comment why this special figure of 6 for length check ? Because as per key example, dirObjectId is at 4th index itself ?
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -337,21 +407,34 @@ private void handleDeleteEvent(OMDBUpdateEvent<String, Object> event,
private void handleSizeRelatedTableDeleteEvent(
OMDBUpdateEvent<String, Object> event, String countKey,
- String unReplicatedSizeKey, String replicatedSizeKey,
+ String unReplicatedSizeKey, String replicatedSizeKey, String tableName,
HashMap<String, Long> objectCountMap,
HashMap<String, Long> unreplicatedSizeCountMap,
- HashMap<String, Long> replicatedSizeCountMap) {
+ HashMap<String, Long> replicatedSizeCountMap) throws IOException {
+
+ Boolean isFileTable = tableName.equals(OPEN_FILE_TABLE);
+ Boolean isKeyTable = tableName.equals(OPEN_KEY_TABLE);
+ Boolean isDeletedDirTable = tableName.equals(DELETED_DIR_TABLE);
+
if (event.getValue() instanceof OmKeyInfo) {
// Handle DELETE for OpenKeyTable & OpenFileTable
OmKeyInfo omKeyInfo = (OmKeyInfo) event.getValue();
objectCountMap.computeIfPresent(countKey,
(k, count) -> count > 0 ? count - 1L : 0L);
- unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
- (k, size) -> size > omKeyInfo.getDataSize() ?
- size - omKeyInfo.getDataSize() : 0L);
- replicatedSizeCountMap.computeIfPresent(replicatedSizeKey,
- (k, size) -> size > omKeyInfo.getReplicatedSize() ?
- size - omKeyInfo.getReplicatedSize() : 0L);
+ if (isFileTable || isKeyTable) {
Review Comment:
Try to reduce if else complexity as this method can grow in future for file and key tables also.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -287,18 +344,30 @@ private void handlePutEvent(OMDBUpdateEvent<String, Object> event,
private void handleSizeRelatedTablePutEvent(
OMDBUpdateEvent<String, Object> event, String countKey,
- String unReplicatedSizeKey, String replicatedSizeKey,
+ String unReplicatedSizeKey, String replicatedSizeKey, String tableName,
HashMap<String, Long> objectCountMap,
HashMap<String, Long> unreplicatedSizeCountMap,
- HashMap<String, Long> replicatedSizeCountMap) {
+ HashMap<String, Long> replicatedSizeCountMap) throws IOException {
+
+ boolean isFileTable = tableName.equals(OPEN_FILE_TABLE);
+ boolean isKeyTable = tableName.equals(OPEN_KEY_TABLE);
+ boolean isDeletedDirTable = tableName.equals(DELETED_DIR_TABLE);
+
// Handle PUT for OpenKeyTable & OpenFileTable
if (event.getValue() instanceof OmKeyInfo) {
OmKeyInfo omKeyInfo = (OmKeyInfo) event.getValue();
objectCountMap.computeIfPresent(countKey, (k, count) -> count + 1L);
- unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
- (k, size) -> size + omKeyInfo.getDataSize());
- replicatedSizeCountMap.computeIfPresent(replicatedSizeKey,
- (k, size) -> size + omKeyInfo.getReplicatedSize());
+ if (isFileTable || isKeyTable) {
Review Comment:
We can reduce if else complexity here, as we know that size of tables are meant based on files/keys which can be computed later at bucket or volume level as well or can be computed for file and key table as well , try to keep the code generic as much as possible.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -287,18 +344,30 @@ private void handlePutEvent(OMDBUpdateEvent<String, Object> event,
private void handleSizeRelatedTablePutEvent(
OMDBUpdateEvent<String, Object> event, String countKey,
- String unReplicatedSizeKey, String replicatedSizeKey,
+ String unReplicatedSizeKey, String replicatedSizeKey, String tableName,
HashMap<String, Long> objectCountMap,
HashMap<String, Long> unreplicatedSizeCountMap,
- HashMap<String, Long> replicatedSizeCountMap) {
+ HashMap<String, Long> replicatedSizeCountMap) throws IOException {
+
+ boolean isFileTable = tableName.equals(OPEN_FILE_TABLE);
Review Comment:
variable name is confusing as these are for openfile and openkey tables, though these might not be needed if we can control at the caller of this method itself.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -139,28 +150,38 @@ public Pair<String, Boolean> reprocess(OMMetadataManager omMetadataManager) {
* Returns a triple with the total count of records (left), total unreplicated
* size (middle), and total replicated size (right) in the given iterator.
* Increments count for each record and adds the dataSize if a record's value
- * is an instance of OmKeyInfo. If the iterator is null, returns (0, 0, 0).
+ * is an instance of OmKeyInfo,RepeatedOmKeyInfo.
+ * If the iterator is null, returns (0, 0, 0).
*
* @param iterator The iterator over the table to be iterated.
* @return A Triple with three Long values representing the count,
* unreplicated size and replicated size.
* @throws IOException If an I/O error occurs during the iterator traversal.
*/
private Triple<Long, Long, Long> getTableSizeAndCount(
- TableIterator<String, ? extends Table.KeyValue<String, ?>> iterator)
+ TableIterator<String, ? extends Table.KeyValue<String, ?>> iterator,
Review Comment:
@ArafatKhan2198 thanks for working on this patch, can you pls elaborate the purpose of this method since you added tableName as extra parameter now. Is this method only for counting records of tables which are having keys as OMKeyInfo and RepeatedOmKeyInfo only ?. Also if GlobalStatsTable will not have counts of OpenFile and OpenKey table ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1469134319
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -65,14 +65,29 @@ public class OmTableInsightTask implements ReconOmTask {
private GlobalStatsDao globalStatsDao;
private Configuration sqlConfiguration;
private ReconOMMetadataManager reconOMMetadataManager;
+ private ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager;
+ private Table<Long, NSSummary> nsSummaryTable;
+ private Map<String, OmTableHandler> tableHandlers;
@Inject
public OmTableInsightTask(GlobalStatsDao globalStatsDao,
- Configuration sqlConfiguration,
- ReconOMMetadataManager reconOMMetadataManager) {
+ Configuration sqlConfiguration,
+ ReconOMMetadataManager reconOMMetadataManager,
+ ReconNamespaceSummaryManagerImpl
+ reconNamespaceSummaryManager) {
this.globalStatsDao = globalStatsDao;
this.sqlConfiguration = sqlConfiguration;
this.reconOMMetadataManager = reconOMMetadataManager;
+ this.reconNamespaceSummaryManager = reconNamespaceSummaryManager;
+ this.nsSummaryTable = reconNamespaceSummaryManager.getNSSummaryTable();
+
+ // Initialize table handlers
+ tableHandlers = new HashMap<>();
+ tableHandlers.put(OPEN_KEY_TABLE, new OpenKeysInsightHandler());
Review Comment:
Changes made!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1415697808
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/DeletedDirectoryTableHandler.java:
##########
@@ -0,0 +1,165 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.spi.impl.ReconNamespaceSummaryManagerImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Manages records in the Deleted Directory Table, updating counts and sizes of
+ * pending Directory Deletions in the backend.
+ */
+public class DeletedDirectoryTableHandler implements OmTableHandler {
+
+ private ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DeletedTableHandler.class);
+
+ public DeletedDirectoryTableHandler(
+ ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager) {
+ this.reconNamespaceSummaryManager = reconNamespaceSummaryManager;
+ }
+
+ /**
+ * Invoked by the process method to add information on those directories that
+ * have been backlogged in the backend for deletion.
+ */
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeCountMap,
+ HashMap<String, Long> replicatedSizeCountMap)
+ throws IOException {
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ OmKeyInfo omKeyInfo = (OmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey, (k, count) -> count + 1L);
+ Long newDeletedDirectorySize =
+ fetchSizeForDeletedDirectory(omKeyInfo.getObjectID());
+ unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size + newDeletedDirectorySize);
+ } else {
+ LOG.warn("Put event does not have the Key Info for {}.",
+ event.getKey());
+ }
+ }
+
+ /**
+ * Invoked by the process method to remove information on those directories
+ * that have been successfully deleted from the backend.
+ */
+ @Override
+ public void handleDeleteEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeCountMap,
+ HashMap<String, Long> replicatedSizeCountMap)
+ throws IOException {
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ OmKeyInfo omKeyInfo = (OmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey, (k, count) -> count - 1L);
+ Long newDeletedDirectorySize =
+ fetchSizeForDeletedDirectory(omKeyInfo.getObjectID());
+ unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
Review Comment:
I've implemented a method to retrieve the replication factor and then multiply it with the size of unReplicated data to calculate the total size of replicated data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1426676256
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableHandler.java:
##########
@@ -0,0 +1,89 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+/**
+ * Interface for handling PUT, DELETE and UPDATE events for size-related
+ * tables for OM Insights.
+ */
+public interface OmTableHandler {
+
+ /**
+ * Handles a PUT event for size-related tables by updating both the data
+ * sizes and their corresponding record counts in the tables.
+ *
+ * @param event The PUT event to be processed.
+ * @param tableName Table name associated with the event.
+ * @param sizeRelatedTables Tables Requiring Size Calculation.
+ * @param objectCountMap A map storing object counts.
+ * @param unReplicatedSizeMap A map storing unReplicated size counts.
+ * @param replicatedSizeMap A map storing replicated size counts.
+ * @throws IOException If an I/O error occurs during processing.
+ */
+ void handlePutEvent(OMDBUpdateEvent<String, Object> event,
Review Comment:
Pls remove throws clause from comment if not throwing any exception.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableHandler.java:
##########
@@ -0,0 +1,89 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+/**
+ * Interface for handling PUT, DELETE and UPDATE events for size-related
+ * tables for OM Insights.
+ */
+public interface OmTableHandler {
+
+ /**
+ * Handles a PUT event for size-related tables by updating both the data
+ * sizes and their corresponding record counts in the tables.
+ *
+ * @param event The PUT event to be processed.
+ * @param tableName Table name associated with the event.
+ * @param sizeRelatedTables Tables Requiring Size Calculation.
+ * @param objectCountMap A map storing object counts.
+ * @param unReplicatedSizeMap A map storing unReplicated size counts.
+ * @param replicatedSizeMap A map storing replicated size counts.
+ * @throws IOException If an I/O error occurs during processing.
+ */
+ void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap);
+ /**
+ * Handles a DELETE event for size-related tables by updating both the data
+ * sizes and their corresponding record counts in the tables.
+ *
+ * @param event The DELETE event to be processed.
+ * @param tableName Table name associated with the event.
+ * @param sizeRelatedTables Tables Requiring Size Calculation.
+ * @param objectCountMap A map storing object counts.
+ * @param unReplicatedSizeMap A map storing unReplicated size counts.
+ * @param replicatedSizeMap A map storing replicated size counts.
+ * @throws IOException If an I/O error occurs during processing.
+ */
+ void handleDeleteEvent(OMDBUpdateEvent<String, Object> event,
Review Comment:
Pls remove throws clause from comment if not throwing any exception.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/DeletedKeysInsightHandler.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.RepeatedOmKeyInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+/**
+ * Manages records in the Deleted Table, updating counts and sizes of
+ * pending Key Deletions in the backend.
+ */
+public class DeletedKeysInsightHandler implements OmTableHandler {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DeletedKeysInsightHandler.class);
+
+ /**
+ * Invoked by the process method to add information on those keys that have
+ * been backlogged in the backend for deletion.
+ */
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo =
+ (RepeatedOmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey,
+ (k, count) -> count + repeatedOmKeyInfo.getOmKeyInfoList().size());
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unReplicatedSizeMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size + result.getLeft());
+ replicatedSizeMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size + result.getRight());
+ } else {
+ LOG.warn("Put event does not have the Key Info for {}.",
+ event.getKey());
+ }
+
+ }
+
+ /**
+ * Invoked by the process method to remove information on those keys that have
+ * been successfully deleted from the backend.
+ */
+ @Override
+ public void handleDeleteEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo =
+ (RepeatedOmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey, (k, count) ->
+ count > 0 ? count - repeatedOmKeyInfo.getOmKeyInfoList().size() : 0L);
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unReplicatedSizeMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size > result.getLeft() ? size - result.getLeft() : 0L);
+ replicatedSizeMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size > result.getRight() ? size - result.getRight() :
+ 0L);
+ } else {
+ LOG.warn("Delete event does not have the Key Info for {}.",
+ event.getKey());
+ }
+ }
+
+ /**
+ * Invoked by the process method to update the statistics on the keys
+ * pending to be deleted.
+ */
+ @Override
+ public void handleUpdateEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+ // The size of deleted directories cannot change hence no-op.
Review Comment:
This event is for keys, pls correct the comment.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/DeletedKeysInsightHandler.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.RepeatedOmKeyInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+/**
+ * Manages records in the Deleted Table, updating counts and sizes of
+ * pending Key Deletions in the backend.
+ */
+public class DeletedKeysInsightHandler implements OmTableHandler {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DeletedKeysInsightHandler.class);
+
+ /**
+ * Invoked by the process method to add information on those keys that have
+ * been backlogged in the backend for deletion.
+ */
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo =
+ (RepeatedOmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey,
+ (k, count) -> count + repeatedOmKeyInfo.getOmKeyInfoList().size());
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unReplicatedSizeMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size + result.getLeft());
+ replicatedSizeMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size + result.getRight());
+ } else {
+ LOG.warn("Put event does not have the Key Info for {}.",
+ event.getKey());
+ }
+
+ }
+
+ /**
+ * Invoked by the process method to remove information on those keys that have
+ * been successfully deleted from the backend.
+ */
+ @Override
+ public void handleDeleteEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo =
+ (RepeatedOmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey, (k, count) ->
+ count > 0 ? count - repeatedOmKeyInfo.getOmKeyInfoList().size() : 0L);
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unReplicatedSizeMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size > result.getLeft() ? size - result.getLeft() : 0L);
+ replicatedSizeMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size > result.getRight() ? size - result.getRight() :
+ 0L);
+ } else {
+ LOG.warn("Delete event does not have the Key Info for {}.",
+ event.getKey());
+ }
+ }
+
+ /**
+ * Invoked by the process method to update the statistics on the keys
+ * pending to be deleted.
+ */
+ @Override
+ public void handleUpdateEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+ // The size of deleted directories cannot change hence no-op.
+ return;
+ }
+
+ /**
+ * Invoked by the reprocess method to calculate the records count of the
+ * deleted table and the sizes of replicated and unreplicated keys that are
+ * pending deletion in Ozone.
+ */
+ @Override
+ public Triple<Long, Long, Long> getTableSizeAndCount(
+ TableIterator<String, ? extends Table.KeyValue<String, ?>> iterator)
+ throws IOException {
+ long count = 0;
+ long unReplicatedSize = 0;
+ long replicatedSize = 0;
+
+ if (iterator != null) {
+ while (iterator.hasNext()) {
+ Table.KeyValue<String, ?> kv = iterator.next();
+ if (kv != null && kv.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo = (RepeatedOmKeyInfo) kv
+ .getValue();
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unReplicatedSize += result.getRight();
+ replicatedSize += result.getLeft();
+ // Since we can have multiple deleted keys of same name
Review Comment:
We may not have multiple deleted keys of same name.. Pls correct the comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1406118882
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/DeletedTableHandler.java:
##########
@@ -0,0 +1,176 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.RepeatedOmKeyInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+/**
+ * Manages records in the Deleted Table, updating counts and sizes of
+ * pending Key Deletions in the backend.
+ */
+public class DeletedTableHandler implements OmTableHandler {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DeletedTableHandler.class);
+
+ /**
+ * Invoked by the process method to add information on those keys that have
+ * been backlogged in the backend for deletion.
+ */
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeCountMap,
+ HashMap<String, Long> replicatedSizeCountMap)
+ throws IOException {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo =
+ (RepeatedOmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey,
+ (k, count) -> count + repeatedOmKeyInfo.getOmKeyInfoList().size());
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size + result.getLeft());
+ replicatedSizeCountMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size + result.getRight());
+ } else {
+ LOG.warn("Put event does not have the Key Info for {}.",
+ event.getKey());
+ }
+
+ }
+
+ /**
+ * Invoked by the process method to remove information on those keys that have
+ * been successfully deleted from the backend.
+ */
+ @Override
+ public void handleDeleteEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeCountMap,
+ HashMap<String, Long> replicatedSizeCountMap)
+ throws IOException {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo =
+ (RepeatedOmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey, (k, count) ->
+ count > 0 ? count - repeatedOmKeyInfo.getOmKeyInfoList().size() : 0L);
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size > result.getLeft() ? size - result.getLeft() : 0L);
+ replicatedSizeCountMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size > result.getRight() ? size - result.getRight() :
+ 0L);
+ } else {
+ LOG.warn("Delete event does not have the Key Info for {}.",
+ event.getKey());
+ }
+ }
+
+ /**
+ * Invoked by the process method to update the statistics on the keys
+ * pending to be deleted.
+ */
+ @Override
+ public void handleUpdateEvent(OMDBUpdateEvent<String, Object> event,
Review Comment:
Can you pls specify the business use case when any update event will be there for deleted table ? Do OM perform any update operations on deletedTable ?
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/DeletedTableHandler.java:
##########
@@ -0,0 +1,176 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.RepeatedOmKeyInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+/**
+ * Manages records in the Deleted Table, updating counts and sizes of
+ * pending Key Deletions in the backend.
+ */
+public class DeletedTableHandler implements OmTableHandler {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DeletedTableHandler.class);
+
+ /**
+ * Invoked by the process method to add information on those keys that have
+ * been backlogged in the backend for deletion.
+ */
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeCountMap,
+ HashMap<String, Long> replicatedSizeCountMap)
+ throws IOException {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo =
+ (RepeatedOmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey,
+ (k, count) -> count + repeatedOmKeyInfo.getOmKeyInfoList().size());
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size + result.getLeft());
+ replicatedSizeCountMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size + result.getRight());
+ } else {
+ LOG.warn("Put event does not have the Key Info for {}.",
+ event.getKey());
+ }
+
+ }
+
+ /**
+ * Invoked by the process method to remove information on those keys that have
+ * been successfully deleted from the backend.
+ */
+ @Override
+ public void handleDeleteEvent(OMDBUpdateEvent<String, Object> event,
Review Comment:
Because this service is not snapshot aware, handling delete may not give correct info how many files pending for delete and the amount of data to be reclaimed. Because if a key is deleted and present in deletedTable and at this point if snapshot taken then the key will be removed from active DB table and still hold and occupying space in snapshot DB table.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/DeletedDirectoryTableHandler.java:
##########
@@ -0,0 +1,165 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.spi.impl.ReconNamespaceSummaryManagerImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Manages records in the Deleted Directory Table, updating counts and sizes of
+ * pending Directory Deletions in the backend.
+ */
+public class DeletedDirectoryTableHandler implements OmTableHandler {
+
+ private ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DeletedTableHandler.class);
+
+ public DeletedDirectoryTableHandler(
+ ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager) {
+ this.reconNamespaceSummaryManager = reconNamespaceSummaryManager;
+ }
+
+ /**
+ * Invoked by the process method to add information on those directories that
+ * have been backlogged in the backend for deletion.
+ */
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeCountMap,
+ HashMap<String, Long> replicatedSizeCountMap)
+ throws IOException {
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ OmKeyInfo omKeyInfo = (OmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey, (k, count) -> count + 1L);
+ Long newDeletedDirectorySize =
+ fetchSizeForDeletedDirectory(omKeyInfo.getObjectID());
+ unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
Review Comment:
Based on replication config, pls check replicatedSize can be computed.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/DeletedDirectoryTableHandler.java:
##########
@@ -0,0 +1,165 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.spi.impl.ReconNamespaceSummaryManagerImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Manages records in the Deleted Directory Table, updating counts and sizes of
+ * pending Directory Deletions in the backend.
+ */
+public class DeletedDirectoryTableHandler implements OmTableHandler {
+
+ private ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DeletedTableHandler.class);
+
+ public DeletedDirectoryTableHandler(
+ ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager) {
+ this.reconNamespaceSummaryManager = reconNamespaceSummaryManager;
+ }
+
+ /**
+ * Invoked by the process method to add information on those directories that
+ * have been backlogged in the backend for deletion.
+ */
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeCountMap,
+ HashMap<String, Long> replicatedSizeCountMap)
+ throws IOException {
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ OmKeyInfo omKeyInfo = (OmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey, (k, count) -> count + 1L);
+ Long newDeletedDirectorySize =
+ fetchSizeForDeletedDirectory(omKeyInfo.getObjectID());
+ unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size + newDeletedDirectorySize);
+ } else {
+ LOG.warn("Put event does not have the Key Info for {}.",
+ event.getKey());
+ }
+ }
+
+ /**
+ * Invoked by the process method to remove information on those directories
+ * that have been successfully deleted from the backend.
+ */
+ @Override
+ public void handleDeleteEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeCountMap,
+ HashMap<String, Long> replicatedSizeCountMap)
+ throws IOException {
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ OmKeyInfo omKeyInfo = (OmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey, (k, count) -> count - 1L);
+ Long newDeletedDirectorySize =
+ fetchSizeForDeletedDirectory(omKeyInfo.getObjectID());
Review Comment:
Same comment related to Snapshot aware as above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1415698625
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OpenFileTableHandler.java:
##########
@@ -0,0 +1,144 @@
+package org.apache.hadoop.ozone.recon.tasks;
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories [ozone]
Posted by "dombizita (via GitHub)" <gi...@apache.org>.
dombizita commented on PR #5037:
URL: https://github.com/apache/ozone/pull/5037#issuecomment-1935758644
Please adjust the Jira ([HDDS-8627](https://issues.apache.org/jira/browse/HDDS-8627)) title and description to the changes discussed during the PR review @ArafatKhan2198 @sumitagrawl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1415770727
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/DeletedTableHandler.java:
##########
@@ -0,0 +1,176 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.RepeatedOmKeyInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+/**
+ * Manages records in the Deleted Table, updating counts and sizes of
+ * pending Key Deletions in the backend.
+ */
+public class DeletedTableHandler implements OmTableHandler {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DeletedTableHandler.class);
+
+ /**
+ * Invoked by the process method to add information on those keys that have
+ * been backlogged in the backend for deletion.
+ */
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeCountMap,
+ HashMap<String, Long> replicatedSizeCountMap)
+ throws IOException {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo =
+ (RepeatedOmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey,
+ (k, count) -> count + repeatedOmKeyInfo.getOmKeyInfoList().size());
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size + result.getLeft());
+ replicatedSizeCountMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size + result.getRight());
+ } else {
+ LOG.warn("Put event does not have the Key Info for {}.",
+ event.getKey());
+ }
+
+ }
+
+ /**
+ * Invoked by the process method to remove information on those keys that have
+ * been successfully deleted from the backend.
+ */
+ @Override
+ public void handleDeleteEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeCountMap,
+ HashMap<String, Long> replicatedSizeCountMap)
+ throws IOException {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo =
+ (RepeatedOmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey, (k, count) ->
+ count > 0 ? count - repeatedOmKeyInfo.getOmKeyInfoList().size() : 0L);
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size > result.getLeft() ? size - result.getLeft() : 0L);
+ replicatedSizeCountMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size > result.getRight() ? size - result.getRight() :
+ 0L);
+ } else {
+ LOG.warn("Delete event does not have the Key Info for {}.",
+ event.getKey());
+ }
+ }
+
+ /**
+ * Invoked by the process method to update the statistics on the keys
+ * pending to be deleted.
+ */
+ @Override
+ public void handleUpdateEvent(OMDBUpdateEvent<String, Object> event,
Review Comment:
The original idea at the time was to add an `update` event handler for deleted tables because the size of the `keyInfoList` could change when new keys with similar names were deleted. This change would then impact the overall count and size of the deleted keys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1415777390
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/DeletedDirectoryTableHandler.java:
##########
@@ -0,0 +1,165 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.spi.impl.ReconNamespaceSummaryManagerImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Manages records in the Deleted Directory Table, updating counts and sizes of
+ * pending Directory Deletions in the backend.
+ */
+public class DeletedDirectoryTableHandler implements OmTableHandler {
+
+ private ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DeletedTableHandler.class);
+
+ public DeletedDirectoryTableHandler(
+ ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager) {
+ this.reconNamespaceSummaryManager = reconNamespaceSummaryManager;
+ }
+
+ /**
+ * Invoked by the process method to add information on those directories that
+ * have been backlogged in the backend for deletion.
+ */
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeCountMap,
+ HashMap<String, Long> replicatedSizeCountMap)
+ throws IOException {
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ OmKeyInfo omKeyInfo = (OmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey, (k, count) -> count + 1L);
+ Long newDeletedDirectorySize =
+ fetchSizeForDeletedDirectory(omKeyInfo.getObjectID());
+ unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size + newDeletedDirectorySize);
+ } else {
+ LOG.warn("Put event does not have the Key Info for {}.",
+ event.getKey());
+ }
+ }
+
+ /**
+ * Invoked by the process method to remove information on those directories
+ * that have been successfully deleted from the backend.
+ */
+ @Override
+ public void handleDeleteEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeCountMap,
+ HashMap<String, Long> replicatedSizeCountMap)
+ throws IOException {
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ OmKeyInfo omKeyInfo = (OmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey, (k, count) -> count - 1L);
+ Long newDeletedDirectorySize =
+ fetchSizeForDeletedDirectory(omKeyInfo.getObjectID());
Review Comment:
You are right @devmadhuu, we can address that in a separate Jira issue. It will require its own design discussions and testing, and including it in this pull request would make it overly complex since this PR is already quite big. Currently, I am in the process of creating a small document outlining how we can make this service snapshot-aware, and I will share it with here shortly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1414312501
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OpenFileTableHandler.java:
##########
@@ -0,0 +1,144 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.RepeatedOmKeyInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+import static org.apache.hadoop.ozone.om.OmMetadataManagerImpl.*;
+
+
+public class OpenFileTableHandler implements OmTableHandler {
Review Comment:
Thank you for your comment, Devesh. As discussed previously, I have removed the mentioned class. We will now use a single class, referred to as the OpenKeysInsightHandler, for both the KeyTable and FileTable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ArafatKhan2198 commented on a diff in pull request #5037: HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories.
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1258700031
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -179,6 +200,40 @@ private Triple<Long, Long, Long> getTableSizeAndCount(
return Triple.of(count, unReplicatedSize, replicatedSize);
}
+ /**
+ * Fetches the size of a deleted directory identified by the given path.
+ * The size is obtained from the NSSummary table using the directory objectID.
+ * The path is expected to be in the format :-
+ * "volumeId/bucketId/parentId/dirName/dirObjectId".
+ *
+ * @param path The path of the deleted directory.
+ * @return The size of the deleted directory.
+ * @throws IOException If an I/O error occurs while retrieving the size.
+ */
+ public long fetchSizeForDeletedDirectory(String path)
+ throws IOException {
+ if (path == null || path.isEmpty()) {
+ return 0L;
+ }
+ String[] parts = path.split("/");
+ String directoryObjectId = parts.length >= 6 ? parts[5] : "";
Review Comment:
My appologies for the mistake, I forgot to add a slash `"/"` at the starting of the path given in comment
The fetchSizeForDeletedDirectory method is used to fetch the size of a deleted directory based on its path. The path is expected to be in the format: `"/volumeId/bucketId/parentId/dirName/dirObjectId"`.
Here's a breakdown of how the method works:
1. The method first checks if the provided path is null or empty. If it is, an error is logged, and the method returns a size of 0.
2. The path is then split using the `"/"` delimiter. The resulting array, `parts`, will contain the individual components of the path. For example, if the path is `"/volumeId/bucketId/parentId/dirName/dirObjectId"`, the parts array will be **_`["", "volumeId", "bucketId", "parentId", "dirName", "objectId"]`_**.
3. The method checks if the parts array has at least 6 elements. If it doesn't, it indicates an invalid path format, and an error is logged. The method returns a size of 0 in this case.
4. The directory object ID is extracted from the parts array. In the example path, the directory object ID would be "`objectId`" `(parts[5])`.
5. The directory object ID is parsed as a long value. If parsing is successful, the method retrieves the corresponding NSSummary object from the nsSummaryTable using the directory object ID as the key.
6. If an NSSummary object is found, the method returns the size of the deleted directory obtained from the NSSummary object using the getSizeOfFiles() method.
7. If no NSSummary object is found, an error is logged indicating that the NSSummary was not found for the directory.
If any error occurs during the process, the method returns a size of 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ArafatKhan2198 commented on a diff in pull request #5037: HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories.
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1258655747
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -139,28 +150,38 @@ public Pair<String, Boolean> reprocess(OMMetadataManager omMetadataManager) {
* Returns a triple with the total count of records (left), total unreplicated
* size (middle), and total replicated size (right) in the given iterator.
* Increments count for each record and adds the dataSize if a record's value
- * is an instance of OmKeyInfo. If the iterator is null, returns (0, 0, 0).
+ * is an instance of OmKeyInfo,RepeatedOmKeyInfo.
+ * If the iterator is null, returns (0, 0, 0).
*
* @param iterator The iterator over the table to be iterated.
* @return A Triple with three Long values representing the count,
* unreplicated size and replicated size.
* @throws IOException If an I/O error occurs during the iterator traversal.
*/
private Triple<Long, Long, Long> getTableSizeAndCount(
- TableIterator<String, ? extends Table.KeyValue<String, ?>> iterator)
+ TableIterator<String, ? extends Table.KeyValue<String, ?>> iterator,
Review Comment:
Thanks for the comment @devmadhuu
The purpose of the `getTableSizeAndCount` method is to calculate the **total count of records**, **total unreplicated size, and total replicated** **size** in a given iterator over a table, based on the specified tableName. It is only used for tables which want both the size as well as the count to be calculated hence in our case it will only work for `OPEN_KEY_TABLE`, `OPEN_FILE_TABLE`, `DELETED_TABLE` & `DELETED_DIR_TABLE`.
**Table** **Name** | **Object Associated**
-- | --
OPEN_KEY_TABLE | OmKeyInfo
OPEN_FILE_TABLE | OmKeyInfo
DELETED_DIR_TABLE | OmKeyInfo
DELETED_TABLE | RepeatedOmKeyInfo
In our case, the OPEN_KEY_TABLE, OPEN_FILE_TABLE, and DELETED_DIR_TABLE all have `OmKeyInfo` associated with them. Therefore, we need to differentiate between them based on the table name. The reason for this differentiation is because the calculation of data differs for the OPEN_KEY_TABLE and OPEN_FILE_TABLE compared to the DELETED_DIR_TABLE.
To summarize, when calculating data for the OPEN_KEY_TABLE and OPEN_FILE_TABLE, we consider the dataSize of the keys stored in the OmKeyInfo objects. However, for the DELETED_DIR_TABLE, we retrieve the size of the deleted directory from the `NSSummary` table. **It's important to note that the replicatedSize value is not applicable to the DELETED_DIR_TABLE.**
So, to clarify, the `globalStats` table stores both the `sizes` and `counts` of tables, and the `getTableSizeAndCount` method calculates these values and updates the globalStats table accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ashishkumar50 commented on a diff in pull request #5037: HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories.
Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1258226978
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -179,6 +200,40 @@ private Triple<Long, Long, Long> getTableSizeAndCount(
return Triple.of(count, unReplicatedSize, replicatedSize);
}
+ /**
+ * Fetches the size of a deleted directory identified by the given path.
+ * The size is obtained from the NSSummary table using the directory objectID.
+ * The path is expected to be in the format :-
+ * "volumeId/bucketId/parentId/dirName/dirObjectId".
+ *
+ * @param path The path of the deleted directory.
+ * @return The size of the deleted directory.
+ * @throws IOException If an I/O error occurs while retrieving the size.
+ */
+ public long fetchSizeForDeletedDirectory(String path)
+ throws IOException {
+ if (path == null || path.isEmpty()) {
Review Comment:
```suggestion
Strings.isNullOrEmpty(path) {
```
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -179,6 +200,40 @@ private Triple<Long, Long, Long> getTableSizeAndCount(
return Triple.of(count, unReplicatedSize, replicatedSize);
}
+ /**
+ * Fetches the size of a deleted directory identified by the given path.
+ * The size is obtained from the NSSummary table using the directory objectID.
+ * The path is expected to be in the format :-
+ * "volumeId/bucketId/parentId/dirName/dirObjectId".
+ *
+ * @param path The path of the deleted directory.
+ * @return The size of the deleted directory.
+ * @throws IOException If an I/O error occurs while retrieving the size.
+ */
+ public long fetchSizeForDeletedDirectory(String path)
+ throws IOException {
+ if (path == null || path.isEmpty()) {
+ return 0L;
+ }
+ String[] parts = path.split("/");
+ String directoryObjectId = parts.length >= 6 ? parts[5] : "";
+ /* DB key in DeletedDirectoryTable =>
+ "volumeId/bucketId/parentId/dirName/dirObjectId" */
+ try {
+ long convertedValue = Long.parseLong(directoryObjectId);
Review Comment:
In case `parts.length < 6` we are assigning `directoryObjectId` with "" which will throw `NumberFormatException`. Instead we should return error or 0 when this happens.
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/TestOmTableInsightTask.java:
##########
@@ -21,26 +21,36 @@
import org.apache.commons.lang3.tuple.ImmutablePair;
import org.apache.commons.lang3.tuple.Pair;
import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
import org.apache.hadoop.hdds.utils.db.TypedTable;
import org.apache.hadoop.ozone.om.OMMetadataManager;
import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
-import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
-import org.apache.hadoop.ozone.om.helpers.RepeatedOmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.*;
Review Comment:
Avoid using .* in import.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -139,28 +150,38 @@ public Pair<String, Boolean> reprocess(OMMetadataManager omMetadataManager) {
* Returns a triple with the total count of records (left), total unreplicated
* size (middle), and total replicated size (right) in the given iterator.
* Increments count for each record and adds the dataSize if a record's value
- * is an instance of OmKeyInfo. If the iterator is null, returns (0, 0, 0).
+ * is an instance of OmKeyInfo,RepeatedOmKeyInfo.
+ * If the iterator is null, returns (0, 0, 0).
*
* @param iterator The iterator over the table to be iterated.
Review Comment:
Add new param tableName in javadoc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on PR #5037:
URL: https://github.com/apache/ozone/pull/5037#issuecomment-1914054004
Initially, we attempted to implement the DU space calculation of **deleted directories** using `NSSummaries`. However, Sumit highlighted the difficulty in determining the total size of all deleted directories using our original approach. Currently, we iterate through the `DeletedDirectoryTable`, calculate the size of each deleted directory, and then find the sum total size. The issue arises when both the parent and child entries are encountered in the DeletedDirectoryTable, resulting in the same size being calculated twice.
To address the challenge of calculating the size of deleted directories, especially when encountering both parent and child entries in the `DeletedDirectoryTable`, we are adjusting our approach. To obtain the count of deletePending directories and their corresponding data size, we can derive the data size from the keys of the deleted directories. These keys are already being calculated and are an integral part of the deleted directories.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1471461731
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestEndpoints.java:
##########
@@ -263,6 +265,8 @@ private void initializeInjector() throws Exception {
reconScm = (ReconStorageContainerManagerFacade)
reconTestInjector.getInstance(OzoneStorageContainerManager.class);
+ reconNamespaceSummaryManager = reconTestInjector.getInstance(
Review Comment:
Done!
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OpenKeysInsightHandler.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+
+/**
+ * Manages records in the OpenKey Table, updating counts and sizes of
+ * open keys in the backend.
+ */
+public class OpenKeysInsightHandler implements OmTableHandler {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OpenKeysInsightHandler.class);
+
+ /**
+ * Invoked by the process method to add information on those keys that have
+ * been open in the backend.
+ */
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ OmKeyInfo omKeyInfo = (OmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey, (k, count) -> count + 1L);
+ unReplicatedSizeMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size + omKeyInfo.getDataSize());
+ replicatedSizeMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size + omKeyInfo.getReplicatedSize());
+ } else {
+ LOG.warn("Put event does not have the Key Info for {}.",
+ event.getKey());
+ }
+ }
+
+ /**
+ * Invoked by the process method to delete information on those keys that are
+ * no longer closed in the backend.
+ */
+ @Override
+ public void handleDeleteEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ OmKeyInfo omKeyInfo = (OmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey,
+ (k, count) -> count > 0 ? count - 1L : 0L);
+ unReplicatedSizeMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size > omKeyInfo.getDataSize() ?
+ size - omKeyInfo.getDataSize() : 0L);
+ replicatedSizeMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size > omKeyInfo.getReplicatedSize() ?
+ size - omKeyInfo.getReplicatedSize() : 0L);
+ } else {
+ LOG.warn("Delete event does not have the Key Info for {}.",
+ event.getKey());
+ }
+ }
+
+ /**
+ * Invoked by the process method to update information on those open keys that
+ * have been updated in the backend.
+ */
+ @Override
+ public void handleUpdateEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+
+ if (event.getValue() != null) {
+ if (event.getOldValue() == null) {
+ LOG.warn("Update event does not have the old Key Info for {}.",
+ event.getKey());
+ return;
+ }
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ // In Update event the count for the open table will not change. So we
+ // don't need to update the count.
+ OmKeyInfo oldKeyInfo = (OmKeyInfo) event.getOldValue();
+ OmKeyInfo newKeyInfo = (OmKeyInfo) event.getValue();
+ unReplicatedSizeMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size - oldKeyInfo.getDataSize() +
+ newKeyInfo.getDataSize());
+ replicatedSizeMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size - oldKeyInfo.getReplicatedSize() +
+ newKeyInfo.getReplicatedSize());
+ } else {
+ LOG.warn("Update event does not have the Key Info for {}.",
+ event.getKey());
+ }
+ }
+
+ /**
+ * This method is called by the reprocess method. It calculates the record
+ * counts for both the open key table and the open file table. Additionally,
+ * it computes the sizes of both replicated and unreplicated keys
+ * that are currently open in the backend.
+ */
+ @Override
+ public Triple<Long, Long, Long> getTableSizeAndCount(
+ TableIterator<String, ? extends Table.KeyValue<String, ?>> iterator)
+ throws IOException {
+ long count = 0;
+ long unReplicatedSize = 0;
+ long replicatedSize = 0;
+
+ if (iterator != null) {
+ while (iterator.hasNext()) {
+ Table.KeyValue<String, ?> kv = iterator.next();
+ if (kv != null && kv.getValue() != null) {
+ OmKeyInfo omKeyInfo = (OmKeyInfo) kv.getValue();
+ unReplicatedSize += omKeyInfo.getDataSize();
+ replicatedSize += omKeyInfo.getReplicatedSize();
+ count++;
+ }
+ }
+ }
+ return Triple.of(count, unReplicatedSize, replicatedSize);
+ }
+
+ public static String getTableCountKeyFromTable(String tableName) {
Review Comment:
Done!
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -335,109 +271,33 @@ private void handleDeleteEvent(OMDBUpdateEvent<String, Object> event,
}
}
- private void handleSizeRelatedTableDeleteEvent(
- OMDBUpdateEvent<String, Object> event,
- String tableName,
- HashMap<String, Long> objectCountMap,
- HashMap<String, Long> unreplicatedSizeCountMap,
- HashMap<String, Long> replicatedSizeCountMap) {
-
- String countKey = getTableCountKeyFromTable(tableName);
- String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
- String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
-
- if (event.getValue() instanceof OmKeyInfo) {
- // Handle DELETE for OpenKeyTable & OpenFileTable
- OmKeyInfo omKeyInfo = (OmKeyInfo) event.getValue();
- objectCountMap.computeIfPresent(countKey,
- (k, count) -> count > 0 ? count - 1L : 0L);
- unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
- (k, size) -> size > omKeyInfo.getDataSize() ?
- size - omKeyInfo.getDataSize() : 0L);
- replicatedSizeCountMap.computeIfPresent(replicatedSizeKey,
- (k, size) -> size > omKeyInfo.getReplicatedSize() ?
- size - omKeyInfo.getReplicatedSize() : 0L);
- } else if (event.getValue() instanceof RepeatedOmKeyInfo) {
- // Handle DELETE for DeletedTable
- RepeatedOmKeyInfo repeatedOmKeyInfo =
- (RepeatedOmKeyInfo) event.getValue();
- objectCountMap.computeIfPresent(countKey, (k, count) -> count > 0 ?
- count - repeatedOmKeyInfo.getOmKeyInfoList().size() : 0L);
- Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
- unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
- (k, size) -> size > result.getLeft() ? size - result.getLeft() : 0L);
- replicatedSizeCountMap.computeIfPresent(replicatedSizeKey,
- (k, size) -> size > result.getRight() ? size - result.getRight() :
- 0L);
- }
- }
+ /**
+ * Handle update events for size related tables.
+ */
private void handleUpdateEvent(OMDBUpdateEvent<String, Object> event,
String tableName,
Collection<String> sizeRelatedTables,
HashMap<String, Long> objectCountMap,
- HashMap<String, Long> unreplicatedSizeCountMap,
- HashMap<String, Long> replicatedSizeCountMap) {
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+
+ OmTableHandler tableHandler = tableHandlers.get(tableName);
if (event.getValue() != null) {
if (sizeRelatedTables.contains(tableName)) {
Review Comment:
Done!
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -270,63 +236,33 @@ private void handlePutEvent(OMDBUpdateEvent<String, Object> event,
String tableName,
Collection<String> sizeRelatedTables,
HashMap<String, Long> objectCountMap,
- HashMap<String, Long> unreplicatedSizeCountMap,
- HashMap<String, Long> replicatedSizeCountMap) {
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap)
+ throws IOException {
+ OmTableHandler tableHandler = tableHandlers.get(tableName);
- if (sizeRelatedTables.contains(tableName)) {
- handleSizeRelatedTablePutEvent(event, tableName, objectCountMap,
- unreplicatedSizeCountMap, replicatedSizeCountMap);
+ if (sizeRelatedTables.contains(tableName) && tableHandler != null) {
+ tableHandler.handlePutEvent(event, tableName, objectCountMap,
+ unReplicatedSizeMap, replicatedSizeMap);
} else {
String countKey = getTableCountKeyFromTable(tableName);
objectCountMap.computeIfPresent(countKey, (k, count) -> count + 1L);
}
}
- private void handleSizeRelatedTablePutEvent(
- OMDBUpdateEvent<String, Object> event,
- String tableName,
- HashMap<String, Long> objectCountMap,
- HashMap<String, Long> unreplicatedSizeCountMap,
- HashMap<String, Long> replicatedSizeCountMap) {
-
- String countKey = getTableCountKeyFromTable(tableName);
- String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
- String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
-
- if (event.getValue() instanceof OmKeyInfo) {
- // Handle PUT for OpenKeyTable & OpenFileTable
- OmKeyInfo omKeyInfo = (OmKeyInfo) event.getValue();
- objectCountMap.computeIfPresent(countKey, (k, count) -> count + 1L);
- unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
- (k, size) -> size + omKeyInfo.getDataSize());
- replicatedSizeCountMap.computeIfPresent(replicatedSizeKey,
- (k, size) -> size + omKeyInfo.getReplicatedSize());
- } else if (event.getValue() instanceof RepeatedOmKeyInfo) {
- // Handle PUT for DeletedTable
- RepeatedOmKeyInfo repeatedOmKeyInfo =
- (RepeatedOmKeyInfo) event.getValue();
- objectCountMap.computeIfPresent(countKey,
- (k, count) -> count + repeatedOmKeyInfo.getOmKeyInfoList().size());
- Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
- unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
- (k, size) -> size + result.getLeft());
- replicatedSizeCountMap.computeIfPresent(replicatedSizeKey,
- (k, size) -> size + result.getRight());
- }
- }
-
-
private void handleDeleteEvent(OMDBUpdateEvent<String, Object> event,
String tableName,
Collection<String> sizeRelatedTables,
HashMap<String, Long> objectCountMap,
- HashMap<String, Long> unreplicatedSizeCountMap,
- HashMap<String, Long> replicatedSizeCountMap) {
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap)
+ throws IOException {
+ OmTableHandler tableHandler = tableHandlers.get(tableName);
if (event.getValue() != null) {
if (sizeRelatedTables.contains(tableName)) {
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1471462671
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/ReconNamespaceSummaryManagerImpl.java:
##########
@@ -84,4 +85,9 @@ public void commitBatchOperation(RDBBatchOperation rdbBatchOperation)
public Table getNSSummaryTable() {
return nsSummaryTable;
}
+
+ @VisibleForTesting
+ public void setNsSummaryTable(Table<Long, NSSummary> nsSummaryTable) {
Review Comment:
Done!
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/OMDBInsightEndpoint.java:
##########
@@ -651,6 +652,36 @@ public Response getDeletedDirInfo(
return Response.ok(deletedDirInsightInfo).build();
}
+ /**
+ * Retrieves the summary of deleted directories.
+ *
+ * This method calculates and returns a summary of deleted directories.
+ * @return The HTTP response body includes a map with the following entries:
+ * - "totalDeletedDirectories": the total number of deleted directories
+ *
+ * Example response:
+ * {
+ * "totalDeletedDirectories": 8,
+ * }
+ */
+ @GET
+ @Path("/deletePending/dirs/summary")
+ public Response getDeletedDirectorySummary() throws IOException {
Review Comment:
Done !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl merged PR #5037:
URL: https://github.com/apache/ozone/pull/5037
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1415776817
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/DeletedTableHandler.java:
##########
@@ -0,0 +1,176 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.RepeatedOmKeyInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+/**
+ * Manages records in the Deleted Table, updating counts and sizes of
+ * pending Key Deletions in the backend.
+ */
+public class DeletedTableHandler implements OmTableHandler {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DeletedTableHandler.class);
+
+ /**
+ * Invoked by the process method to add information on those keys that have
+ * been backlogged in the backend for deletion.
+ */
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeCountMap,
+ HashMap<String, Long> replicatedSizeCountMap)
+ throws IOException {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo =
+ (RepeatedOmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey,
+ (k, count) -> count + repeatedOmKeyInfo.getOmKeyInfoList().size());
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size + result.getLeft());
+ replicatedSizeCountMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size + result.getRight());
+ } else {
+ LOG.warn("Put event does not have the Key Info for {}.",
+ event.getKey());
+ }
+
+ }
+
+ /**
+ * Invoked by the process method to remove information on those keys that have
+ * been successfully deleted from the backend.
+ */
+ @Override
+ public void handleDeleteEvent(OMDBUpdateEvent<String, Object> event,
Review Comment:
You are right @devmadhuu, we can address that in a separate Jira issue. It will require its own design discussions and testing, and including it in this pull request would make it overly complex since this PR is already quite big. Currently, I am in the process of creating a small document outlining how we can make this service snapshot-aware, and I will share it with here shortly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1414310088
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OpenKeysInsightHandler.java:
##########
@@ -0,0 +1,59 @@
+package org.apache.hadoop.ozone.recon.tasks;
Review Comment:
Thanks I have added it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on PR #5037:
URL: https://github.com/apache/ozone/pull/5037#issuecomment-1935760345
> Please adjust the Jira ([HDDS-8627](https://issues.apache.org/jira/browse/HDDS-8627)) title and description to the changes discussed during the PR review @ArafatKhan2198 @sumitagrawl
Thanks for pointing it out will do!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ArafatKhan2198 commented on a diff in pull request #5037: HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories.
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1258701081
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -179,6 +200,40 @@ private Triple<Long, Long, Long> getTableSizeAndCount(
return Triple.of(count, unReplicatedSize, replicatedSize);
}
+ /**
+ * Fetches the size of a deleted directory identified by the given path.
+ * The size is obtained from the NSSummary table using the directory objectID.
+ * The path is expected to be in the format :-
+ * "volumeId/bucketId/parentId/dirName/dirObjectId".
+ *
+ * @param path The path of the deleted directory.
+ * @return The size of the deleted directory.
+ * @throws IOException If an I/O error occurs while retrieving the size.
+ */
+ public long fetchSizeForDeletedDirectory(String path)
+ throws IOException {
+ if (path == null || path.isEmpty()) {
+ return 0L;
+ }
+ String[] parts = path.split("/");
+ String directoryObjectId = parts.length >= 6 ? parts[5] : "";
Review Comment:
I have also verified the path from the `OmMetadataManagerImpl` clas where the formats of all the OM tables are mentioned :-
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ArafatKhan2198 commented on a diff in pull request #5037: HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories.
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1258655747
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -139,28 +150,38 @@ public Pair<String, Boolean> reprocess(OMMetadataManager omMetadataManager) {
* Returns a triple with the total count of records (left), total unreplicated
* size (middle), and total replicated size (right) in the given iterator.
* Increments count for each record and adds the dataSize if a record's value
- * is an instance of OmKeyInfo. If the iterator is null, returns (0, 0, 0).
+ * is an instance of OmKeyInfo,RepeatedOmKeyInfo.
+ * If the iterator is null, returns (0, 0, 0).
*
* @param iterator The iterator over the table to be iterated.
* @return A Triple with three Long values representing the count,
* unreplicated size and replicated size.
* @throws IOException If an I/O error occurs during the iterator traversal.
*/
private Triple<Long, Long, Long> getTableSizeAndCount(
- TableIterator<String, ? extends Table.KeyValue<String, ?>> iterator)
+ TableIterator<String, ? extends Table.KeyValue<String, ?>> iterator,
Review Comment:
Thanks for the comment @devmadhuu
The purpose of the `getTableSizeAndCount` method is to calculate the **total count of records**, **total unreplicated size, and total replicated** **size** in a given iterator over a table, based on the specified tableName. It is only used for tables which want both the size as well as the count to be calculated hence in our case it will only work for `OPEN_KEY_TABLE`, `OPEN_FILE_TABLE`, `DELETED_TABLE` & `DELETED_DIR_TABLE`.
**Table** **Name** | **Object Associated**
-- | --
OPEN_KEY_TABLE | OmKeyInfo
OPEN_FILE_TABLE | OmKeyInfo
DELETED_DIR_TABLE | OmKeyInfo
DELETED_TABLE | RepeatedOmKeyInfo
In our case, the OPEN_KEY_TABLE, OPEN_FILE_TABLE, and DELETED_DIR_TABLE all have `OmKeyInfo` associated with them. Therefore, we need to differentiate between them based on the table name. The reason for this differentiation is because the calculation of data differs for the OPEN_KEY_TABLE and OPEN_FILE_TABLE compared to the DELETED_DIR_TABLE.
To summarize, when calculating data for the OPEN_KEY_TABLE and OPEN_FILE_TABLE, we consider the dataSize of the keys stored in the OmKeyInfo objects. However, for the DELETED_DIR_TABLE, we retrieve the size of the deleted directory from the `NSSummary` table. **It's important to note that the replicatedSize value is not applicable to the DELETED_DIR_TABLE.**
So, to clarify, the `globalStats` table stores both the `sizes` and `counts` of tables, and the getTableSizeAndCount method calculates these values and updates the globalStats table accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1415770727
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/DeletedTableHandler.java:
##########
@@ -0,0 +1,176 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.RepeatedOmKeyInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+/**
+ * Manages records in the Deleted Table, updating counts and sizes of
+ * pending Key Deletions in the backend.
+ */
+public class DeletedTableHandler implements OmTableHandler {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DeletedTableHandler.class);
+
+ /**
+ * Invoked by the process method to add information on those keys that have
+ * been backlogged in the backend for deletion.
+ */
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeCountMap,
+ HashMap<String, Long> replicatedSizeCountMap)
+ throws IOException {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo =
+ (RepeatedOmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey,
+ (k, count) -> count + repeatedOmKeyInfo.getOmKeyInfoList().size());
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size + result.getLeft());
+ replicatedSizeCountMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size + result.getRight());
+ } else {
+ LOG.warn("Put event does not have the Key Info for {}.",
+ event.getKey());
+ }
+
+ }
+
+ /**
+ * Invoked by the process method to remove information on those keys that have
+ * been successfully deleted from the backend.
+ */
+ @Override
+ public void handleDeleteEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeCountMap,
+ HashMap<String, Long> replicatedSizeCountMap)
+ throws IOException {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo =
+ (RepeatedOmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey, (k, count) ->
+ count > 0 ? count - repeatedOmKeyInfo.getOmKeyInfoList().size() : 0L);
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size > result.getLeft() ? size - result.getLeft() : 0L);
+ replicatedSizeCountMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size > result.getRight() ? size - result.getRight() :
+ 0L);
+ } else {
+ LOG.warn("Delete event does not have the Key Info for {}.",
+ event.getKey());
+ }
+ }
+
+ /**
+ * Invoked by the process method to update the statistics on the keys
+ * pending to be deleted.
+ */
+ @Override
+ public void handleUpdateEvent(OMDBUpdateEvent<String, Object> event,
Review Comment:
The original idea at the time was to add an `update` event handler for deleted tables because the size of the `keyInfoList` could change when new keys with similar names were deleted. This change would then impact the overall count and size of the deleted keys.
But since we are no longer storing duplicate keys in `repeatedOmKeyInfo` we do not require any update event handler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1415770727
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/DeletedTableHandler.java:
##########
@@ -0,0 +1,176 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.RepeatedOmKeyInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+/**
+ * Manages records in the Deleted Table, updating counts and sizes of
+ * pending Key Deletions in the backend.
+ */
+public class DeletedTableHandler implements OmTableHandler {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DeletedTableHandler.class);
+
+ /**
+ * Invoked by the process method to add information on those keys that have
+ * been backlogged in the backend for deletion.
+ */
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeCountMap,
+ HashMap<String, Long> replicatedSizeCountMap)
+ throws IOException {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo =
+ (RepeatedOmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey,
+ (k, count) -> count + repeatedOmKeyInfo.getOmKeyInfoList().size());
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size + result.getLeft());
+ replicatedSizeCountMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size + result.getRight());
+ } else {
+ LOG.warn("Put event does not have the Key Info for {}.",
+ event.getKey());
+ }
+
+ }
+
+ /**
+ * Invoked by the process method to remove information on those keys that have
+ * been successfully deleted from the backend.
+ */
+ @Override
+ public void handleDeleteEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeCountMap,
+ HashMap<String, Long> replicatedSizeCountMap)
+ throws IOException {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo =
+ (RepeatedOmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey, (k, count) ->
+ count > 0 ? count - repeatedOmKeyInfo.getOmKeyInfoList().size() : 0L);
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size > result.getLeft() ? size - result.getLeft() : 0L);
+ replicatedSizeCountMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size > result.getRight() ? size - result.getRight() :
+ 0L);
+ } else {
+ LOG.warn("Delete event does not have the Key Info for {}.",
+ event.getKey());
+ }
+ }
+
+ /**
+ * Invoked by the process method to update the statistics on the keys
+ * pending to be deleted.
+ */
+ @Override
+ public void handleUpdateEvent(OMDBUpdateEvent<String, Object> event,
Review Comment:
The original idea at the time was to add an `update` event handler for deleted tables because the size of the `keyInfoList` could change when new keys with similar names were deleted. This change would then impact the overall count and size of the deleted keys.
But since we are no longer storing duplicate keys in RepeatedOmKeyInfo we do not require any update event handler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1415875770
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OpenFileTableHandler.java:
##########
@@ -0,0 +1,144 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+
+/**
+ * Manages records in the OpenFile Table, updating counts and sizes of
+ * open files in the backend.
+ */
+public class OpenFileTableHandler implements OmTableHandler {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OpenFileTableHandler.class);
+
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeCountMap,
Review Comment:
Thanks for the suggestion I have renamed them to `replicatedSizeMap` & `unReplicatedSizeMap`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1406028260
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OpenKeysInsightHandler.java:
##########
@@ -0,0 +1,59 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+
+import java.io.IOException;
+import java.util.HashMap;
+
+public class OpenKeysInsightHandler {
Review Comment:
@ArafatKhan2198 thanks for continued work on this. Pls add a javadoc comment for class mentioning the purpose and details.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OpenKeyTableHandler.java:
##########
@@ -0,0 +1,141 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+
+public class OpenKeyTableHandler implements OmTableHandler {
Review Comment:
@ArafatKhan2198 thanks for continued work on this. Pls add a javadoc comment for class mentioning the purpose and details.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/DeletedDirectoryTableHandler.java:
##########
@@ -0,0 +1,165 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.spi.impl.ReconNamespaceSummaryManagerImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Manages records in the Deleted Directory Table, updating counts and sizes of
+ * pending Directory Deletions in the backend.
+ */
+public class DeletedDirectoryTableHandler implements OmTableHandler {
+
+ private ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DeletedTableHandler.class);
+
+ public DeletedDirectoryTableHandler(
+ ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager) {
+ this.reconNamespaceSummaryManager = reconNamespaceSummaryManager;
+ }
+
+ /**
+ * Invoked by the process method to add information on those directories that
+ * have been backlogged in the backend for deletion.
+ */
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeCountMap,
+ HashMap<String, Long> replicatedSizeCountMap)
+ throws IOException {
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ OmKeyInfo omKeyInfo = (OmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey, (k, count) -> count + 1L);
+ Long newDeletedDirectorySize =
+ fetchSizeForDeletedDirectory(omKeyInfo.getObjectID());
+ unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size + newDeletedDirectorySize);
+ } else {
+ LOG.warn("Put event does not have the Key Info for {}.",
+ event.getKey());
+ }
+ }
+
+ /**
+ * Invoked by the process method to remove information on those directories
+ * that have been successfully deleted from the backend.
+ */
+ @Override
+ public void handleDeleteEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeCountMap,
+ HashMap<String, Long> replicatedSizeCountMap)
+ throws IOException {
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ OmKeyInfo omKeyInfo = (OmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey, (k, count) -> count - 1L);
+ Long newDeletedDirectorySize =
+ fetchSizeForDeletedDirectory(omKeyInfo.getObjectID());
+ unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
Review Comment:
We can compute replicatedSIze using replicationFactor. Pls check that.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableHandler.java:
##########
@@ -0,0 +1,93 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+/**
+ * Interface for handling PUT, DELETE and UPDATE events for size-related
+ * tables for OM Insights.
+ */
+public interface OmTableHandler {
+
+ /**
+ * Handles a PUT event for size-related tables by updating both the data
+ * sizes and their corresponding record counts in the tables.
+ *
+ * @param event The PUT event to be processed.
+ * @param tableName Table name associated with the event.
+ * @param sizeRelatedTables Tables Requiring Size Calculation.
+ * @param objectCountMap A map storing object counts.
+ * @param unreplicatedSizeCountMap A map storing unReplicated size counts.
+ * @param replicatedSizeCountMap A map storing replicated size counts.
+ * @throws IOException If an I/O error occurs during processing.
+ */
+ void handlePutEvent(OMDBUpdateEvent<String, Object> event,
Review Comment:
Pls check if these methods needs to declare throws IOException. I don't see usage. Pls check
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OpenKeyTableHandler.java:
##########
@@ -0,0 +1,141 @@
+package org.apache.hadoop.ozone.recon.tasks;
Review Comment:
Copyright info is missing.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OpenFileTableHandler.java:
##########
@@ -0,0 +1,144 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.RepeatedOmKeyInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+import static org.apache.hadoop.ozone.om.OmMetadataManagerImpl.*;
+
+
+public class OpenFileTableHandler implements OmTableHandler {
Review Comment:
@ArafatKhan2198 thanks for continued work on this. Pls add a javadoc comment for class mentioning the purpose and details.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OpenKeysInsightHandler.java:
##########
@@ -0,0 +1,59 @@
+package org.apache.hadoop.ozone.recon.tasks;
Review Comment:
Copyright info is missing.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OpenFileTableHandler.java:
##########
@@ -0,0 +1,144 @@
+package org.apache.hadoop.ozone.recon.tasks;
Review Comment:
Pls add copyright info. Check for all new classes.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OpenFileTableHandler.java:
##########
@@ -0,0 +1,144 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+
+/**
+ * Manages records in the OpenFile Table, updating counts and sizes of
+ * open files in the backend.
+ */
+public class OpenFileTableHandler implements OmTableHandler {
Review Comment:
Both `OpenFileTableHandler` and `OpenKeyTableHandler` files can be refactored for common code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1458488420
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/OMMetadataManagerTestUtils.java:
##########
@@ -397,23 +397,31 @@ public static void writeDirToOm(OMMetadataManager omMetadataManager,
.build());
}
+ @SuppressWarnings("parameternumber")
public static void writeDeletedDirToOm(OMMetadataManager omMetadataManager,
String bucketName,
String volumeName,
String dirName,
long parentObjectId,
long bucketObjectId,
- long volumeObjectId)
+ long volumeObjectId,
+ long objectId)
throws IOException {
- // DB key in DeletedDirectoryTable => "volumeID/bucketID/parentId/dirName"
- String omKey = omMetadataManager.getOzonePathKey(volumeObjectId,
- bucketObjectId, parentObjectId, dirName);
+ // DB key in DeletedDirectoryTable =>
+ // "volumeID/bucketID/parentId/dirName/dirObjectId"
+
+ String ozoneDbKey = omMetadataManager.getOzonePathKey(volumeObjectId,
+ bucketObjectId, parentObjectId, dirName);
+ String ozoneDeleteKey = omMetadataManager.getOzoneDeletePathKey(
+ objectId, ozoneDbKey);
+
Review Comment:
No, it has always been `volumeID/bucketID/parentId/dirName/dirObjectId`. The earlier comment was incorrect, and I failed to notice the mistake when it was committed earlier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1458486442
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -124,69 +139,26 @@ public Pair<String, Boolean> reprocess(OMMetadataManager omMetadataManager) {
if (!objectCountMap.isEmpty()) {
writeDataToDB(objectCountMap);
}
- if (!unReplicatedSizeCountMap.isEmpty()) {
- writeDataToDB(unReplicatedSizeCountMap);
+ if (!unReplicatedSizeMap.isEmpty()) {
+ writeDataToDB(unReplicatedSizeMap);
}
- if (!replicatedSizeCountMap.isEmpty()) {
- writeDataToDB(replicatedSizeCountMap);
+ if (!replicatedSizeMap.isEmpty()) {
+ writeDataToDB(replicatedSizeMap);
}
LOG.info("Completed a 'reprocess' run of OmTableInsightTask.");
return new ImmutablePair<>(getTaskName(), true);
}
- /**
- * Returns a triple with the total count of records (left), total unreplicated
- * size (middle), and total replicated size (right) in the given iterator.
- * Increments count for each record and adds the dataSize if a record's value
- * is an instance of OmKeyInfo. If the iterator is null, returns (0, 0, 0).
- *
- * @param iterator The iterator over the table to be iterated.
- * @return A Triple with three Long values representing the count,
- * unreplicated size and replicated size.
- * @throws IOException If an I/O error occurs during the iterator traversal.
- */
- private Triple<Long, Long, Long> getTableSizeAndCount(
- TableIterator<String, ? extends Table.KeyValue<String, ?>> iterator)
- throws IOException {
- long count = 0;
- long unReplicatedSize = 0;
- long replicatedSize = 0;
-
- if (iterator != null) {
- while (iterator.hasNext()) {
- Table.KeyValue<String, ?> kv = iterator.next();
- if (kv != null && kv.getValue() != null) {
- if (kv.getValue() instanceof OmKeyInfo) {
- OmKeyInfo omKeyInfo = (OmKeyInfo) kv.getValue();
- unReplicatedSize += omKeyInfo.getDataSize();
- replicatedSize += omKeyInfo.getReplicatedSize();
- count++;
- }
- if (kv.getValue() instanceof RepeatedOmKeyInfo) {
- RepeatedOmKeyInfo repeatedOmKeyInfo = (RepeatedOmKeyInfo) kv
- .getValue();
- Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
- unReplicatedSize += result.getRight();
- replicatedSize += result.getLeft();
- // Since we can have multiple deleted keys of same name
- count += repeatedOmKeyInfo.getOmKeyInfoList().size();
- }
- }
- }
- }
-
- return Triple.of(count, unReplicatedSize, replicatedSize);
- }
-
/**
* Returns a collection of table names that require data size calculation.
*/
public Collection<String> getTablesToCalculateSize() {
- List<String> taskTables = new ArrayList<>();
+ java.util.List<String> taskTables = new ArrayList<>();
Review Comment:
Done!
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableHandler.java:
##########
@@ -0,0 +1,87 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+/**
+ * Interface for handling PUT, DELETE and UPDATE events for size-related
+ * tables for OM Insights.
+ */
+public interface OmTableHandler {
+
+ /**
+ * Handles a PUT event for size-related tables by updating both the data
+ * sizes and their corresponding record counts in the tables.
+ *
+ * @param event The PUT event to be processed.
+ * @param tableName Table name associated with the event.
+ * @param sizeRelatedTables Tables Requiring Size Calculation.
Review Comment:
Done
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/OMMetadataManagerTestUtils.java:
##########
@@ -397,23 +397,31 @@ public static void writeDirToOm(OMMetadataManager omMetadataManager,
.build());
}
+ @SuppressWarnings("parameternumber")
public static void writeDeletedDirToOm(OMMetadataManager omMetadataManager,
String bucketName,
String volumeName,
String dirName,
long parentObjectId,
long bucketObjectId,
- long volumeObjectId)
+ long volumeObjectId,
+ long objectId)
throws IOException {
- // DB key in DeletedDirectoryTable => "volumeID/bucketID/parentId/dirName"
- String omKey = omMetadataManager.getOzonePathKey(volumeObjectId,
- bucketObjectId, parentObjectId, dirName);
+ // DB key in DeletedDirectoryTable =>
+ // "volumeID/bucketID/parentId/dirName/dirObjectId"
+
+ String ozoneDbKey = omMetadataManager.getOzonePathKey(volumeObjectId,
+ bucketObjectId, parentObjectId, dirName);
+ String ozoneDeleteKey = omMetadataManager.getOzoneDeletePathKey(
+ objectId, ozoneDbKey);
+
Review Comment:
Done
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/TestOmTableInsightTask.java:
##########
@@ -1,40 +1,31 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- * <p>
- * http://www.apache.org/licenses/LICENSE-2.0
- * <p>
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1458486535
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/OMMetadataManagerTestUtils.java:
##########
@@ -397,23 +397,31 @@ public static void writeDirToOm(OMMetadataManager omMetadataManager,
.build());
}
+ @SuppressWarnings("parameternumber")
public static void writeDeletedDirToOm(OMMetadataManager omMetadataManager,
String bucketName,
String volumeName,
String dirName,
long parentObjectId,
long bucketObjectId,
- long volumeObjectId)
+ long volumeObjectId,
+ long objectId)
throws IOException {
- // DB key in DeletedDirectoryTable => "volumeID/bucketID/parentId/dirName"
- String omKey = omMetadataManager.getOzonePathKey(volumeObjectId,
- bucketObjectId, parentObjectId, dirName);
+ // DB key in DeletedDirectoryTable =>
+ // "volumeID/bucketID/parentId/dirName/dirObjectId"
+
+ String ozoneDbKey = omMetadataManager.getOzonePathKey(volumeObjectId,
+ bucketObjectId, parentObjectId, dirName);
+ String ozoneDeleteKey = omMetadataManager.getOzoneDeletePathKey(
+ objectId, ozoneDbKey);
+
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1469134319
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -65,14 +65,29 @@ public class OmTableInsightTask implements ReconOmTask {
private GlobalStatsDao globalStatsDao;
private Configuration sqlConfiguration;
private ReconOMMetadataManager reconOMMetadataManager;
+ private ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager;
+ private Table<Long, NSSummary> nsSummaryTable;
+ private Map<String, OmTableHandler> tableHandlers;
@Inject
public OmTableInsightTask(GlobalStatsDao globalStatsDao,
- Configuration sqlConfiguration,
- ReconOMMetadataManager reconOMMetadataManager) {
+ Configuration sqlConfiguration,
+ ReconOMMetadataManager reconOMMetadataManager,
+ ReconNamespaceSummaryManagerImpl
+ reconNamespaceSummaryManager) {
this.globalStatsDao = globalStatsDao;
this.sqlConfiguration = sqlConfiguration;
this.reconOMMetadataManager = reconOMMetadataManager;
+ this.reconNamespaceSummaryManager = reconNamespaceSummaryManager;
+ this.nsSummaryTable = reconNamespaceSummaryManager.getNSSummaryTable();
+
+ // Initialize table handlers
+ tableHandlers = new HashMap<>();
+ tableHandlers.put(OPEN_KEY_TABLE, new OpenKeysInsightHandler());
Review Comment:
Changes made!
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/DeletedDirectoriesInsightHandler.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.client.ReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.spi.impl.ReconNamespaceSummaryManagerImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.hdds.client.ReplicationConfig.parse;
+import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_REPLICATION;
+import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_REPLICATION_DEFAULT;
+
+/**
+ * Manages records in the Deleted Directory Table, updating counts and sizes of
+ * pending Directory Deletions in the backend.
+ */
+public class DeletedDirectoriesInsightHandler implements OmTableHandler {
+
+ private ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager;
+ private int replicationFactor;
+ private OzoneConfiguration conf;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DeletedKeysInsightHandler.class);
+
+ public DeletedDirectoriesInsightHandler(
+ ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager) {
+ this.conf = new OzoneConfiguration();
+ this.replicationFactor = getReplicationFactor();
+ this.reconNamespaceSummaryManager = reconNamespaceSummaryManager;
+ }
+
+ /**
+ * Invoked by the process method to add information on those directories that
+ * have been backlogged in the backend for deletion.
+ */
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ OmKeyInfo omKeyInfo = (OmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey, (k, count) -> count + 1L);
+
+ try {
+ // Attempt to fetch size for deleted directory
+ Long newDeletedDirectorySize =
+ fetchSizeForDeletedDirectory(omKeyInfo.getObjectID());
+
+ unreplicatedSizeMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size + newDeletedDirectorySize);
+ replicatedSizeMap.computeIfPresent(unReplicatedSizeKey,
Review Comment:
Changes made!
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/DeletedDirectoriesInsightHandler.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.client.ReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.spi.impl.ReconNamespaceSummaryManagerImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.hdds.client.ReplicationConfig.parse;
+import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_REPLICATION;
+import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_REPLICATION_DEFAULT;
+
+/**
+ * Manages records in the Deleted Directory Table, updating counts and sizes of
+ * pending Directory Deletions in the backend.
+ */
+public class DeletedDirectoriesInsightHandler implements OmTableHandler {
+
+ private ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager;
+ private int replicationFactor;
+ private OzoneConfiguration conf;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DeletedKeysInsightHandler.class);
+
+ public DeletedDirectoriesInsightHandler(
+ ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager) {
+ this.conf = new OzoneConfiguration();
+ this.replicationFactor = getReplicationFactor();
+ this.reconNamespaceSummaryManager = reconNamespaceSummaryManager;
+ }
+
+ /**
+ * Invoked by the process method to add information on those directories that
+ * have been backlogged in the backend for deletion.
+ */
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+ String countKey = getTableCountKeyFromTable(tableName);
Review Comment:
Changes made!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "dombizita (via GitHub)" <gi...@apache.org>.
dombizita commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1446214166
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -124,69 +139,26 @@ public Pair<String, Boolean> reprocess(OMMetadataManager omMetadataManager) {
if (!objectCountMap.isEmpty()) {
writeDataToDB(objectCountMap);
}
- if (!unReplicatedSizeCountMap.isEmpty()) {
- writeDataToDB(unReplicatedSizeCountMap);
+ if (!unReplicatedSizeMap.isEmpty()) {
+ writeDataToDB(unReplicatedSizeMap);
}
- if (!replicatedSizeCountMap.isEmpty()) {
- writeDataToDB(replicatedSizeCountMap);
+ if (!replicatedSizeMap.isEmpty()) {
+ writeDataToDB(replicatedSizeMap);
}
LOG.info("Completed a 'reprocess' run of OmTableInsightTask.");
return new ImmutablePair<>(getTaskName(), true);
}
- /**
- * Returns a triple with the total count of records (left), total unreplicated
- * size (middle), and total replicated size (right) in the given iterator.
- * Increments count for each record and adds the dataSize if a record's value
- * is an instance of OmKeyInfo. If the iterator is null, returns (0, 0, 0).
- *
- * @param iterator The iterator over the table to be iterated.
- * @return A Triple with three Long values representing the count,
- * unreplicated size and replicated size.
- * @throws IOException If an I/O error occurs during the iterator traversal.
- */
- private Triple<Long, Long, Long> getTableSizeAndCount(
- TableIterator<String, ? extends Table.KeyValue<String, ?>> iterator)
- throws IOException {
- long count = 0;
- long unReplicatedSize = 0;
- long replicatedSize = 0;
-
- if (iterator != null) {
- while (iterator.hasNext()) {
- Table.KeyValue<String, ?> kv = iterator.next();
- if (kv != null && kv.getValue() != null) {
- if (kv.getValue() instanceof OmKeyInfo) {
- OmKeyInfo omKeyInfo = (OmKeyInfo) kv.getValue();
- unReplicatedSize += omKeyInfo.getDataSize();
- replicatedSize += omKeyInfo.getReplicatedSize();
- count++;
- }
- if (kv.getValue() instanceof RepeatedOmKeyInfo) {
- RepeatedOmKeyInfo repeatedOmKeyInfo = (RepeatedOmKeyInfo) kv
- .getValue();
- Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
- unReplicatedSize += result.getRight();
- replicatedSize += result.getLeft();
- // Since we can have multiple deleted keys of same name
- count += repeatedOmKeyInfo.getOmKeyInfoList().size();
- }
- }
- }
- }
-
- return Triple.of(count, unReplicatedSize, replicatedSize);
- }
-
/**
* Returns a collection of table names that require data size calculation.
*/
public Collection<String> getTablesToCalculateSize() {
- List<String> taskTables = new ArrayList<>();
+ java.util.List<String> taskTables = new ArrayList<>();
Review Comment:
Is the package needed here?
```suggestion
List<String> taskTables = new ArrayList<>();
```
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/OMMetadataManagerTestUtils.java:
##########
@@ -397,23 +397,31 @@ public static void writeDirToOm(OMMetadataManager omMetadataManager,
.build());
}
+ @SuppressWarnings("parameternumber")
public static void writeDeletedDirToOm(OMMetadataManager omMetadataManager,
String bucketName,
String volumeName,
String dirName,
long parentObjectId,
long bucketObjectId,
- long volumeObjectId)
+ long volumeObjectId,
+ long objectId)
throws IOException {
- // DB key in DeletedDirectoryTable => "volumeID/bucketID/parentId/dirName"
- String omKey = omMetadataManager.getOzonePathKey(volumeObjectId,
- bucketObjectId, parentObjectId, dirName);
+ // DB key in DeletedDirectoryTable =>
+ // "volumeID/bucketID/parentId/dirName/dirObjectId"
+
+ String ozoneDbKey = omMetadataManager.getOzonePathKey(volumeObjectId,
+ bucketObjectId, parentObjectId, dirName);
+ String ozoneDeleteKey = omMetadataManager.getOzoneDeletePathKey(
+ objectId, ozoneDbKey);
+
Review Comment:
What caused this change? Why did the DB key change? Or this was incorrect before?
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/TestOmTableInsightTask.java:
##########
@@ -1,40 +1,31 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- * <p>
- * http://www.apache.org/licenses/LICENSE-2.0
- * <p>
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
Review Comment:
This shouldn't be removed.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableHandler.java:
##########
@@ -0,0 +1,87 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+/**
+ * Interface for handling PUT, DELETE and UPDATE events for size-related
+ * tables for OM Insights.
+ */
+public interface OmTableHandler {
+
+ /**
+ * Handles a PUT event for size-related tables by updating both the data
+ * sizes and their corresponding record counts in the tables.
+ *
+ * @param event The PUT event to be processed.
+ * @param tableName Table name associated with the event.
+ * @param sizeRelatedTables Tables Requiring Size Calculation.
Review Comment:
Maybe I missed something, but I don't see any usages why we should pass the `sizeRelatedTables` to the `handlePutEvent()`, `handleDeleteEvent()` and to the `handleUpdateEvent()`. Can you check this out? I didn't see usages in any of the `OmTableHandler` implementations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1471462253
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -65,14 +63,27 @@ public class OmTableInsightTask implements ReconOmTask {
private GlobalStatsDao globalStatsDao;
private Configuration sqlConfiguration;
private ReconOMMetadataManager reconOMMetadataManager;
+ private ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager;
+ private Table<Long, NSSummary> nsSummaryTable;
+ private Map<String, OmTableHandler> tableHandlers;
@Inject
public OmTableInsightTask(GlobalStatsDao globalStatsDao,
- Configuration sqlConfiguration,
- ReconOMMetadataManager reconOMMetadataManager) {
+ Configuration sqlConfiguration,
+ ReconOMMetadataManager reconOMMetadataManager,
+ ReconNamespaceSummaryManagerImpl
+ reconNamespaceSummaryManager) {
this.globalStatsDao = globalStatsDao;
this.sqlConfiguration = sqlConfiguration;
this.reconOMMetadataManager = reconOMMetadataManager;
+ this.reconNamespaceSummaryManager = reconNamespaceSummaryManager;
Review Comment:
Done!
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -65,14 +63,27 @@ public class OmTableInsightTask implements ReconOmTask {
private GlobalStatsDao globalStatsDao;
private Configuration sqlConfiguration;
private ReconOMMetadataManager reconOMMetadataManager;
+ private ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager;
+ private Table<Long, NSSummary> nsSummaryTable;
Review Comment:
Done!
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -65,14 +63,27 @@ public class OmTableInsightTask implements ReconOmTask {
private GlobalStatsDao globalStatsDao;
private Configuration sqlConfiguration;
private ReconOMMetadataManager reconOMMetadataManager;
+ private ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager;
Review Comment:
Done!
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/DeletedKeysInsightHandler.java:
##########
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.RepeatedOmKeyInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+
+/**
+ * Manages records in the Deleted Table, updating counts and sizes of
+ * pending Key Deletions in the backend.
+ */
+public class DeletedKeysInsightHandler implements OmTableHandler {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DeletedKeysInsightHandler.class);
+
+ /**
+ * Invoked by the process method to add information on those keys that have
+ * been backlogged in the backend for deletion.
+ */
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo =
+ (RepeatedOmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey,
+ (k, count) -> count + repeatedOmKeyInfo.getOmKeyInfoList().size());
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unReplicatedSizeMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size + result.getLeft());
+ replicatedSizeMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size + result.getRight());
+ } else {
+ LOG.warn("Put event does not have the Key Info for {}.",
+ event.getKey());
+ }
+
+ }
+
+ /**
+ * Invoked by the process method to remove information on those keys that have
+ * been successfully deleted from the backend.
+ */
+ @Override
+ public void handleDeleteEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo =
+ (RepeatedOmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey, (k, count) ->
+ count > 0 ? count - repeatedOmKeyInfo.getOmKeyInfoList().size() : 0L);
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unReplicatedSizeMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size > result.getLeft() ? size - result.getLeft() : 0L);
+ replicatedSizeMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size > result.getRight() ? size - result.getRight() :
+ 0L);
+ } else {
+ LOG.warn("Delete event does not have the Key Info for {}.",
+ event.getKey());
+ }
+ }
+
+ /**
+ * Invoked by the process method to update the statistics on the keys
+ * pending to be deleted.
+ */
+ @Override
+ public void handleUpdateEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+ // The size of deleted keys cannot change hence no-op.
+ return;
+ }
+
+ /**
+ * Invoked by the reprocess method to calculate the records count of the
+ * deleted table and the sizes of replicated and unreplicated keys that are
+ * pending deletion in Ozone.
+ */
+ @Override
+ public Triple<Long, Long, Long> getTableSizeAndCount(
+ TableIterator<String, ? extends Table.KeyValue<String, ?>> iterator)
+ throws IOException {
+ long count = 0;
+ long unReplicatedSize = 0;
+ long replicatedSize = 0;
+
+ if (iterator != null) {
+ while (iterator.hasNext()) {
+ Table.KeyValue<String, ?> kv = iterator.next();
+ if (kv != null && kv.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo = (RepeatedOmKeyInfo) kv
+ .getValue();
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unReplicatedSize += result.getRight();
+ replicatedSize += result.getLeft();
+ count += repeatedOmKeyInfo.getOmKeyInfoList().size();
+ }
+ }
+ }
+ return Triple.of(count, unReplicatedSize, replicatedSize);
+ }
+
+ public static String getTableCountKeyFromTable(String tableName) {
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1429629758
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableHandler.java:
##########
@@ -0,0 +1,89 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+/**
+ * Interface for handling PUT, DELETE and UPDATE events for size-related
+ * tables for OM Insights.
+ */
+public interface OmTableHandler {
+
+ /**
+ * Handles a PUT event for size-related tables by updating both the data
+ * sizes and their corresponding record counts in the tables.
+ *
+ * @param event The PUT event to be processed.
+ * @param tableName Table name associated with the event.
+ * @param sizeRelatedTables Tables Requiring Size Calculation.
+ * @param objectCountMap A map storing object counts.
+ * @param unReplicatedSizeMap A map storing unReplicated size counts.
+ * @param replicatedSizeMap A map storing replicated size counts.
+ * @throws IOException If an I/O error occurs during processing.
+ */
+ void handlePutEvent(OMDBUpdateEvent<String, Object> event,
Review Comment:
Done!
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableHandler.java:
##########
@@ -0,0 +1,89 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+/**
+ * Interface for handling PUT, DELETE and UPDATE events for size-related
+ * tables for OM Insights.
+ */
+public interface OmTableHandler {
+
+ /**
+ * Handles a PUT event for size-related tables by updating both the data
+ * sizes and their corresponding record counts in the tables.
+ *
+ * @param event The PUT event to be processed.
+ * @param tableName Table name associated with the event.
+ * @param sizeRelatedTables Tables Requiring Size Calculation.
+ * @param objectCountMap A map storing object counts.
+ * @param unReplicatedSizeMap A map storing unReplicated size counts.
+ * @param replicatedSizeMap A map storing replicated size counts.
+ * @throws IOException If an I/O error occurs during processing.
+ */
+ void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap);
+ /**
+ * Handles a DELETE event for size-related tables by updating both the data
+ * sizes and their corresponding record counts in the tables.
+ *
+ * @param event The DELETE event to be processed.
+ * @param tableName Table name associated with the event.
+ * @param sizeRelatedTables Tables Requiring Size Calculation.
+ * @param objectCountMap A map storing object counts.
+ * @param unReplicatedSizeMap A map storing unReplicated size counts.
+ * @param replicatedSizeMap A map storing replicated size counts.
+ * @throws IOException If an I/O error occurs during processing.
+ */
+ void handleDeleteEvent(OMDBUpdateEvent<String, Object> event,
Review Comment:
Done!
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/DeletedKeysInsightHandler.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.RepeatedOmKeyInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+/**
+ * Manages records in the Deleted Table, updating counts and sizes of
+ * pending Key Deletions in the backend.
+ */
+public class DeletedKeysInsightHandler implements OmTableHandler {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DeletedKeysInsightHandler.class);
+
+ /**
+ * Invoked by the process method to add information on those keys that have
+ * been backlogged in the backend for deletion.
+ */
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo =
+ (RepeatedOmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey,
+ (k, count) -> count + repeatedOmKeyInfo.getOmKeyInfoList().size());
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unReplicatedSizeMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size + result.getLeft());
+ replicatedSizeMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size + result.getRight());
+ } else {
+ LOG.warn("Put event does not have the Key Info for {}.",
+ event.getKey());
+ }
+
+ }
+
+ /**
+ * Invoked by the process method to remove information on those keys that have
+ * been successfully deleted from the backend.
+ */
+ @Override
+ public void handleDeleteEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo =
+ (RepeatedOmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey, (k, count) ->
+ count > 0 ? count - repeatedOmKeyInfo.getOmKeyInfoList().size() : 0L);
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unReplicatedSizeMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size > result.getLeft() ? size - result.getLeft() : 0L);
+ replicatedSizeMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size > result.getRight() ? size - result.getRight() :
+ 0L);
+ } else {
+ LOG.warn("Delete event does not have the Key Info for {}.",
+ event.getKey());
+ }
+ }
+
+ /**
+ * Invoked by the process method to update the statistics on the keys
+ * pending to be deleted.
+ */
+ @Override
+ public void handleUpdateEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+ // The size of deleted directories cannot change hence no-op.
Review Comment:
Done!
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/DeletedKeysInsightHandler.java:
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.RepeatedOmKeyInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+/**
+ * Manages records in the Deleted Table, updating counts and sizes of
+ * pending Key Deletions in the backend.
+ */
+public class DeletedKeysInsightHandler implements OmTableHandler {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DeletedKeysInsightHandler.class);
+
+ /**
+ * Invoked by the process method to add information on those keys that have
+ * been backlogged in the backend for deletion.
+ */
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo =
+ (RepeatedOmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey,
+ (k, count) -> count + repeatedOmKeyInfo.getOmKeyInfoList().size());
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unReplicatedSizeMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size + result.getLeft());
+ replicatedSizeMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size + result.getRight());
+ } else {
+ LOG.warn("Put event does not have the Key Info for {}.",
+ event.getKey());
+ }
+
+ }
+
+ /**
+ * Invoked by the process method to remove information on those keys that have
+ * been successfully deleted from the backend.
+ */
+ @Override
+ public void handleDeleteEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo =
+ (RepeatedOmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey, (k, count) ->
+ count > 0 ? count - repeatedOmKeyInfo.getOmKeyInfoList().size() : 0L);
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unReplicatedSizeMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size > result.getLeft() ? size - result.getLeft() : 0L);
+ replicatedSizeMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size > result.getRight() ? size - result.getRight() :
+ 0L);
+ } else {
+ LOG.warn("Delete event does not have the Key Info for {}.",
+ event.getKey());
+ }
+ }
+
+ /**
+ * Invoked by the process method to update the statistics on the keys
+ * pending to be deleted.
+ */
+ @Override
+ public void handleUpdateEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+ // The size of deleted directories cannot change hence no-op.
+ return;
+ }
+
+ /**
+ * Invoked by the reprocess method to calculate the records count of the
+ * deleted table and the sizes of replicated and unreplicated keys that are
+ * pending deletion in Ozone.
+ */
+ @Override
+ public Triple<Long, Long, Long> getTableSizeAndCount(
+ TableIterator<String, ? extends Table.KeyValue<String, ?>> iterator)
+ throws IOException {
+ long count = 0;
+ long unReplicatedSize = 0;
+ long replicatedSize = 0;
+
+ if (iterator != null) {
+ while (iterator.hasNext()) {
+ Table.KeyValue<String, ?> kv = iterator.next();
+ if (kv != null && kv.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo = (RepeatedOmKeyInfo) kv
+ .getValue();
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unReplicatedSize += result.getRight();
+ replicatedSize += result.getLeft();
+ // Since we can have multiple deleted keys of same name
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1415772804
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/DeletedDirectoryTableHandler.java:
##########
@@ -0,0 +1,165 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.spi.impl.ReconNamespaceSummaryManagerImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Manages records in the Deleted Directory Table, updating counts and sizes of
+ * pending Directory Deletions in the backend.
+ */
+public class DeletedDirectoryTableHandler implements OmTableHandler {
+
+ private ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DeletedTableHandler.class);
+
+ public DeletedDirectoryTableHandler(
+ ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager) {
+ this.reconNamespaceSummaryManager = reconNamespaceSummaryManager;
+ }
+
+ /**
+ * Invoked by the process method to add information on those directories that
+ * have been backlogged in the backend for deletion.
+ */
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeCountMap,
+ HashMap<String, Long> replicatedSizeCountMap)
+ throws IOException {
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ OmKeyInfo omKeyInfo = (OmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey, (k, count) -> count + 1L);
+ Long newDeletedDirectorySize =
+ fetchSizeForDeletedDirectory(omKeyInfo.getObjectID());
+ unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
Review Comment:
Thanks I have added a method that fetches the replication number from the configs.
We can use it to multiply the unReplicated data present.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1415770727
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/DeletedTableHandler.java:
##########
@@ -0,0 +1,176 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.RepeatedOmKeyInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+/**
+ * Manages records in the Deleted Table, updating counts and sizes of
+ * pending Key Deletions in the backend.
+ */
+public class DeletedTableHandler implements OmTableHandler {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DeletedTableHandler.class);
+
+ /**
+ * Invoked by the process method to add information on those keys that have
+ * been backlogged in the backend for deletion.
+ */
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeCountMap,
+ HashMap<String, Long> replicatedSizeCountMap)
+ throws IOException {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo =
+ (RepeatedOmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey,
+ (k, count) -> count + repeatedOmKeyInfo.getOmKeyInfoList().size());
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size + result.getLeft());
+ replicatedSizeCountMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size + result.getRight());
+ } else {
+ LOG.warn("Put event does not have the Key Info for {}.",
+ event.getKey());
+ }
+
+ }
+
+ /**
+ * Invoked by the process method to remove information on those keys that have
+ * been successfully deleted from the backend.
+ */
+ @Override
+ public void handleDeleteEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeCountMap,
+ HashMap<String, Long> replicatedSizeCountMap)
+ throws IOException {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo =
+ (RepeatedOmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey, (k, count) ->
+ count > 0 ? count - repeatedOmKeyInfo.getOmKeyInfoList().size() : 0L);
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size > result.getLeft() ? size - result.getLeft() : 0L);
+ replicatedSizeCountMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size > result.getRight() ? size - result.getRight() :
+ 0L);
+ } else {
+ LOG.warn("Delete event does not have the Key Info for {}.",
+ event.getKey());
+ }
+ }
+
+ /**
+ * Invoked by the process method to update the statistics on the keys
+ * pending to be deleted.
+ */
+ @Override
+ public void handleUpdateEvent(OMDBUpdateEvent<String, Object> event,
Review Comment:
The original idea at the time was to add an `update` event handler for deleted tables because the size of the `keyInfoList` could change when new keys with similar names were deleted. This change would then impact the overall count and size of the deleted keys.
But since we are no longer storing duplicate keys in `repeatedOmKeyInfo` we do not require any update event method for DeletedKeysInsightHandler. Hence I have removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1415785080
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OpenKeyTableHandler.java:
##########
@@ -0,0 +1,141 @@
+package org.apache.hadoop.ozone.recon.tasks;
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1446967862
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/DeletedDirectoriesInsightHandler.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.client.ReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.spi.impl.ReconNamespaceSummaryManagerImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.hdds.client.ReplicationConfig.parse;
+import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_REPLICATION;
+import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_REPLICATION_DEFAULT;
+
+/**
+ * Manages records in the Deleted Directory Table, updating counts and sizes of
+ * pending Directory Deletions in the backend.
+ */
+public class DeletedDirectoriesInsightHandler implements OmTableHandler {
+
+ private ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager;
+ private int replicationFactor;
+ private OzoneConfiguration conf;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DeletedKeysInsightHandler.class);
+
+ public DeletedDirectoriesInsightHandler(
+ ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager) {
+ this.conf = new OzoneConfiguration();
+ this.replicationFactor = getReplicationFactor();
+ this.reconNamespaceSummaryManager = reconNamespaceSummaryManager;
+ }
+
+ /**
+ * Invoked by the process method to add information on those directories that
+ * have been backlogged in the backend for deletion.
+ */
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ OmKeyInfo omKeyInfo = (OmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey, (k, count) -> count + 1L);
+
+ try {
+ // Attempt to fetch size for deleted directory
+ Long newDeletedDirectorySize =
+ fetchSizeForDeletedDirectory(omKeyInfo.getObjectID());
+
+ unreplicatedSizeMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size + newDeletedDirectorySize);
+ replicatedSizeMap.computeIfPresent(unReplicatedSizeKey,
Review Comment:
size can be different based on key type as EC or Ratis, so can not have direct multiplication as of ratis. Further a directory can have file of different replication type, can not take decision based default config.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -65,14 +65,29 @@ public class OmTableInsightTask implements ReconOmTask {
private GlobalStatsDao globalStatsDao;
private Configuration sqlConfiguration;
private ReconOMMetadataManager reconOMMetadataManager;
+ private ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager;
+ private Table<Long, NSSummary> nsSummaryTable;
+ private Map<String, OmTableHandler> tableHandlers;
@Inject
public OmTableInsightTask(GlobalStatsDao globalStatsDao,
- Configuration sqlConfiguration,
- ReconOMMetadataManager reconOMMetadataManager) {
+ Configuration sqlConfiguration,
+ ReconOMMetadataManager reconOMMetadataManager,
+ ReconNamespaceSummaryManagerImpl
+ reconNamespaceSummaryManager) {
this.globalStatsDao = globalStatsDao;
this.sqlConfiguration = sqlConfiguration;
this.reconOMMetadataManager = reconOMMetadataManager;
+ this.reconNamespaceSummaryManager = reconNamespaceSummaryManager;
+ this.nsSummaryTable = reconNamespaceSummaryManager.getNSSummaryTable();
+
+ // Initialize table handlers
+ tableHandlers = new HashMap<>();
+ tableHandlers.put(OPEN_KEY_TABLE, new OpenKeysInsightHandler());
Review Comment:
getTablesToCalculateSize() can be removed and use tableHandlers only to avoid mismatch while iterate taskTable and using tableHandler to get data.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/DeletedDirectoriesInsightHandler.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.client.ReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.recon.api.types.NSSummary;
+import org.apache.hadoop.ozone.recon.spi.impl.ReconNamespaceSummaryManagerImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.hdds.client.ReplicationConfig.parse;
+import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_REPLICATION;
+import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_REPLICATION_DEFAULT;
+
+/**
+ * Manages records in the Deleted Directory Table, updating counts and sizes of
+ * pending Directory Deletions in the backend.
+ */
+public class DeletedDirectoriesInsightHandler implements OmTableHandler {
+
+ private ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager;
+ private int replicationFactor;
+ private OzoneConfiguration conf;
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DeletedKeysInsightHandler.class);
+
+ public DeletedDirectoriesInsightHandler(
+ ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager) {
+ this.conf = new OzoneConfiguration();
+ this.replicationFactor = getReplicationFactor();
+ this.reconNamespaceSummaryManager = reconNamespaceSummaryManager;
+ }
+
+ /**
+ * Invoked by the process method to add information on those directories that
+ * have been backlogged in the backend for deletion.
+ */
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ Collection<String> sizeRelatedTables,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unreplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+ String countKey = getTableCountKeyFromTable(tableName);
Review Comment:
There are 2 problem which will not give correct result:
1. duplicate counting of file size if files are present in sub-directory
2. size will not get dyanmically updated if some files of directory is deleted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1471802060
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -65,14 +65,29 @@ public class OmTableInsightTask implements ReconOmTask {
private GlobalStatsDao globalStatsDao;
private Configuration sqlConfiguration;
private ReconOMMetadataManager reconOMMetadataManager;
+ private ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager;
+ private Table<Long, NSSummary> nsSummaryTable;
+ private Map<String, OmTableHandler> tableHandlers;
@Inject
public OmTableInsightTask(GlobalStatsDao globalStatsDao,
- Configuration sqlConfiguration,
- ReconOMMetadataManager reconOMMetadataManager) {
+ Configuration sqlConfiguration,
+ ReconOMMetadataManager reconOMMetadataManager,
+ ReconNamespaceSummaryManagerImpl
+ reconNamespaceSummaryManager) {
this.globalStatsDao = globalStatsDao;
this.sqlConfiguration = sqlConfiguration;
this.reconOMMetadataManager = reconOMMetadataManager;
+ this.reconNamespaceSummaryManager = reconNamespaceSummaryManager;
+ this.nsSummaryTable = reconNamespaceSummaryManager.getNSSummaryTable();
+
+ // Initialize table handlers
+ tableHandlers = new HashMap<>();
+ tableHandlers.put(OPEN_KEY_TABLE, new OpenKeysInsightHandler());
Review Comment:
Thats a great suggestion thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories. [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1469207734
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/ReconNamespaceSummaryManagerImpl.java:
##########
@@ -84,4 +85,9 @@ public void commitBatchOperation(RDBBatchOperation rdbBatchOperation)
public Table getNSSummaryTable() {
return nsSummaryTable;
}
+
+ @VisibleForTesting
+ public void setNsSummaryTable(Table<Long, NSSummary> nsSummaryTable) {
Review Comment:
Do not see any usages for this method, pls check once if it is needed ?
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/OMDBInsightEndpoint.java:
##########
@@ -651,6 +652,36 @@ public Response getDeletedDirInfo(
return Response.ok(deletedDirInsightInfo).build();
}
+ /**
+ * Retrieves the summary of deleted directories.
+ *
+ * This method calculates and returns a summary of deleted directories.
+ * @return The HTTP response body includes a map with the following entries:
+ * - "totalDeletedDirectories": the total number of deleted directories
+ *
+ * Example response:
+ * {
+ * "totalDeletedDirectories": 8,
+ * }
+ */
+ @GET
+ @Path("/deletePending/dirs/summary")
+ public Response getDeletedDirectorySummary() throws IOException {
Review Comment:
@ArafatKhan2198 thanks for continued effort on this patch, IMO, we should sum based on all keys included in these `totalDeletedDirectories` and include total replicated and un-replicated size as well in summary, else it will be difficult for end user/admin to manually sum up size of all keys we are showing in deletePending dir details page.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -270,63 +236,33 @@ private void handlePutEvent(OMDBUpdateEvent<String, Object> event,
String tableName,
Collection<String> sizeRelatedTables,
HashMap<String, Long> objectCountMap,
- HashMap<String, Long> unreplicatedSizeCountMap,
- HashMap<String, Long> replicatedSizeCountMap) {
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap)
+ throws IOException {
+ OmTableHandler tableHandler = tableHandlers.get(tableName);
- if (sizeRelatedTables.contains(tableName)) {
- handleSizeRelatedTablePutEvent(event, tableName, objectCountMap,
- unreplicatedSizeCountMap, replicatedSizeCountMap);
+ if (sizeRelatedTables.contains(tableName) && tableHandler != null) {
+ tableHandler.handlePutEvent(event, tableName, objectCountMap,
+ unReplicatedSizeMap, replicatedSizeMap);
} else {
String countKey = getTableCountKeyFromTable(tableName);
objectCountMap.computeIfPresent(countKey, (k, count) -> count + 1L);
}
}
- private void handleSizeRelatedTablePutEvent(
- OMDBUpdateEvent<String, Object> event,
- String tableName,
- HashMap<String, Long> objectCountMap,
- HashMap<String, Long> unreplicatedSizeCountMap,
- HashMap<String, Long> replicatedSizeCountMap) {
-
- String countKey = getTableCountKeyFromTable(tableName);
- String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
- String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
-
- if (event.getValue() instanceof OmKeyInfo) {
- // Handle PUT for OpenKeyTable & OpenFileTable
- OmKeyInfo omKeyInfo = (OmKeyInfo) event.getValue();
- objectCountMap.computeIfPresent(countKey, (k, count) -> count + 1L);
- unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
- (k, size) -> size + omKeyInfo.getDataSize());
- replicatedSizeCountMap.computeIfPresent(replicatedSizeKey,
- (k, size) -> size + omKeyInfo.getReplicatedSize());
- } else if (event.getValue() instanceof RepeatedOmKeyInfo) {
- // Handle PUT for DeletedTable
- RepeatedOmKeyInfo repeatedOmKeyInfo =
- (RepeatedOmKeyInfo) event.getValue();
- objectCountMap.computeIfPresent(countKey,
- (k, count) -> count + repeatedOmKeyInfo.getOmKeyInfoList().size());
- Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
- unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
- (k, size) -> size + result.getLeft());
- replicatedSizeCountMap.computeIfPresent(replicatedSizeKey,
- (k, size) -> size + result.getRight());
- }
- }
-
-
private void handleDeleteEvent(OMDBUpdateEvent<String, Object> event,
String tableName,
Collection<String> sizeRelatedTables,
HashMap<String, Long> objectCountMap,
- HashMap<String, Long> unreplicatedSizeCountMap,
- HashMap<String, Long> replicatedSizeCountMap) {
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap)
+ throws IOException {
+ OmTableHandler tableHandler = tableHandlers.get(tableName);
if (event.getValue() != null) {
if (sizeRelatedTables.contains(tableName)) {
Review Comment:
Any reason `tableHandler` null check is not done here as being done in `handlePutEvent` ?
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OpenKeysInsightHandler.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+
+/**
+ * Manages records in the OpenKey Table, updating counts and sizes of
+ * open keys in the backend.
+ */
+public class OpenKeysInsightHandler implements OmTableHandler {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OpenKeysInsightHandler.class);
+
+ /**
+ * Invoked by the process method to add information on those keys that have
+ * been open in the backend.
+ */
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ OmKeyInfo omKeyInfo = (OmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey, (k, count) -> count + 1L);
+ unReplicatedSizeMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size + omKeyInfo.getDataSize());
+ replicatedSizeMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size + omKeyInfo.getReplicatedSize());
+ } else {
+ LOG.warn("Put event does not have the Key Info for {}.",
+ event.getKey());
+ }
+ }
+
+ /**
+ * Invoked by the process method to delete information on those keys that are
+ * no longer closed in the backend.
+ */
+ @Override
+ public void handleDeleteEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ OmKeyInfo omKeyInfo = (OmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey,
+ (k, count) -> count > 0 ? count - 1L : 0L);
+ unReplicatedSizeMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size > omKeyInfo.getDataSize() ?
+ size - omKeyInfo.getDataSize() : 0L);
+ replicatedSizeMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size > omKeyInfo.getReplicatedSize() ?
+ size - omKeyInfo.getReplicatedSize() : 0L);
+ } else {
+ LOG.warn("Delete event does not have the Key Info for {}.",
+ event.getKey());
+ }
+ }
+
+ /**
+ * Invoked by the process method to update information on those open keys that
+ * have been updated in the backend.
+ */
+ @Override
+ public void handleUpdateEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+
+ if (event.getValue() != null) {
+ if (event.getOldValue() == null) {
+ LOG.warn("Update event does not have the old Key Info for {}.",
+ event.getKey());
+ return;
+ }
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ // In Update event the count for the open table will not change. So we
+ // don't need to update the count.
+ OmKeyInfo oldKeyInfo = (OmKeyInfo) event.getOldValue();
+ OmKeyInfo newKeyInfo = (OmKeyInfo) event.getValue();
+ unReplicatedSizeMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size - oldKeyInfo.getDataSize() +
+ newKeyInfo.getDataSize());
+ replicatedSizeMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size - oldKeyInfo.getReplicatedSize() +
+ newKeyInfo.getReplicatedSize());
+ } else {
+ LOG.warn("Update event does not have the Key Info for {}.",
+ event.getKey());
+ }
+ }
+
+ /**
+ * This method is called by the reprocess method. It calculates the record
+ * counts for both the open key table and the open file table. Additionally,
+ * it computes the sizes of both replicated and unreplicated keys
+ * that are currently open in the backend.
+ */
+ @Override
+ public Triple<Long, Long, Long> getTableSizeAndCount(
+ TableIterator<String, ? extends Table.KeyValue<String, ?>> iterator)
+ throws IOException {
+ long count = 0;
+ long unReplicatedSize = 0;
+ long replicatedSize = 0;
+
+ if (iterator != null) {
+ while (iterator.hasNext()) {
+ Table.KeyValue<String, ?> kv = iterator.next();
+ if (kv != null && kv.getValue() != null) {
+ OmKeyInfo omKeyInfo = (OmKeyInfo) kv.getValue();
+ unReplicatedSize += omKeyInfo.getDataSize();
+ replicatedSize += omKeyInfo.getReplicatedSize();
+ count++;
+ }
+ }
+ }
+ return Triple.of(count, unReplicatedSize, replicatedSize);
+ }
+
+ public static String getTableCountKeyFromTable(String tableName) {
Review Comment:
Same comment as above given for `DeleteKeyInsightHandler`
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -65,14 +63,27 @@ public class OmTableInsightTask implements ReconOmTask {
private GlobalStatsDao globalStatsDao;
private Configuration sqlConfiguration;
private ReconOMMetadataManager reconOMMetadataManager;
+ private ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager;
+ private Table<Long, NSSummary> nsSummaryTable;
Review Comment:
Check if we are still using this `nsSummaryTable`
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/OMDBInsightEndpoint.java:
##########
@@ -651,6 +652,36 @@ public Response getDeletedDirInfo(
return Response.ok(deletedDirInsightInfo).build();
}
+ /**
+ * Retrieves the summary of deleted directories.
+ *
+ * This method calculates and returns a summary of deleted directories.
+ * @return The HTTP response body includes a map with the following entries:
+ * - "totalDeletedDirectories": the total number of deleted directories
+ *
+ * Example response:
+ * {
+ * "totalDeletedDirectories": 8,
+ * }
+ */
+ @GET
+ @Path("/deletePending/dirs/summary")
+ public Response getDeletedDirectorySummary() throws IOException {
Review Comment:
I think this method doesn't throw `IOException`, so not sure why we would need throw clause. Pls check once.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -65,14 +63,27 @@ public class OmTableInsightTask implements ReconOmTask {
private GlobalStatsDao globalStatsDao;
private Configuration sqlConfiguration;
private ReconOMMetadataManager reconOMMetadataManager;
+ private ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager;
+ private Table<Long, NSSummary> nsSummaryTable;
+ private Map<String, OmTableHandler> tableHandlers;
@Inject
public OmTableInsightTask(GlobalStatsDao globalStatsDao,
- Configuration sqlConfiguration,
- ReconOMMetadataManager reconOMMetadataManager) {
+ Configuration sqlConfiguration,
+ ReconOMMetadataManager reconOMMetadataManager,
+ ReconNamespaceSummaryManagerImpl
+ reconNamespaceSummaryManager) {
this.globalStatsDao = globalStatsDao;
this.sqlConfiguration = sqlConfiguration;
this.reconOMMetadataManager = reconOMMetadataManager;
+ this.reconNamespaceSummaryManager = reconNamespaceSummaryManager;
Review Comment:
Don't see any usage. Did I miss anything ?
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -65,14 +63,27 @@ public class OmTableInsightTask implements ReconOmTask {
private GlobalStatsDao globalStatsDao;
private Configuration sqlConfiguration;
private ReconOMMetadataManager reconOMMetadataManager;
+ private ReconNamespaceSummaryManagerImpl reconNamespaceSummaryManager;
Review Comment:
Check if we are still using this `reconNamespaceSummaryManager`
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestEndpoints.java:
##########
@@ -263,6 +265,8 @@ private void initializeInjector() throws Exception {
reconScm = (ReconStorageContainerManagerFacade)
reconTestInjector.getInstance(OzoneStorageContainerManager.class);
+ reconNamespaceSummaryManager = reconTestInjector.getInstance(
Review Comment:
Check if you need this
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/DeletedKeysInsightHandler.java:
##########
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.helpers.RepeatedOmKeyInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+
+/**
+ * Manages records in the Deleted Table, updating counts and sizes of
+ * pending Key Deletions in the backend.
+ */
+public class DeletedKeysInsightHandler implements OmTableHandler {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DeletedKeysInsightHandler.class);
+
+ /**
+ * Invoked by the process method to add information on those keys that have
+ * been backlogged in the backend for deletion.
+ */
+ @Override
+ public void handlePutEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo =
+ (RepeatedOmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey,
+ (k, count) -> count + repeatedOmKeyInfo.getOmKeyInfoList().size());
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unReplicatedSizeMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size + result.getLeft());
+ replicatedSizeMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size + result.getRight());
+ } else {
+ LOG.warn("Put event does not have the Key Info for {}.",
+ event.getKey());
+ }
+
+ }
+
+ /**
+ * Invoked by the process method to remove information on those keys that have
+ * been successfully deleted from the backend.
+ */
+ @Override
+ public void handleDeleteEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+
+ String countKey = getTableCountKeyFromTable(tableName);
+ String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
+ String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
+
+ if (event.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo =
+ (RepeatedOmKeyInfo) event.getValue();
+ objectCountMap.computeIfPresent(countKey, (k, count) ->
+ count > 0 ? count - repeatedOmKeyInfo.getOmKeyInfoList().size() : 0L);
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unReplicatedSizeMap.computeIfPresent(unReplicatedSizeKey,
+ (k, size) -> size > result.getLeft() ? size - result.getLeft() : 0L);
+ replicatedSizeMap.computeIfPresent(replicatedSizeKey,
+ (k, size) -> size > result.getRight() ? size - result.getRight() :
+ 0L);
+ } else {
+ LOG.warn("Delete event does not have the Key Info for {}.",
+ event.getKey());
+ }
+ }
+
+ /**
+ * Invoked by the process method to update the statistics on the keys
+ * pending to be deleted.
+ */
+ @Override
+ public void handleUpdateEvent(OMDBUpdateEvent<String, Object> event,
+ String tableName,
+ HashMap<String, Long> objectCountMap,
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+ // The size of deleted keys cannot change hence no-op.
+ return;
+ }
+
+ /**
+ * Invoked by the reprocess method to calculate the records count of the
+ * deleted table and the sizes of replicated and unreplicated keys that are
+ * pending deletion in Ozone.
+ */
+ @Override
+ public Triple<Long, Long, Long> getTableSizeAndCount(
+ TableIterator<String, ? extends Table.KeyValue<String, ?>> iterator)
+ throws IOException {
+ long count = 0;
+ long unReplicatedSize = 0;
+ long replicatedSize = 0;
+
+ if (iterator != null) {
+ while (iterator.hasNext()) {
+ Table.KeyValue<String, ?> kv = iterator.next();
+ if (kv != null && kv.getValue() != null) {
+ RepeatedOmKeyInfo repeatedOmKeyInfo = (RepeatedOmKeyInfo) kv
+ .getValue();
+ Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
+ unReplicatedSize += result.getRight();
+ replicatedSize += result.getLeft();
+ count += repeatedOmKeyInfo.getOmKeyInfoList().size();
+ }
+ }
+ }
+ return Triple.of(count, unReplicatedSize, replicatedSize);
+ }
+
+ public static String getTableCountKeyFromTable(String tableName) {
Review Comment:
Can we move these 3 methods in `OmTableHandler` as default implementation ?
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -335,109 +271,33 @@ private void handleDeleteEvent(OMDBUpdateEvent<String, Object> event,
}
}
- private void handleSizeRelatedTableDeleteEvent(
- OMDBUpdateEvent<String, Object> event,
- String tableName,
- HashMap<String, Long> objectCountMap,
- HashMap<String, Long> unreplicatedSizeCountMap,
- HashMap<String, Long> replicatedSizeCountMap) {
-
- String countKey = getTableCountKeyFromTable(tableName);
- String unReplicatedSizeKey = getUnReplicatedSizeKeyFromTable(tableName);
- String replicatedSizeKey = getReplicatedSizeKeyFromTable(tableName);
-
- if (event.getValue() instanceof OmKeyInfo) {
- // Handle DELETE for OpenKeyTable & OpenFileTable
- OmKeyInfo omKeyInfo = (OmKeyInfo) event.getValue();
- objectCountMap.computeIfPresent(countKey,
- (k, count) -> count > 0 ? count - 1L : 0L);
- unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
- (k, size) -> size > omKeyInfo.getDataSize() ?
- size - omKeyInfo.getDataSize() : 0L);
- replicatedSizeCountMap.computeIfPresent(replicatedSizeKey,
- (k, size) -> size > omKeyInfo.getReplicatedSize() ?
- size - omKeyInfo.getReplicatedSize() : 0L);
- } else if (event.getValue() instanceof RepeatedOmKeyInfo) {
- // Handle DELETE for DeletedTable
- RepeatedOmKeyInfo repeatedOmKeyInfo =
- (RepeatedOmKeyInfo) event.getValue();
- objectCountMap.computeIfPresent(countKey, (k, count) -> count > 0 ?
- count - repeatedOmKeyInfo.getOmKeyInfoList().size() : 0L);
- Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
- unreplicatedSizeCountMap.computeIfPresent(unReplicatedSizeKey,
- (k, size) -> size > result.getLeft() ? size - result.getLeft() : 0L);
- replicatedSizeCountMap.computeIfPresent(replicatedSizeKey,
- (k, size) -> size > result.getRight() ? size - result.getRight() :
- 0L);
- }
- }
+ /**
+ * Handle update events for size related tables.
+ */
private void handleUpdateEvent(OMDBUpdateEvent<String, Object> event,
String tableName,
Collection<String> sizeRelatedTables,
HashMap<String, Long> objectCountMap,
- HashMap<String, Long> unreplicatedSizeCountMap,
- HashMap<String, Long> replicatedSizeCountMap) {
+ HashMap<String, Long> unReplicatedSizeMap,
+ HashMap<String, Long> replicatedSizeMap) {
+
+ OmTableHandler tableHandler = tableHandlers.get(tableName);
if (event.getValue() != null) {
if (sizeRelatedTables.contains(tableName)) {
Review Comment:
Any reason `tableHandler` null check is not done here as being done in `handlePutEvent` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ArafatKhan2198 commented on a diff in pull request #5037: HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories.
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1258457136
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/TestOmTableInsightTask.java:
##########
@@ -21,26 +21,36 @@
import org.apache.commons.lang3.tuple.ImmutablePair;
import org.apache.commons.lang3.tuple.Pair;
import org.apache.hadoop.hdds.client.StandaloneReplicationConfig;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.server.OzoneStorageContainerManager;
+import org.apache.hadoop.hdds.utils.db.Table;
import org.apache.hadoop.hdds.utils.db.TypedTable;
import org.apache.hadoop.ozone.om.OMMetadataManager;
import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
-import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
-import org.apache.hadoop.ozone.om.helpers.RepeatedOmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.*;
Review Comment:
Done!
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -139,28 +150,38 @@ public Pair<String, Boolean> reprocess(OMMetadataManager omMetadataManager) {
* Returns a triple with the total count of records (left), total unreplicated
* size (middle), and total replicated size (right) in the given iterator.
* Increments count for each record and adds the dataSize if a record's value
- * is an instance of OmKeyInfo. If the iterator is null, returns (0, 0, 0).
+ * is an instance of OmKeyInfo,RepeatedOmKeyInfo.
+ * If the iterator is null, returns (0, 0, 0).
*
* @param iterator The iterator over the table to be iterated.
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org