You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ArafatKhan2198 (via GitHub)" <gi...@apache.org> on 2023/02/09 20:43:46 UTC
[GitHub] [ozone] ArafatKhan2198 commented on a diff in pull request #4206: HDDS-7832. Support FSO buckets for FileSizeCountTask.
ArafatKhan2198 commented on code in PR #4206:
URL: https://github.com/apache/ozone/pull/4206#discussion_r1102001051
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/FileSizeCountTask.java:
##########
@@ -76,27 +76,38 @@ public FileSizeCountTask(FileCountBySizeDao fileCountBySizeDao,
*/
@Override
public Pair<String, Boolean> reprocess(OMMetadataManager omMetadataManager) {
- Table<String, OmKeyInfo> omKeyInfoTable =
- omMetadataManager.getKeyTable(getBucketLayout());
+ // Map to store the count of files based on file size
Map<FileSizeCountKey, Long> fileSizeCountMap = new HashMap<>();
+
+ // Call reprocessBucket method for FILE_SYSTEM_OPTIMIZED bucket layout
+ reprocessBucket(BucketLayout.FILE_SYSTEM_OPTIMIZED, omMetadataManager,
+ fileSizeCountMap);
+ // Call reprocessBucket method for LEGACY bucket layout
+ reprocessBucket(BucketLayout.LEGACY, omMetadataManager, fileSizeCountMap);
+
+ // Delete all records from FILE_COUNT_BY_SIZE table
+ int execute = dslContext.delete(FILE_COUNT_BY_SIZE).execute();
+ LOG.info("Deleted {} records from {}", execute, FILE_COUNT_BY_SIZE);
+ writeCountsToDB(true, fileSizeCountMap);
+ LOG.info("Completed a 'reprocess' run of FileSizeCountTask.");
+ return new ImmutablePair<>(getTaskName(), true);
+ }
+
+ private void reprocessBucket(BucketLayout bucketLayout,
+ OMMetadataManager omMetadataManager,
+ Map<FileSizeCountKey, Long> fileSizeCountMap) {
+ Table<String, OmKeyInfo> omKeyInfoTable =
+ omMetadataManager.getKeyTable(bucketLayout);
try (TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
- keyIter = omKeyInfoTable.iterator()) {
+ keyIter = omKeyInfoTable.iterator()) {
while (keyIter.hasNext()) {
Table.KeyValue<String, OmKeyInfo> kv = keyIter.next();
handlePutKeyEvent(kv.getValue(), fileSizeCountMap);
Review Comment:
Thanks for the suggestion @devmadhuu
To ensure efficient processing, I have set a limit of 100,000 records. Once this limit is reached, the records will be flushed into the database and the map will be cleared to make room for the next batch of iterations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org