You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/03/13 19:41:27 UTC

[GitHub] [hadoop-ozone] avijayanhwx opened a new pull request #675: HDDS-3170. Fix issues in File count by size task.

avijayanhwx opened a new pull request #675: HDDS-3170. Fix issues in File count by size task.
URL: https://github.com/apache/hadoop-ozone/pull/675
 
 
   ## What changes were proposed in this pull request?
   - Handle DELETE key operation correctly.
   - Handle PUT key operation for an existing key.
   
   ## What is the link to the Apache JIRA
   https://issues.apache.org/jira/browse/HDDS-3170
   
   ## How was this patch tested?
   Manually tested on docker.
   Added unit tests for the new cases.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org

[GitHub] [hadoop-ozone] avijayanhwx commented on issue #675: HDDS-3170. Fix issues in File count by size task.

Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on issue #675: HDDS-3170. Fix issues in File count by size task.
URL: https://github.com/apache/hadoop-ozone/pull/675#issuecomment-599767574
 
 
   Thank you for the reviews @swagle , @vivekratnavel & @adoroszlai. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org

[GitHub] [hadoop-ozone] swagle commented on a change in pull request #675: HDDS-3170. Fix issues in File count by size task.

Posted by GitBox <gi...@apache.org>.
swagle commented on a change in pull request #675: HDDS-3170. Fix issues in File count by size task.
URL: https://github.com/apache/hadoop-ozone/pull/675#discussion_r392439126
 
 

 ##########
 File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/FileSizeCountTask.java
 ##########
 @@ -229,22 +228,39 @@ void populateFileCountBySizeDB() {
    * Used by reprocess() and process().
    *
    * @param omKeyInfo OmKey being updated for count
-   * @param operation (PUT, DELETE)
    */
-  void updateUpperBoundCount(OmKeyInfo omKeyInfo,
-      OMDBUpdateEvent.OMDBUpdateAction operation) {
+  void handlePutKeyEvent(OmKeyInfo omKeyInfo) {
     int binIndex = calculateBinIndex(omKeyInfo.getDataSize());
-    if (operation == PUT) {
-      upperBoundCount[binIndex]++;
-    } else if (operation == DELETE) {
-      if (upperBoundCount[binIndex] != 0) {
+    upperBoundCount[binIndex]++;
+  }
+
+  /**
+   * Calculate and update the count of files being tracked by
+   * upperBoundCount[].
+   * Used by reprocess() and process().
+   *
+   * @param omKeyInfo OmKey being updated for count
+   */
+  void handleDeleteKeyEvent(String key, OmKeyInfo omKeyInfo) {
+    if (omKeyInfo == null) {
+      LOG.warn("Unexpected error while handling DELETE key event. Key not " +
+          "found in Recon OM DB : {}", key);
+    } else {
+      int binIndex = calculateBinIndex(omKeyInfo.getDataSize());
+      if (upperBoundCount[binIndex] > 0) {
         //decrement only if it had files before, default DB value is 0
         upperBoundCount[binIndex]--;
       } else {
         LOG.warn("Unexpected error while updating bin count. Found 0 count " +
-            "for index : {} while processing DELETE event for {}", binIndex,
+                "for index : {} while processing DELETE event for {}", binIndex,
 
 Review comment:
   Minor: bad indent

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org

[GitHub] [hadoop-ozone] swagle commented on a change in pull request #675: HDDS-3170. Fix issues in File count by size task.

Posted by GitBox <gi...@apache.org>.
swagle commented on a change in pull request #675: HDDS-3170. Fix issues in File count by size task.
URL: https://github.com/apache/hadoop-ozone/pull/675#discussion_r392438979
 
 

 ##########
 File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OMDBUpdatesHandler.java
 ##########
 @@ -88,30 +93,47 @@ private void processEvent(int cfIndex, byte[] keyBytes, byte[]
       valueBytes, OMDBUpdateEvent.OMDBUpdateAction action)
       throws IOException {
     String tableName = tablesNames.get(cfIndex);
-    Class keyType = getKeyType();
+    Class<String> keyType = getKeyType();
     Class valueType = getValueType(tableName);
     if (valueType != null) {
       OMDBUpdateEvent.OMUpdateEventBuilder builder =
           new OMDBUpdateEvent.OMUpdateEventBuilder<>();
       builder.setTable(tableName);
+      builder.setAction(action);
 
-      Object key = codecRegistry.asObject(keyBytes, keyType);
+      String key = codecRegistry.asObject(keyBytes, keyType);
       builder.setKey(key);
 
-      if (!action.equals(OMDBUpdateEvent.OMDBUpdateAction.DELETE)) {
+      if (action.equals(PUT)) {
 
 Review comment:
   Minor: Can be changed to == enum compare instead.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org

[GitHub] [hadoop-ozone] avijayanhwx commented on issue #675: HDDS-3170. Fix issues in File count by size task.

Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on issue #675: HDDS-3170. Fix issues in File count by size task.
URL: https://github.com/apache/hadoop-ozone/pull/675#issuecomment-598883906
 
 
   @vivekratnavel / @shwetayakkali / @swagle Please review.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org

[GitHub] [hadoop-ozone] avijayanhwx merged pull request #675: HDDS-3170. Fix issues in File count by size task.

Posted by GitBox <gi...@apache.org>.
avijayanhwx merged pull request #675: HDDS-3170. Fix issues in File count by size task.
URL: https://github.com/apache/hadoop-ozone/pull/675
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org

[GitHub] [hadoop-ozone] vivekratnavel commented on issue #675: HDDS-3170. Fix issues in File count by size task.

Posted by GitBox <gi...@apache.org>.
vivekratnavel commented on issue #675: HDDS-3170. Fix issues in File count by size task.
URL: https://github.com/apache/hadoop-ozone/pull/675#issuecomment-599175045
 
 
   +1 LGTM

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org