[jira] [Commented] (HADOOP-14096) s3guard: regression in dirListingUnion

["Aaron Fabbri (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-14096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15874045#comment-15874045 ] 

Aaron Fabbri commented on HADOOP-14096:
---------------------------------------

I was able to exercise the bug by nuking my DDB table then listing my bucket that has a bunch of files already in it.  As expected, some parts of the listing were missing.

Also confirmed this fixes it:

{noformat}
--- a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3Guard.java
+++ b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3Guard.java
@@ -195,7 +195,8 @@ public static S3AFileStatus putAndReturn(MetadataStore ms,
       // Any FileSystem has similar race conditions, but we could persist
       // a stale entry longer.  We could expose an atomic
       // DirListingMetadata#putIfNotPresent()
-      changed = changed || dirMeta.put(s);
+      boolean updated = dirMeta.put(s);
+      changed = changed || updated;
     }
 
     if (changed && isAuthoritative) {
{noformat}


> s3guard: regression in dirListingUnion
> --------------------------------------
>
>                 Key: HADOOP-14096
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14096
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: HADOOP-13345
>            Reporter: Aaron Fabbri
>            Priority: Critical
>
> Just noticed HADOOP-14020 introduced a bug in S3Guard#dirListingUnion.
> The offending change is here:
> {noformat}
> -      if (dirMeta.get(s.getPath()) == null) {
> -        dirMeta.put(s);
> -      }
> +      changed = changed || dirMeta.put(s);
> +    }
> +
> {noformat}
> hint: Logical OR is a short-circuit operator.
>   Easy fix, but should probably come with a unit test for dirListingUnion().



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org