[GitHub] [hadoop] steveloughran commented on issue #1810: HADOOP-16746 mkdirs and s3guard auth mode

[GitBox <gi...@apache.org>]

steveloughran commented on issue #1810: HADOOP-16746 mkdirs and s3guard auth mode
URL: https://github.com/apache/hadoop/pull/1810#issuecomment-575730037
 
 
   
   The lates patch fixes the problem wherein a listStatus of a parent dir
   would stamp on the isAuthoritative flag of every child entry.
   
   This was caused by us blindly overriding entries with new ones.
   
   This patch
   
   * avoids writing unchanged entries by building a list of those and passing it
     down to the meta store in the put(DirListMeta) call.
   * in DDB: filter out of those entries.
   * in non-auth mode, build up a list of entries to add, and write in a batch
     at the end. This is more efficient than the one-by-one operation which
     was being performed, especially as there is no BulkOperationState to cache
     previous work.
   * adds tests to verify no DDB writes take place on repeated lists
   * fixes up all calls of put() to handle the new list of unchanged entries
   * Fixes a bug in TestS3Guard which caused NPEs
   
   The code in S3Guard.dirListingUnion() is a bit ugly as it
   has two very different sequences (auth vs nonauth) intermingled.
   We should consider splitting up the two union mechanisms,
   so as to make it easier to understand how the different update/unions
   work.
   
   Note: we should see a reduction in DDB writes here, in both modes.
   
   ----- 
   
   Updated PR is failing two tests, both clearly related
   
   ```
   [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1 s - in org.apache.hadoop.fs.s3a.ITestBlockingThreadPoolExecutorService
   [INFO] Running org.apache.hadoop.fs.s3a.ITestS3GuardWriteBack
   [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.373 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3GuardWriteBack
   [ERROR] testListStatusWriteBack(org.apache.hadoop.fs.s3a.ITestS3GuardWriteBack)  Time elapsed: 6.154 s  <<< FAILURE!
   java.lang.AssertionError: Metadata store without write back should still only know about /OnS3AndMS, but it has: DirListingMetadata{path=s3a://hwdev-steve-ireland-new/fork-0004/test/ListStatusWriteBack, listMap={s3a://hwdev-steve-ireland-new/fork-0004/test/ListStatusWriteBack/OnS3AndMS=DDBPathMetadata{isAuthoritativeDir=true, PathMetadata=PathMetadata{fileStatus=S3AFileStatus{path=s3a://hwdev-steve-ireland-new/fork-0004/test/ListStatusWriteBack/OnS3AndMS; isDirectory=true; modification_time=0; access_time=0; owner=stevel; group=stevel; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=UNKNOWN eTag=null versionId=null; isEmptyDirectory=UNKNOWN; isDeleted=false; lastUpdated=1579283208710}}, s3a://hwdev-steve-ireland-new/fork-0004/test/ListStatusWriteBack/OnS3=DDBPathMetadata{isAuthoritativeDir=false, PathMetadata=PathMetadata{fileStatus=S3AFileStatus{path=s3a://hwdev-steve-ireland-new/fork-0004/test/ListStatusWriteBack/OnS3; isDirectory=true; modification_time=0; access_time=0; owner=stevel; group=stevel; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=UNKNOWN eTag=null versionId=null; isEmptyDirectory=UNKNOWN; isDeleted=false; lastUpdated=1579283208918}}}, isAuthoritative=false, lastUpdated=1579283208450} expected:<1> but was:<2>
   	at org.junit.Assert.fail(Assert.java:88)
   	at org.junit.Assert.failNotEquals(Assert.java:834)
   	at org.junit.Assert.assertEquals(Assert.java:645)
   	at org.apache.hadoop.fs.s3a.ITestS3GuardWriteBack.testListStatusWriteBack(ITestS3GuardWriteBack.java:89)
   ```
   
   ```
   [ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 53.855 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess
   [ERROR] testNoReadAccess[nonauth](org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess)  Time elapsed: 19.955 s  <<< FAILURE!
   java.lang.AssertionError: Expected a java.nio.file.AccessDeniedException to be thrown, but got the result: : true
   	at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:499)
   	at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384)
   	at org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess.accessDenied(ITestRestrictedReadAccess.java:681)
   	at org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess.checkDeleteOperations(ITestRestrictedReadAccess.java:638)
   	at org.apache.hadoop.fs.s3a.auth.ITestRestrictedReadAccess.testNoReadAccess(ITestRestrictedReadAccess.java:307)
    ```
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org