[jira] [Commented] (HADOOP-14107) ITestS3GuardListConsistency fails intermittently

["Mingliang Liu (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-14107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15956028#comment-15956028 ] 

Mingliang Liu commented on HADOOP-14107:
----------------------------------------

[~stevel@apache.org], thanks for your input.

I reran the test but did not reproduce the failure. There are two possibilities that may fail this test:
# In {{noWriteBack.listStatus(directory);}}, the directory on S3 only may not be listed because of the eventually consistency in S3 list. S3Guard is not guaranteed to discover new S3 objects in a timely manner. This is possible but I did not reproduce this in test.
# If the test data was not cleaned up during last test process, then this test will fail, as the stack trace shows in description section.

The #1 is a corner case and we can simply sleep for 3~5 seconds. The #2 can be addressed by deleting the existing test directory explicitly. We may still need to sleep for a while before delete tracking is supported. Meanwhile, per your comment, I changed the coding style, comments, variable names etc in the v0 patch to make the test purpose clearer.

The major point is that, is the test doing the right thing? I think it is. It creates a directory on S3 only (via noS3Guard), and another directory on S3 and DDB aka metadata store (via noWriteBack). Then if we list the file system with S3Guard enabled, regardless of S3Guard writing back or not, it should return both of the data. The only exception is the point #1 above: if the data only in S3 is not visible yet. That's being said, both noWriteBack and yesWriteBack will discover S3-only object, and union that with DDB result. However, noWriteBack should not populate the newly found S3-only object to DDB while yesWriteBack should always populate the S3-only object to DDB. We assert that via querying DDB directly.

> ITestS3GuardListConsistency fails intermittently
> ------------------------------------------------
>
>                 Key: HADOOP-14107
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14107
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: HADOOP-13345
>            Reporter: Mingliang Liu
>            Assignee: Mingliang Liu
>         Attachments: HADOOP-14107-HADOOP-13345.000.patch
>
>
> {code}
> mvn -Dit.test='ITestS3GuardListConsistency' -Dtest=none -Dscale -Ds3guard -Ddynamo -q clean verify
> -------------------------------------------------------
>  T E S T S
> -------------------------------------------------------
> Running org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.544 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency
> testListStatusWriteBack(org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency)  Time elapsed: 3.147 sec  <<< FAILURE!
> java.lang.AssertionError: Unexpected number of results from metastore. Metastore should only know about /XYZ: DirListingMetadata{path=s3a://mliu-s3guard/test/ListStatusWriteBack, listMap={s3a://mliu-s3guard/test/ListStatusWriteBack/XYZ=PathMetadata{fileStatus=S3AFileStatus{path=s3a://mliu-s3guard/test/ListStatusWriteBack/XYZ; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true}, s3a://mliu-s3guard/test/ListStatusWriteBack/123=PathMetadata{fileStatus=S3AFileStatus{path=s3a://mliu-s3guard/test/ListStatusWriteBack/123; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true}}, isAuthoritative=false}
> 	at org.junit.Assert.fail(Assert.java:88)
> 	at org.junit.Assert.assertTrue(Assert.java:41)
> 	at org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testListStatusWriteBack(ITestS3GuardListConsistency.java:127)
> {code}
> See discussion on the parent JIRA [HADOOP-13345].



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org