You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Mingliang Liu (JIRA)" <ji...@apache.org> on 2016/12/06 08:10:59 UTC
[jira] [Updated] (HADOOP-13449) S3Guard: Implement
DynamoDBMetadataStore.
[ https://issues.apache.org/jira/browse/HADOOP-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mingliang Liu updated HADOOP-13449:
-----------------------------------
Attachment: HADOOP-13449-HADOOP-13345.011.patch
The v11 patch:
# Rebases from the feature branch, and also enforces that when putting a new item, it also put all its ancestors to the DDB table. One downside of this change is that, {{testPutNew()}} will fail as it assumes we don't create&put its ancestors when we put a new item. We have to update that.
# Removes the {{isEmpty}} field in the DDB table; it queries the DDB for determining whether directory items are empty.
# Simply borrowes the idea of a new class DynamoDBClientFactor. The unit test will always use a mocked S3 client and null metadata store, except the {{TestDynamoDBMetadataStore}} which starts a local DDB server for comprehensive test.
I know [~fabbri] has also some in-progress changes. I hope you don't mind rebasing and uploading a new one after that.
Will run and pass more integration tests soon.
{code}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus
Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp
Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractDelete
Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractCreate
Tests run: 10, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 64.509 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractCreate
Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractMkdir
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 72.058 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractDelete
Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractOpen
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.762 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractOpen
Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 46.983 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractMkdir
Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 57.878 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename
Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractSeek
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 247.155 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp
Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContext
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.202 sec - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContext
Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextCreateMkdir
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 250.842 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus
Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 139.282 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractSeek
Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics
Tests run: 9, Failures: 3, Errors: 1, Skipped: 0, Time elapsed: 208.542 sec <<< FAILURE! - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir
testRecursiveRootListing(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir) Time elapsed: 119.093 sec <<< FAILURE!
java.lang.AssertionError: files mismatch: between
"s3a://mliu-test-aws-s3guard/Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aws/target/test-dir/2/qw4OLiKxn8/file"
"s3a://mliu-test-aws-s3guard/file.txt"
"s3a://mliu-test-aws-s3guard/fork-2/test/ITestS3AContractDistCp/largeFilesFromRemote/inputDir/file2"
"s3a://mliu-test-aws-s3guard/fork-2/test/ITestS3AContractDistCp/largeFilesFromRemote/inputDir/file1"
"s3a://mliu-test-aws-s3guard/fork-2/test/ITestS3AContractDistCp/largeFilesFromRemote/inputDir/file3"
"s3a://mliu-test-aws-s3guard/user/mliu/test/parentdir/child"
"s3a://mliu-test-aws-s3guard/user/mliu/test/file"
] and
"s3a://mliu-test-aws-s3guard/Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aws/target/test-dir/2/qw4OLiKxn8/file"
"s3a://mliu-test-aws-s3guard/file.txt"
"s3a://mliu-test-aws-s3guard/fork-2/test/ITestS3AContractDistCp/largeFilesToRemote/outputDir/inputDir/file3"
"s3a://mliu-test-aws-s3guard/fork-4/test/testComplexDirActions/dir-0-0000/file--0-0000-0000.txt"
"s3a://mliu-test-aws-s3guard/fork-4/test/testComplexDirActions/dir-0-0000/file--0-0000-0001.txt"
"s3a://mliu-test-aws-s3guard/fork-4/test/testComplexDirActions/dir-0-0000/file--0-0000-0002.txt"
"s3a://mliu-test-aws-s3guard/fork-4/test/testComplexDirActions/dir-0-0000/file--0-0000-0003.txt"
"s3a://mliu-test-aws-s3guard/fork-4/test/testComplexDirActions/file--0-0000.txt"
"s3a://mliu-test-aws-s3guard/fork-4/test/testComplexDirActions/file--0-0001.txt"
"s3a://mliu-test-aws-s3guard/fork-4/test/testComplexDirActions/file--0-0002.txt"
"s3a://mliu-test-aws-s3guard/fork-4/test/testComplexDirActions/file--0-0003.txt"
"s3a://mliu-test-aws-s3guard/user/mliu/test/file"
"s3a://mliu-test-aws-s3guard/user/mliu/test/parentdir/child"
]
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.apache.hadoop.fs.contract.ContractTestUtils$TreeScanResults.assertFieldsEquivalent(ContractTestUtils.java:1369)
at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testRecursiveRootListing(AbstractContractRootDirectoryTest.java:222)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
testRmEmptyRootDirNonRecursive(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir) Time elapsed: 35.451 sec <<< FAILURE!
java.lang.AssertionError: After 1 attempts: listing after rm /* not empty
final [00] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/Users; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
[01] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/fork-3; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
deleted [00] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/Users; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
[01] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/fork-3; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
[02] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/path; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
[03] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/fork-4; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
[04] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/fork-1; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
[05] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/file.txt; isDirectory=false; length=0; replication=1; blocksize=33554432; modification_time=1481010208231; access_time=0; owner=mliu; group=mliu; permission=rw-rw-rw-; isSymlink=false} isEmptyDirectory=false
[06] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/fork-2; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
[07] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/user; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
original [00] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/Users; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
[01] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/fork-3; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
[02] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/path; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
[03] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/fork-4; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
[04] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/fork-1; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
[05] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/file.txt; isDirectory=false; length=0; replication=1; blocksize=33554432; modification_time=1481010208231; access_time=0; owner=mliu; group=mliu; permission=rw-rw-rw-; isSymlink=false} isEmptyDirectory=false
[06] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/fork-2; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
[07] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/user; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
at org.junit.Assert.fail(Assert.java:88)
at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest$1.call(AbstractContractRootDirectoryTest.java:103)
at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest$1.call(AbstractContractRootDirectoryTest.java:97)
at org.apache.hadoop.test.LambdaTestUtils.eventually(LambdaTestUtils.java:234)
at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testRmEmptyRootDirNonRecursive(AbstractContractRootDirectoryTest.java:95)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
testListEmptyRootDirectory(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir) Time elapsed: 34.391 sec <<< ERROR!
java.lang.NullPointerException: null
at org.apache.hadoop.fs.s3a.s3guard.DescendantsIterator.next(DescendantsIterator.java:133)
at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.deleteSubtree(DynamoDBMetadataStore.java:262)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerDelete(S3AFileSystem.java:1340)
at org.apache.hadoop.fs.s3a.S3AFileSystem.delete(S3AFileSystem.java:1261)
at org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:609)
at org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:590)
at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testListEmptyRootDirectory(AbstractContractRootDirectoryTest.java:186)
at org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir.testListEmptyRootDirectory(ITestS3AContractRootDir.java:49)
testSimpleRootListing(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir) Time elapsed: 2.493 sec <<< FAILURE!
java.lang.AssertionError: expected:<3> but was:<2>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testSimpleRootListing(AbstractContractRootDirectoryTest.java:207)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.604 sec - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics
Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextUtil
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.459 sec - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextUtil
Running org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider
Tests run: 4, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 6.343 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider
testAnonymousProvider(org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider) Time elapsed: 1.866 sec <<< ERROR!
org.apache.hadoop.fs.s3a.AWSServiceIOException: initializing on s3a://landsat-pds/scene_list.gz: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: Request is missing Authentication Token (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: MissingAuthenticationTokenException; Request ID: 62EEHHLVUVIDPQ5V29A4R5O8RBVV4KQNSO5AEMVJF66Q9ASUAAJG): Request is missing Authentication Token (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: MissingAuthenticationTokenException; Request ID: 62EEHHLVUVIDPQ5V29A4R5O8RBVV4KQNSO5AEMVJF66Q9ASUAAJG)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1529)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1167)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:1722)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1698)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.createTable(AmazonDynamoDBClient.java:743)
at com.amazonaws.services.dynamodbv2.document.DynamoDB.createTable(DynamoDB.java:96)
at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.createTable(DynamoDBMetadataStore.java:455)
at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:184)
at org.apache.hadoop.fs.s3a.s3guard.S3Guard.getMetadataStore(S3Guard.java:138)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3246)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3295)
at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:3269)
at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:529)
at org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider.testAnonymousProvider(ITestS3AAWSCredentialsProvider.java:133)
testBadCredentials(org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider) Time elapsed: 1.301 sec <<< ERROR!
org.apache.hadoop.fs.s3a.AWSServiceIOException: initializing on s3a://mliu-test-aws-s3guard/: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: The security token included in the request is invalid. (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: UnrecognizedClientException; Request ID: ICJT511MGKE0PTBRS2D5OH75FJVV4KQNSO5AEMVJF66Q9ASUAAJG): The security token included in the request is invalid. (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: UnrecognizedClientException; Request ID: ICJT511MGKE0PTBRS2D5OH75FJVV4KQNSO5AEMVJF66Q9ASUAAJG)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1529)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1167)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:1722)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1698)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.createTable(AmazonDynamoDBClient.java:743)
at com.amazonaws.services.dynamodbv2.document.DynamoDB.createTable(DynamoDB.java:96)
at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.createTable(DynamoDBMetadataStore.java:455)
at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:184)
at org.apache.hadoop.fs.s3a.s3guard.S3Guard.getMetadataStore(S3Guard.java:138)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)
at org.apache.hadoop.fs.s3a.S3ATestUtils.createTestFileSystem(S3ATestUtils.java:103)
at org.apache.hadoop.fs.s3a.S3ATestUtils.createTestFileSystem(S3ATestUtils.java:63)
at org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider.createFailingFS(ITestS3AAWSCredentialsProvider.java:76)
at org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider.testBadCredentials(ITestS3AAWSCredentialsProvider.java:102)
Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray
Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 143.379 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextCreateMkdir
testMkdirsRecursiveWithExistingDir(org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextCreateMkdir) Time elapsed: 34.171 sec <<< ERROR!
java.io.FileNotFoundException: No such file or directory: s3a://mliu-test-aws-s3guard/Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aws/target/test-dir/2/dYvFHHBXOC/aDir/bDir/cDir
at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1710)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:119)
at org.apache.hadoop.fs.DelegateToFileSystem.getFileStatus(DelegateToFileSystem.java:126)
at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1167)
at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1163)
at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
at org.apache.hadoop.fs.FileContext.getFileStatus(FileContext.java:1169)
at org.apache.hadoop.fs.FileContextCreateMkdirBaseTest.testMkdirsRecursiveWithExistingDir(FileContextCreateMkdirBaseTest.java:125)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputByteBuffer
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.146 sec - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray
Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.391 sec - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputByteBuffer
Running org.apache.hadoop.fs.s3a.ITestS3ABlocksize
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.089 sec - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk
Running org.apache.hadoop.fs.s3a.ITestS3AConfiguration
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.792 sec - in org.apache.hadoop.fs.s3a.ITestS3ABlocksize
Running org.apache.hadoop.fs.s3a.ITestS3ACredentialsInURL
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.613 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3ACredentialsInURL
testInvalidCredentialsFail(org.apache.hadoop.fs.s3a.ITestS3ACredentialsInURL) Time elapsed: 1.443 sec <<< FAILURE!
java.lang.AssertionError: Expected an AccessDeniedException, got S3AFileStatus{path=s3a://mliu-test-aws-s3guard/; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=false
at org.junit.Assert.fail(Assert.java:88)
at org.apache.hadoop.fs.s3a.ITestS3ACredentialsInURL.testInvalidCredentialsFail(ITestS3ACredentialsInURL.java:130)
Running org.apache.hadoop.fs.s3a.ITestS3AEncryption
Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.095 sec - in org.apache.hadoop.fs.s3a.ITestS3AConfiguration
Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmPropagation
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.756 sec - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmPropagation
Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionBlockOutputStream
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 71.234 sec - in org.apache.hadoop.fs.s3a.ITestS3AEncryption
Running org.apache.hadoop.fs.s3a.ITestS3AFailureHandling
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.71 sec - in org.apache.hadoop.fs.s3a.ITestS3AFailureHandling
Running org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.232 sec - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionBlockOutputStream
Running org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract
Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 69.269 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost
testFakeDirectoryDeletion(org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost) Time elapsed: 38.245 sec <<< FAILURE!
java.lang.AssertionError: after rename(srcFilePath, destFilePath): directories_created expected:<1> but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.apache.hadoop.fs.s3a.S3ATestUtils$MetricDiff.assertDiffEquals(S3ATestUtils.java:431)
at org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost.testFakeDirectoryDeletion(ITestS3AFileOperationCost.java:254)
Running org.apache.hadoop.fs.s3a.ITestS3AMiscOperations
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.188 sec - in org.apache.hadoop.fs.s3a.ITestS3AMiscOperations
Running org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.366 sec - in org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials
Running org.apache.hadoop.fs.s3a.ITestS3ATestUtils
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.123 sec - in org.apache.hadoop.fs.s3a.ITestS3ATestUtils
Running org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteFilesOneByOne
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 4.396 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteFilesOneByOne
Running org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteManyFiles
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 4.342 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteManyFiles
Running org.apache.hadoop.fs.s3a.scale.ITestS3ADirectoryPerformance
Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 17.954 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3ADirectoryPerformance
Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesArrayBlocks
Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 23.965 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesArrayBlocks
Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesByteBufferBlocks
Tests run: 17, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 353.478 sec - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI
Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesClassicOutput
Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 15.458 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesByteBufferBlocks
Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesDiskBlocks
Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 15.381 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesClassicOutput
Running org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 16.141 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesDiskBlocks
Running org.apache.hadoop.fs.s3a.yarn.ITestS3A
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.081 sec - in org.apache.hadoop.fs.s3a.yarn.ITestS3A
Running org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster
Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 24.094 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.678 sec - in org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster
Tests run: 43, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 279.272 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract
testRenameToDirWithSamePrefixAllowed(org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract) Time elapsed: 4.693 sec <<< ERROR!
org.apache.hadoop.fs.s3a.AWSServiceIOException: createTable: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: Provided list of item keys contains duplicates (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: P128RFOF3H9IAHJTKRUR8PIHENVV4KQNSO5AEMVJF66Q9ASUAAJG): Provided list of item keys contains duplicates (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: P128RFOF3H9IAHJTKRUR8PIHENVV4KQNSO5AEMVJF66Q9ASUAAJG)
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:166)
at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.move(DynamoDBMetadataStore.java:347)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:862)
at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:655)
at org.apache.hadoop.fs.FileSystemContractBaseTest.rename(FileSystemContractBaseTest.java:512)
at org.apache.hadoop.fs.FileSystemContractBaseTest.testRenameToDirWithSamePrefixAllowed(FileSystemContractBaseTest.java:656)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:255)
at junit.framework.TestSuite.run(TestSuite.java:250)
at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Caused by: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: Provided list of item keys contains duplicates (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: P128RFOF3H9IAHJTKRUR8PIHENVV4KQNSO5AEMVJF66Q9ASUAAJG)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1529)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1167)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:1722)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1698)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.batchWriteItem(AmazonDynamoDBClient.java:668)
at com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.doBatchWriteItem(BatchWriteItemImpl.java:111)
at com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.batchWriteItem(BatchWriteItemImpl.java:52)
at com.amazonaws.services.dynamodbv2.document.DynamoDB.batchWriteItem(DynamoDB.java:178)
at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.move(DynamoDBMetadataStore.java:338)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:862)
at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:655)
at org.apache.hadoop.fs.FileSystemContractBaseTest.rename(FileSystemContractBaseTest.java:512)
at org.apache.hadoop.fs.FileSystemContractBaseTest.testRenameToDirWithSamePrefixAllowed(FileSystemContractBaseTest.java:656)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:255)
at junit.framework.TestSuite.run(TestSuite.java:250)
at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
{code}
> S3Guard: Implement DynamoDBMetadataStore.
> -----------------------------------------
>
> Key: HADOOP-13449
> URL: https://issues.apache.org/jira/browse/HADOOP-13449
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Chris Nauroth
> Assignee: Mingliang Liu
> Attachments: HADOOP-13449-HADOOP-13345.000.patch, HADOOP-13449-HADOOP-13345.001.patch, HADOOP-13449-HADOOP-13345.002.patch, HADOOP-13449-HADOOP-13345.003.patch, HADOOP-13449-HADOOP-13345.004.patch, HADOOP-13449-HADOOP-13345.005.patch, HADOOP-13449-HADOOP-13345.006.patch, HADOOP-13449-HADOOP-13345.007.patch, HADOOP-13449-HADOOP-13345.008.patch, HADOOP-13449-HADOOP-13345.009.patch, HADOOP-13449-HADOOP-13345.010.patch, HADOOP-13449-HADOOP-13345.011.patch
>
>
> Provide an implementation of the metadata store backed by DynamoDB.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org