You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2019/03/09 16:36:29 UTC
[GitHub] [hadoop] steveloughran commented on issue #583: HADOOP-16109.
Parquet reading S3AFileSystem causes EOF
steveloughran commented on issue #583: HADOOP-16109. Parquet reading S3AFileSystem causes EOF
URL: https://github.com/apache/hadoop/pull/583#issuecomment-471198559
Tested: S3 London w/ versioning enabled; s3guard + DDB.
All tests passed except ITestS3AMiniYarnCluster.
```
INFO] Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.754 s - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputByteBuffer
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray
[INFO] Tests run: 27, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 42.389 s - in org.apache.hadoop.fs.s3a.auth.ITestAssumeRole
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AMiscOperations
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 20.11 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster
[ERROR] testWithMiniCluster(org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster) Time elapsed: 19.655 s <<< ERROR!
java.io.FileNotFoundException: No such file or directory: s3a://hwdev-steve-london/fork-0002/test/MiniClusterWordCount/output/part-r-00000
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2269)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2163)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2102)
at org.apache.hadoop.fs.s3a.S3AFileSystem.open(S3AFileSystem.java:702)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:899)
at org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster.readStringFromFile(ITestS3AMiniYarnCluster.java:144)
at org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster.testWithMiniCluster(ITestS3AMiniYarnCluster.java:103)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 11.255 s - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.883 s - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray
[INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.904 s - in org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost
[INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB
```
Now, before HADOOP-14556 went in, (and it's non on this branch...) this test uses the classic MR committers to commit the output, so is vulnerable to list inconsistencies. We may have just seen one. Maybe the fact that versioning is enabled is changing this. More work needed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org