You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Abraham Fine (JIRA)" <ji...@apache.org> on 2018/01/30 22:37:00 UTC
[jira] [Comment Edited] (HADOOP-15140) S3guard mistakes root URI
without / as non-absolute path
[ https://issues.apache.org/jira/browse/HADOOP-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344201#comment-16344201 ]
Abraham Fine edited comment on HADOOP-15140 at 1/30/18 10:36 PM:
-----------------------------------------------------------------
[~stevel@apache.org] I went ahead and added two tests to {{AbstractContractGetFileStatusTest}}:
{code:java}
@Test
public void testGetFileStatusRootURI() throws Throwable {
String fileSystemURI = getFileSystem().getUri().toString();
assertTrue("uri should not end with '/': " + fileSystemURI, fileSystemURI.endsWith("//") || !fileSystemURI.endsWith("/"));
ContractTestUtils.assertIsDirectory(
getFileSystem().getFileStatus(new Path(fileSystemURI)));
}
@Test
public void testGetFileStatusRootFromChild() throws Throwable {
ContractTestUtils.assertIsDirectory(
getFileSystem().getFileStatus(new Path("/dir").getParent()));
}
{code}
These tests ran against S3, Azure, Azure Data Lake, and HDFS using: {{mvn test -Dtest="**/\*ContractGetFileStatus\*" -DS3guard -fae -Dmaven.test.failure.ignore=true}}
{{testGetFileStatusRootFromChild}} never appears to fail. {{testGetFileStatusRootURI}} does on HDFS, S3, and Azure Data Lake (Azure native passes). Local file systems also pass.
Here are the failures and their corresponding stack traces:
{code:java}
[INFO] Running org.apache.hadoop.fs.contract.hdfs.TestHDFSContractGetFileStatus
[ERROR] Tests run: 20, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 6.344 s <<< FAILURE! - in org.apache.hadoop.fs.contract.hdfs.TestHDFSContractGetFileStatus
[ERROR] testGetFileStatusRootURI(org.apache.hadoop.fs.contract.hdfs.TestHDFSContractGetFileStatus) Time elapsed: 0.02 s <<< ERROR!
java.lang.IllegalArgumentException: Pathname from hdfs://localhost:63826 is not a valid DFS filename.
at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:242)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1568)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1565)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1580)
at org.apache.hadoop.fs.contract.AbstractContractGetFileStatusTest.testGetFileStatusRootURI(AbstractContractGetFileStatusTest.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus
[ERROR] Tests run: 20, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 113.029 s <<< FAILURE! - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus
[ERROR] testGetFileStatusRootURI(org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus) Time elapsed: 1.27 s <<< ERROR!
java.lang.IllegalArgumentException: path must be absolute
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:68)
at org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:60)
at org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:56)
at org.apache.hadoop.fs.s3a.s3guard.S3Guard.putAndReturn(S3Guard.java:149)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2130)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2070)
at org.apache.hadoop.fs.contract.AbstractContractGetFileStatusTest.testGetFileStatusRootURI(AbstractContractGetFileStatusTest.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AContractGetFileStatusV1List
[ERROR] Tests run: 20, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 110.904 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AContractGetFileStatusV1List
[ERROR] testGetFileStatusRootURI(org.apache.hadoop.fs.s3a.ITestS3AContractGetFileStatusV1List) Time elapsed: 1.278 s <<< ERROR!
java.lang.IllegalArgumentException: path must be absolute
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:68)
at org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:60)
at org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:56)
at org.apache.hadoop.fs.s3a.s3guard.S3Guard.putAndReturn(S3Guard.java:149)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2130)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2070)
at org.apache.hadoop.fs.contract.AbstractContractGetFileStatusTest.testGetFileStatusRootURI(AbstractContractGetFileStatusTest.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
[INFO] Running org.apache.hadoop.fs.adl.live.TestAdlContractGetFileStatusLive
[ERROR] Tests run: 20, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 43.848 s <<< FAILURE! - in org.apache.hadoop.fs.adl.live.TestAdlContractGetFileStatusLive
[ERROR] testGetFileStatusRootURI(org.apache.hadoop.fs.adl.live.TestAdlContractGetFileStatusLive) Time elapsed: 1.052 s <<< ERROR!
com.microsoft.azure.datalake.store.ADLException:
Error getting info for file
path is null
Last encountered exception thrown after 1 tries. [HTTP0(null)]
[ServerRequestId:null]
at com.microsoft.azure.datalake.store.ADLStoreClient.getExceptionFromResponse(ADLStoreClient.java:1147)
at com.microsoft.azure.datalake.store.ADLStoreClient.getDirectoryEntry(ADLStoreClient.java:725)
at org.apache.hadoop.fs.adl.AdlFileSystem.getFileStatus(AdlFileSystem.java:476)
at org.apache.hadoop.fs.contract.AbstractContractGetFileStatusTest.testGetFileStatusRootURI(AbstractContractGetFileStatusTest.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
So it seems to me the current behavior in S3 is correct (perhaps the error message could be improved) and we should work to make all the implementations consistent. What do you think?
was (Author: abrahamfine):
[~stevel@apache.org] I went ahead and added two tests to {{AbstractContractGetFileStatusTest}}:
{code:java}
@Test
public void testGetFileStatusRootURI() throws Throwable {
String fileSystemURI = getFileSystem().getUri().toString();
assertTrue("uri should not end with '/': " + fileSystemURI, fileSystemURI.endsWith("//") || !fileSystemURI.endsWith("/"));
ContractTestUtils.assertIsDirectory(
getFileSystem().getFileStatus(new Path(fileSystemURI)));
}
@Test
public void testGetFileStatusRootFromChild() throws Throwable {
ContractTestUtils.assertIsDirectory(
getFileSystem().getFileStatus(new Path("/dir").getParent()));
}
{code}
These tests ran against S3, Azure, Azure Data Lake, and HDFS using: {{mvn test -Dtest="**/*ContractGetFileStatus*" -DS3guard -fae -Dmaven.test.failure.ignore=true}}
{{testGetFileStatusRootFromChild}} never appears to fail. {{testGetFileStatusRootURI}} does on HDFS, S3, and Azure Data Lake (Azure native passes). Local file systems also pass.
Here are the failures and their corresponding stack traces:
{code:java}
[INFO] Running org.apache.hadoop.fs.contract.hdfs.TestHDFSContractGetFileStatus
[ERROR] Tests run: 20, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 6.344 s <<< FAILURE! - in org.apache.hadoop.fs.contract.hdfs.TestHDFSContractGetFileStatus
[ERROR] testGetFileStatusRootURI(org.apache.hadoop.fs.contract.hdfs.TestHDFSContractGetFileStatus) Time elapsed: 0.02 s <<< ERROR!
java.lang.IllegalArgumentException: Pathname from hdfs://localhost:63826 is not a valid DFS filename.
at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:242)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1568)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1565)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1580)
at org.apache.hadoop.fs.contract.AbstractContractGetFileStatusTest.testGetFileStatusRootURI(AbstractContractGetFileStatusTest.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus
[ERROR] Tests run: 20, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 113.029 s <<< FAILURE! - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus
[ERROR] testGetFileStatusRootURI(org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus) Time elapsed: 1.27 s <<< ERROR!
java.lang.IllegalArgumentException: path must be absolute
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:68)
at org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:60)
at org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:56)
at org.apache.hadoop.fs.s3a.s3guard.S3Guard.putAndReturn(S3Guard.java:149)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2130)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2070)
at org.apache.hadoop.fs.contract.AbstractContractGetFileStatusTest.testGetFileStatusRootURI(AbstractContractGetFileStatusTest.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
[INFO] Running org.apache.hadoop.fs.s3a.ITestS3AContractGetFileStatusV1List
[ERROR] Tests run: 20, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 110.904 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AContractGetFileStatusV1List
[ERROR] testGetFileStatusRootURI(org.apache.hadoop.fs.s3a.ITestS3AContractGetFileStatusV1List) Time elapsed: 1.278 s <<< ERROR!
java.lang.IllegalArgumentException: path must be absolute
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:68)
at org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:60)
at org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:56)
at org.apache.hadoop.fs.s3a.s3guard.S3Guard.putAndReturn(S3Guard.java:149)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2130)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2070)
at org.apache.hadoop.fs.contract.AbstractContractGetFileStatusTest.testGetFileStatusRootURI(AbstractContractGetFileStatusTest.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
[INFO] Running org.apache.hadoop.fs.adl.live.TestAdlContractGetFileStatusLive
[ERROR] Tests run: 20, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 43.848 s <<< FAILURE! - in org.apache.hadoop.fs.adl.live.TestAdlContractGetFileStatusLive
[ERROR] testGetFileStatusRootURI(org.apache.hadoop.fs.adl.live.TestAdlContractGetFileStatusLive) Time elapsed: 1.052 s <<< ERROR!
com.microsoft.azure.datalake.store.ADLException:
Error getting info for file
path is null
Last encountered exception thrown after 1 tries. [HTTP0(null)]
[ServerRequestId:null]
at com.microsoft.azure.datalake.store.ADLStoreClient.getExceptionFromResponse(ADLStoreClient.java:1147)
at com.microsoft.azure.datalake.store.ADLStoreClient.getDirectoryEntry(ADLStoreClient.java:725)
at org.apache.hadoop.fs.adl.AdlFileSystem.getFileStatus(AdlFileSystem.java:476)
at org.apache.hadoop.fs.contract.AbstractContractGetFileStatusTest.testGetFileStatusRootURI(AbstractContractGetFileStatusTest.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
So it seems to me the current behavior in S3 is correct (perhaps the error message could be improved) and we should work to make all the implementations consistent. What do you think?
> S3guard mistakes root URI without / as non-absolute path
> --------------------------------------------------------
>
> Key: HADOOP-15140
> URL: https://issues.apache.org/jira/browse/HADOOP-15140
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0
> Reporter: Steve Loughran
> Assignee: Abraham Fine
> Priority: Major
>
> If you call {{getFileStatus("s3a://bucket")}} then S3Guard will throw an exception in putMetadata, as it mistakes the empty path for "non-absolute path"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org