You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2018/08/07 01:44:00 UTC
[jira] [Comment Edited] (HADOOP-15546) ABFS: tune imports &
javadocs; stabilise tests
[ https://issues.apache.org/jira/browse/HADOOP-15546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16571010#comment-16571010 ]
Steve Loughran edited comment on HADOOP-15546 at 8/7/18 1:43 AM:
-----------------------------------------------------------------
HADOOP-15546 patch 009
* resource azure-bfs-test.xml imports azure-bfs-auth-keys.xml so you can set fs.azure.test.account.name to an abfs account
* and the abfs tests check that this is true
* fix test which didn't compile with the touch() call in the superclass
* log4j.properties back to logging all at debug. noisier, but sets you up to debug things.
* ITestAzureNativeContractAppend.testAppendDirectory() expects the exception raised by that FS
The sole reason for needing a separate {{azure-bfs-auth-keys.xml}} seems to be for reusing the {{fs.azure.test.account.name}} key to refer to an abfs account with the abfs tests. Given this doesn't get along with the waze tests, I don't understand why this is needed. Can't we just have a separate config key , say {{fs.abfs.test.account.name}} for the abfs tests? That will also provide an easy way to skip all the wasb tests in the module: simply undefine {fs.azure.test.account.name}} from the config resources
Testing: wasb against UK; abfs against US. I'm in the US right now and a lot of the WASB tests were timing out in a (very slow) parallel test run,
The only real outlier here was {{ITestBlobDataValidation}}, where the assertion that MD5 checksums were coming back failed. This happens repeatedly, even when run as a single test. I have no idea what is going on here.
{code}
[ERROR] Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 21.424 s <<< FAILURE! - in org.apache.hadoop.fs.azure.ITestBlobDataValidation
[ERROR] testCheckBlockMd5(org.apache.hadoop.fs.azure.ITestBlobDataValidation) Time elapsed: 2.541 s <<< ERROR!
java.io.IOException: java.lang.AssertionError Please see the cause for further information.
at com.microsoft.azure.storage.core.Utility.initIOException(Utility.java:778)
at com.microsoft.azure.storage.blob.BlobOutputStreamInternal.flush(BlobOutputStreamInternal.java:533)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
at org.apache.hadoop.fs.azure.NativeAzureFileSystem$NativeAzureFsOutputStream.close(NativeAzureFileSystem.java:1058)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
at org.apache.hadoop.fs.azure.ITestBlobDataValidation.testCheckBlockMd5(ITestBlobDataValidation.java:235)
at org.apache.hadoop.fs.azure.ITestBlobDataValidation.testCheckBlockMd5(ITestBlobDataValidation.java:151)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Suppressed: java.io.IOException: java.lang.AssertionError Please see the cause for further information.
at com.microsoft.azure.storage.core.Utility.initIOException(Utility.java:778)
at com.microsoft.azure.storage.blob.BlobOutputStreamInternal.flush(BlobOutputStreamInternal.java:533)
at com.microsoft.azure.storage.blob.BlobOutputStreamInternal.close(BlobOutputStreamInternal.java:317)
at java.io.FilterOutputStream.close(FilterOutputStream.java:159)
... 17 more
Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at com.microsoft.azure.storage.blob.BlobOutputStreamInternal.flush(BlobOutputStreamInternal.java:530)
... 19 more
Caused by: java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertNotNull(Assert.java:621)
at org.junit.Assert.assertNotNull(Assert.java:631)
at org.apache.hadoop.fs.azure.ITestBlobDataValidation$ContentMD5Checker.checkObtainedMd5(ITestBlobDataValidation.java:193)
at org.apache.hadoop.fs.azure.ITestBlobDataValidation$ContentMD5Checker.eventOccurred(ITestBlobDataValidation.java:186)
at org.apache.hadoop.fs.azure.ITestBlobDataValidation$ContentMD5Checker.eventOccurred(ITestBlobDataValidation.java:170)
at com.microsoft.azure.storage.StorageEventMultiCaster.fireEvent(StorageEventMultiCaster.java:52)
at com.microsoft.azure.storage.core.ExecutionEngine.fireResponseReceivedEvent(ExecutionEngine.java:383)
at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:127)
at com.microsoft.azure.storage.blob.CloudBlockBlob.uploadBlockInternal(CloudBlockBlob.java:1097)
at com.microsoft.azure.storage.blob.CloudBlockBlob.uploadBlock(CloudBlockBlob.java:1069)
at com.microsoft.azure.storage.blob.BlobOutputStreamInternal.writeBlock(BlobOutputStreamInternal.java:456)
at com.microsoft.azure.storage.blob.BlobOutputStreamInternal.access$000(BlobOutputStreamInternal.java:47)
at com.microsoft.azure.storage.blob.BlobOutputStreamInternal$1.call(BlobOutputStreamInternal.java:406)
at com.microsoft.azure.storage.blob.BlobOutputStreamInternal$1.call(BlobOutputStreamInternal.java:403)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at com.microsoft.azure.storage.blob.BlobOutputStreamInternal.flush(BlobOutputStreamInternal.java:530)
... 19 more
[CIRCULAR REFERENCE:java.lang.AssertionError]
{code}
I'm assuming that test failure is unrelated to anything I've done, but am curious what others see...
was (Author: stevel@apache.org):
HADOOP-15546 patch 009
* resource azure-bfs-test.xml imports azure-bfs-auth-keys.xml so you can set fs.azure.test.account.name to an abfs account
* and the abfs tests check that this is true
* fix test which didn't compile with the touch() call in the superclass
* log4j.properties back to logging all at debug. noisier, but sets you up to debug things.
* ITestAzureNativeContractAppend.testAppendDirectory() expects the exception raised by that FS
The sole reason for needing a separate {{azure-bfs-auth-keys.xml}} seems to be for reusing the {{fs.azure.test.account.name}}
key to refer to an abfs account with the abfs tests. Given this doesn't get along with the waze tests, I don't understand why
this is needed. Can't we just have a separate config key , say {{fs.abfs.test.account.name}} for the abfs tests? That will
also provide an easy way to skip all the wasb tests in the module: simply undefine {{fs.azure.test.account.name}} from the config resources
Testing: wasb against UK; abfs against US. I'm in the US right now and a lot of the WASB tests were timing out in a (very slow) parallel test run,
The only real outlier here was {{ITestBlobDataValidation}}, where the assertion that MD5 checksums were coming back failed. This happens
repeatedly, even when run as a single test. I have no idea what is going on here.
{code}
[ERROR] Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 21.424 s <<< FAILURE! - in org.apache.hadoop.fs.azure.ITestBlobDataValidation
[ERROR] testCheckBlockMd5(org.apache.hadoop.fs.azure.ITestBlobDataValidation) Time elapsed: 2.541 s <<< ERROR!
java.io.IOException: java.lang.AssertionError Please see the cause for further information.
at com.microsoft.azure.storage.core.Utility.initIOException(Utility.java:778)
at com.microsoft.azure.storage.blob.BlobOutputStreamInternal.flush(BlobOutputStreamInternal.java:533)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
at org.apache.hadoop.fs.azure.NativeAzureFileSystem$NativeAzureFsOutputStream.close(NativeAzureFileSystem.java:1058)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
at org.apache.hadoop.fs.azure.ITestBlobDataValidation.testCheckBlockMd5(ITestBlobDataValidation.java:235)
at org.apache.hadoop.fs.azure.ITestBlobDataValidation.testCheckBlockMd5(ITestBlobDataValidation.java:151)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Suppressed: java.io.IOException: java.lang.AssertionError Please see the cause for further information.
at com.microsoft.azure.storage.core.Utility.initIOException(Utility.java:778)
at com.microsoft.azure.storage.blob.BlobOutputStreamInternal.flush(BlobOutputStreamInternal.java:533)
at com.microsoft.azure.storage.blob.BlobOutputStreamInternal.close(BlobOutputStreamInternal.java:317)
at java.io.FilterOutputStream.close(FilterOutputStream.java:159)
... 17 more
Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at com.microsoft.azure.storage.blob.BlobOutputStreamInternal.flush(BlobOutputStreamInternal.java:530)
... 19 more
Caused by: java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertNotNull(Assert.java:621)
at org.junit.Assert.assertNotNull(Assert.java:631)
at org.apache.hadoop.fs.azure.ITestBlobDataValidation$ContentMD5Checker.checkObtainedMd5(ITestBlobDataValidation.java:193)
at org.apache.hadoop.fs.azure.ITestBlobDataValidation$ContentMD5Checker.eventOccurred(ITestBlobDataValidation.java:186)
at org.apache.hadoop.fs.azure.ITestBlobDataValidation$ContentMD5Checker.eventOccurred(ITestBlobDataValidation.java:170)
at com.microsoft.azure.storage.StorageEventMultiCaster.fireEvent(StorageEventMultiCaster.java:52)
at com.microsoft.azure.storage.core.ExecutionEngine.fireResponseReceivedEvent(ExecutionEngine.java:383)
at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:127)
at com.microsoft.azure.storage.blob.CloudBlockBlob.uploadBlockInternal(CloudBlockBlob.java:1097)
at com.microsoft.azure.storage.blob.CloudBlockBlob.uploadBlock(CloudBlockBlob.java:1069)
at com.microsoft.azure.storage.blob.BlobOutputStreamInternal.writeBlock(BlobOutputStreamInternal.java:456)
at com.microsoft.azure.storage.blob.BlobOutputStreamInternal.access$000(BlobOutputStreamInternal.java:47)
at com.microsoft.azure.storage.blob.BlobOutputStreamInternal$1.call(BlobOutputStreamInternal.java:406)
at com.microsoft.azure.storage.blob.BlobOutputStreamInternal$1.call(BlobOutputStreamInternal.java:403)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at com.microsoft.azure.storage.blob.BlobOutputStreamInternal.flush(BlobOutputStreamInternal.java:530)
... 19 more
[CIRCULAR REFERENCE:java.lang.AssertionError]
{code}
I'm assuming that test failure is unrelated to anything I've done, but am curious what others see...
> ABFS: tune imports & javadocs; stabilise tests
> ----------------------------------------------
>
> Key: HADOOP-15546
> URL: https://issues.apache.org/jira/browse/HADOOP-15546
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: HADOOP-15407
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Attachments: HADOOP-15546-001.patch, HADOOP-15546-HADOOP-15407-001.patch, HADOOP-15546-HADOOP-15407-002.patch, HADOOP-15546-HADOOP-15407-003.patch, HADOOP-15546-HADOOP-15407-004.patch, HADOOP-15546-HADOOP-15407-005.patch, HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-006.patch, HADOOP-15546-HADOOP-15407-007.patch, HADOOP-15546-HADOOP-15407-008.patch, HADOOP-15546-HADOOP-15407-009.patch, azure-auth-keys.xml
>
>
> Followup on HADOOP-15540 with some initial review tuning
> h2. Tuning
> * ordering of imports
> * rely on azure-auth-keys.xml to store credentials (change imports, docs,.gitignore)
> * log4j -> info
> * add a "." to the first sentence of all the javadocs I noticed.
> * remove @Public annotations except for some constants (which includes some commitment to maintain them).
> * move the AbstractFS declarations out of the src/test/resources XML file into core-default.xml for all to use
> * other IDE-suggested tweaks
> h2. Testing
> Review the tests, move to ContractTestUtil assertions, make more consistent to contract test setup, and general work to make the tests work well over slower links, document, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org