You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Chris Nauroth (JIRA)" <ji...@apache.org> on 2016/08/03 00:08:20 UTC
[jira] [Updated] (HADOOP-13459) hadoop-aws runs several test cases
repeatedly, causing unnecessarily long running time.
[ https://issues.apache.org/jira/browse/HADOOP-13459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Nauroth updated HADOOP-13459:
-----------------------------------
Attachment: HADOOP-13459.001.patch
This patch refactors the tests to introduce a new abstract class: {{AbstractWasbTestBase}}. The vast majority of test suites can subclass this to get just the setup and teardown logic without accidentally re-running test cases.
Here is some information on running time for the impacted test suites before and after the patch.
*Before:*
Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
Tests run: 62, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 325.178 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
Running org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
Tests run: 58, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 316.563 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
Tests run: 52, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 392.583 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
Tests run: 47, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 284.985 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
Tests run: 48, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 300.001 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 296.564 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.994 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
*After:*
Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.088 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
Running org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.198 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 76.855 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.6 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.944 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 307.217 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.915 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
> hadoop-aws runs several test cases repeatedly, causing unnecessarily long running time.
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-13459
> URL: https://issues.apache.org/jira/browse/HADOOP-13459
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure, test
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Priority: Minor
> Attachments: HADOOP-13459.001.patch
>
>
> Within hadoop-azure, we have the {{NativeAzureFileSystemBaseTest}} abstract class, which defines setup and teardown to handle the Azure storage account and also defines multiple test cases. This class originally was contributed to provide a layer of indirection for running the same test cases in live mode or mock mode: {{TestNativeAzureFileSystemLive}} and {{TestNativeAzureFileSystemMocked}}. It appears that since then, we created multiple new test suites that subclassed {{NativeAzureFileSystemBaseTest}} for the benefit of getting the common setup and teardown code, but also with the side effect of running the inherited test cases repeatedly. This is a significant factor in the overall execution time of the hadoop-azure tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org