You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Aaron Fabbri (JIRA)" <ji...@apache.org> on 2017/06/15 20:55:00 UTC
[jira] [Commented] (HADOOP-14527) ITestS3GuardListConsistency is
too slow
[ https://issues.apache.org/jira/browse/HADOOP-14527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051053#comment-16051053 ]
Aaron Fabbri commented on HADOOP-14527:
---------------------------------------
Added a method to clear any delayed keys from the inconsistent client, instead of doing Thread.sleep() to wait for timeout. This only gave small speedup. Pasting runtimes of main test cases here:
{noformat}
2017-06-15 11:29:24,976 [JUnit-testConsistentRenameAfterDelete] INFO s3a.AbstractS3ATestBase (ITestS3GuardListConsistency.java:testConsistentRenameAfterDelete(244)) - took 17939 msec
2017-06-15 11:29:43,360 [JUnit-testConsistentListAfterDelete] INFO s3a.AbstractS3ATestBase (ITestS3GuardListConsistency.java:testConsistentListAfterDelete(194)) - took 16404 msec
2017-06-15 11:33:15,135 [JUnit-testConsistentListFiles] INFO s3a.AbstractS3ATestBase (ITestS3GuardListConsistency.java:testConsistentListFiles(373)) - took 209143 msec
2017-06-15 11:34:57,714 [JUnit-testConsistentListLocatedStatusAfterPut] INFO s3a.AbstractS3ATestBase (ITestS3GuardListConsistency.java:testConsistentListLocatedStatusAfterPut(305)) - took 73391 msec
2017-06-15 11:35:13,635 [JUnit-testConsistentListAfterRename] INFO s3a.AbstractS3ATestBase (ITestS3GuardListConsistency.java:testConsistentListAfterRename(121)) - took 9961 msec
2017-06-15 11:35:18,434 [JUnit-testConsistentListStatusAfterPut] INFO s3a.AbstractS3ATestBase (ITestS3GuardListConsistency.java:testConsistentListStatusAfterPut(279)) - took 3212 msec
2017-06-15 11:35:56,313 [JUnit-testRollingRenames] INFO s3a.AbstractS3ATestBase (ITestS3GuardListConsistency.java:testRollingRenames(153)) - took 35460 msec
{noformat}
Will look at reducing the big ones when I get a chance.
> ITestS3GuardListConsistency is too slow
> ---------------------------------------
>
> Key: HADOOP-14527
> URL: https://issues.apache.org/jira/browse/HADOOP-14527
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: HADOOP-13345
> Reporter: Aaron Fabbri
> Assignee: Aaron Fabbri
> Priority: Minor
>
> I'm really glad to see folks adopting the inconsistency injection stuff and adding test cases to ITestS3GuardListConsistency. That test class has become very slow, however, mostly due to {{Thread.sleep()}} calls that wait for the inconsistency injection timers to expire.
> I will take a stab at speeding up this test class. As is it takes about 8 minutes to run.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org