[jira] [Commented] (HBASE-16458) Shorten backup / restore test execution time

["Vladimir Rodionov (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-16458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16606490#comment-16606490 ] 

Vladimir Rodionov commented on HBASE-16458:
-------------------------------------------

Reopened.

Every test class starts a new HBase cluster, because surefire plugin was configured to run every class in a new JVM. This overhead accounts for ~ 25% of a total execution time (63 min total, 17 min cluster start up time). So it make sense to modify backup tests to be able to share single JVM and modify surefire plugin configuration to reuse JVM during B&R UTs execution.

> Shorten backup / restore test execution time
> --------------------------------------------
>
>                 Key: HBASE-16458
>                 URL: https://issues.apache.org/jira/browse/HBASE-16458
>             Project: HBase
>          Issue Type: Test
>            Reporter: Ted Yu
>            Assignee: Ted Yu
>            Priority: Major
>              Labels: backup
>         Attachments: 16458.HBASE-7912.v3.txt, 16458.HBASE-7912.v4.txt, 16458.HBASE-7912.v5.txt, 16458.v1.txt, 16458.v2.txt, 16458.v3.txt
>
>
> Below was timing information for all the backup / restore tests (today's result):
> {code}
> Running org.apache.hadoop.hbase.backup.TestIncrementalBackup
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 576.273 sec - in org.apache.hadoop.hbase.backup.TestIncrementalBackup
> Running org.apache.hadoop.hbase.backup.TestBackupBoundaryTests
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 124.67 sec - in org.apache.hadoop.hbase.backup.TestBackupBoundaryTests
> Running org.apache.hadoop.hbase.backup.TestBackupStatusProgress
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 102.34 sec - in org.apache.hadoop.hbase.backup.TestBackupStatusProgress
> Running org.apache.hadoop.hbase.backup.TestBackupAdmin
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 490.251 sec - in org.apache.hadoop.hbase.backup.TestBackupAdmin
> Running org.apache.hadoop.hbase.backup.TestHFileArchiving
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 84.323 sec - in org.apache.hadoop.hbase.backup.TestHFileArchiving
> Running org.apache.hadoop.hbase.backup.TestSystemTableSnapshot
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 65.492 sec - in org.apache.hadoop.hbase.backup.TestSystemTableSnapshot
> Running org.apache.hadoop.hbase.backup.TestBackupDescribe
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 93.758 sec - in org.apache.hadoop.hbase.backup.TestBackupDescribe
> Running org.apache.hadoop.hbase.backup.TestBackupLogCleaner
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 109.187 sec - in org.apache.hadoop.hbase.backup.TestBackupLogCleaner
> Running org.apache.hadoop.hbase.backup.TestIncrementalBackupNoDataLoss
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 330.539 sec - in org.apache.hadoop.hbase.backup.TestIncrementalBackupNoDataLoss
> Running org.apache.hadoop.hbase.backup.TestRemoteBackup
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 84.371 sec - in org.apache.hadoop.hbase.backup.TestRemoteBackup
> Running org.apache.hadoop.hbase.backup.TestBackupSystemTable
> Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 67.893 sec - in org.apache.hadoop.hbase.backup.TestBackupSystemTable
> Running org.apache.hadoop.hbase.backup.TestRestoreBoundaryTests
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 120.779 sec - in org.apache.hadoop.hbase.backup.TestRestoreBoundaryTests
> Running org.apache.hadoop.hbase.backup.TestFullBackupSetRestoreSet
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 117.815 sec - in org.apache.hadoop.hbase.backup.TestFullBackupSetRestoreSet
> Running org.apache.hadoop.hbase.backup.TestBackupShowHistory
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 136.517 sec - in org.apache.hadoop.hbase.backup.TestBackupShowHistory
> Running org.apache.hadoop.hbase.backup.TestRemoteRestore
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 91.799 sec - in org.apache.hadoop.hbase.backup.TestRemoteRestore
> Running org.apache.hadoop.hbase.backup.TestFullRestore
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 317.711 sec - in org.apache.hadoop.hbase.backup.TestFullRestore
> Running org.apache.hadoop.hbase.backup.TestFullBackupSet
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 87.045 sec - in org.apache.hadoop.hbase.backup.TestFullBackupSet
> Running org.apache.hadoop.hbase.backup.TestBackupDelete
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 86.214 sec - in org.apache.hadoop.hbase.backup.TestBackupDelete
> Running org.apache.hadoop.hbase.backup.TestBackupDeleteRestore
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 77.631 sec - in org.apache.hadoop.hbase.backup.TestBackupDeleteRestore
> Running org.apache.hadoop.hbase.backup.TestIncrementalBackupDeleteTable
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 190.358 sec - in org.apache.hadoop.hbase.backup.TestIncrementalBackupDeleteTable
> Running org.apache.hadoop.hbase.backup.TestBackupShowHistoryFromBackupDestination
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 93.726 sec - in org.apache.hadoop.hbase.backup.TestBackupShowHistoryFromBackupDestination
> Running org.apache.hadoop.hbase.backup.TestFullBackup
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 177.137 sec - in org.apache.hadoop.hbase.backup.TestFullBackup
> {code}
> The duration was close to 1 hour:
> {code}
> [INFO] Apache HBase - Server .............................. SUCCESS [  01:02 h]
> {code}
> We should:
> * merge duplicate backup / restore test cases
> * shorten backup / restore test execution time (ingesting less data, etc)
> * use multi wal for selected test(s) since rolling / replaying more WAL files takes more time



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)