You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Will Berkeley (Code Review)" <ge...@cloudera.org> on 2019/05/23 20:51:46 UTC
[kudu-CR] [backup] KUDU-2787 Allow single table failures for backup and restore
Will Berkeley has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13415
Change subject: [backup] KUDU-2787 Allow single table failures for backup and restore
......................................................................
[backup] KUDU-2787 Allow single table failures for backup and restore
This path enhances backup to be able to survive the failures of
individual tables. It adds a new option `--failOnFirstError` to the
backup job, which defaults to false, and which when true causes the
backup job to fail as soon as any table has an error (the previous
behavior).
A follow up will add a similar option to the restore job.
Change-Id: I4f3731f8df04791727c44af9ade9989f2cc6ef95
---
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/Options.scala
M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
3 files changed, 134 insertions(+), 60 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/15/13415/1
--
To view, visit http://gerrit.cloudera.org:8080/13415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I4f3731f8df04791727c44af9ade9989f2cc6ef95
Gerrit-Change-Number: 13415
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
[kudu-CR] [backup] KUDU-2787 Allow single table failures for backup and restore
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Hello Mike Percy, Kudu Jenkins, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13415
to look at the new patch set (#3).
Change subject: [backup] KUDU-2787 Allow single table failures for backup and restore
......................................................................
[backup] KUDU-2787 Allow single table failures for backup and restore
This path enhances backup to be able to survive the failures of
individual tables. It adds a new option `--failOnFirstError` to the
backup job, which defaults to false, and which when true causes the
backup job to fail as soon as any table has an error (the previous
behavior).
A follow up will add a similar option to the restore job.
Change-Id: I4f3731f8df04791727c44af9ade9989f2cc6ef95
---
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/Options.scala
M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestOptions.scala
4 files changed, 134 insertions(+), 60 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/15/13415/3
--
To view, visit http://gerrit.cloudera.org:8080/13415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4f3731f8df04791727c44af9ade9989f2cc6ef95
Gerrit-Change-Number: 13415
Gerrit-PatchSet: 3
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
[kudu-CR] [backup] KUDU-2787 Allow single table failures for backup and restore
Posted by "Mike Percy (Code Review)" <ge...@cloudera.org>.
Mike Percy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13415 )
Change subject: [backup] KUDU-2787 Allow single table failures for backup and restore
......................................................................
Patch Set 4:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/13415/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13415/4//COMMIT_MSG@7
PS4, Line 7: [backup] KUDU-2787 Allow single table failures for backup and restore
s/and restore//
http://gerrit.cloudera.org:8080/#/c/13415/4/java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
File java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala:
http://gerrit.cloudera.org:8080/#/c/13415/4/java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala@596
PS4, Line 596: doBackup
nit: Would you mind renaming this function to runBackup() to avoid confusion, since the helper in KuduBackup.scala is also named doBackup()?
--
To view, visit http://gerrit.cloudera.org:8080/13415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f3731f8df04791727c44af9ade9989f2cc6ef95
Gerrit-Change-Number: 13415
Gerrit-PatchSet: 4
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Comment-Date: Tue, 28 May 2019 22:53:52 +0000
Gerrit-HasComments: Yes
[kudu-CR] [backup] KUDU-2787 Allow single table failures for backup
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Hello Mike Percy, Kudu Jenkins, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13415
to look at the new patch set (#5).
Change subject: [backup] KUDU-2787 Allow single table failures for backup
......................................................................
[backup] KUDU-2787 Allow single table failures for backup
This path enhances backup to be able to survive the failures of
individual tables. It adds a new option `--failOnFirstError` to the
backup job, which defaults to false, and which when true causes the
backup job to fail as soon as any table has an error (the previous
behavior).
A follow up will add a similar option to the restore job.
Change-Id: I4f3731f8df04791727c44af9ade9989f2cc6ef95
---
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/Options.scala
M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestOptions.scala
4 files changed, 134 insertions(+), 60 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/15/13415/5
--
To view, visit http://gerrit.cloudera.org:8080/13415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4f3731f8df04791727c44af9ade9989f2cc6ef95
Gerrit-Change-Number: 13415
Gerrit-PatchSet: 5
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
[kudu-CR] [backup] KUDU-2787 Allow single table failures for backup and restore
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13415
to look at the new patch set (#2).
Change subject: [backup] KUDU-2787 Allow single table failures for backup and restore
......................................................................
[backup] KUDU-2787 Allow single table failures for backup and restore
This path enhances backup to be able to survive the failures of
individual tables. It adds a new option `--failOnFirstError` to the
backup job, which defaults to false, and which when true causes the
backup job to fail as soon as any table has an error (the previous
behavior).
A follow up will add a similar option to the restore job.
Change-Id: I4f3731f8df04791727c44af9ade9989f2cc6ef95
---
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/Options.scala
M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
3 files changed, 133 insertions(+), 60 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/15/13415/2
--
To view, visit http://gerrit.cloudera.org:8080/13415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4f3731f8df04791727c44af9ade9989f2cc6ef95
Gerrit-Change-Number: 13415
Gerrit-PatchSet: 2
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [backup] KUDU-2787 Allow single table failures for backup
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13415 )
Change subject: [backup] KUDU-2787 Allow single table failures for backup
......................................................................
Patch Set 5:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/13415/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13415/4//COMMIT_MSG@7
PS4, Line 7: [backup] KUDU-2787 Allow single table failures for backup
> s/and restore//
Done
http://gerrit.cloudera.org:8080/#/c/13415/4/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala
File java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala:
http://gerrit.cloudera.org:8080/#/c/13415/4/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala@140
PS4, Line 140: s"Failed to back up table $tableName: Look back in the logs for the full exception. Error: ${ex.toString}")
> Is ex.toString not the full exception? You should be able to use `log.error
It doesn't include the stack trace. I don't want that here because it should be printed above by the `log.error`.
http://gerrit.cloudera.org:8080/#/c/13415/4/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala@142
PS4, Line 142: if (backupResults.exists(_._2.isFailure))
> Maybe return a boolean instead of an int here. That way this is just:
I like that it returns the exit code. It makes `main` very clean.
http://gerrit.cloudera.org:8080/#/c/13415/4/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala@158
PS4, Line 158: System.exit(run(options, session))
> Instead of calling System.exit which is difficult to capture in testing, ca
Call `run` in tests.
http://gerrit.cloudera.org:8080/#/c/13415/4/java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
File java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala:
http://gerrit.cloudera.org:8080/#/c/13415/4/java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala@107
PS4, Line 107: // Check that a backup of a missing table fails fast with an exception if the
> Is there a test for failOnFirstError = false?
Yes. See the second backup taken in `testFailedTableDoesNotFailOtherTables`.
http://gerrit.cloudera.org:8080/#/c/13415/4/java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala@596
PS4, Line 596: runBacku
> nit: Would you mind renaming this function to runBackup() to avoid confusio
Done
--
To view, visit http://gerrit.cloudera.org:8080/13415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f3731f8df04791727c44af9ade9989f2cc6ef95
Gerrit-Change-Number: 13415
Gerrit-PatchSet: 5
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Wed, 29 May 2019 18:05:36 +0000
Gerrit-HasComments: Yes
[kudu-CR] [backup] KUDU-2787 Allow single table failures for backup
Posted by "Mike Percy (Code Review)" <ge...@cloudera.org>.
Mike Percy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13415 )
Change subject: [backup] KUDU-2787 Allow single table failures for backup
......................................................................
Patch Set 5: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/13415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f3731f8df04791727c44af9ade9989f2cc6ef95
Gerrit-Change-Number: 13415
Gerrit-PatchSet: 5
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Thu, 30 May 2019 16:51:30 +0000
Gerrit-HasComments: No
[kudu-CR] [backup] KUDU-2787 Allow single table failures for backup
Posted by "Mike Percy (Code Review)" <ge...@cloudera.org>.
Mike Percy has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13415 )
Change subject: [backup] KUDU-2787 Allow single table failures for backup
......................................................................
[backup] KUDU-2787 Allow single table failures for backup
This path enhances backup to be able to survive the failures of
individual tables. It adds a new option `--failOnFirstError` to the
backup job, which defaults to false, and which when true causes the
backup job to fail as soon as any table has an error (the previous
behavior).
A follow up will add a similar option to the restore job.
Change-Id: I4f3731f8df04791727c44af9ade9989f2cc6ef95
Reviewed-on: http://gerrit.cloudera.org:8080/13415
Tested-by: Kudu Jenkins
Reviewed-by: Grant Henke <gr...@apache.org>
Reviewed-by: Mike Percy <mp...@apache.org>
---
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/Options.scala
M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestOptions.scala
4 files changed, 134 insertions(+), 60 deletions(-)
Approvals:
Kudu Jenkins: Verified
Grant Henke: Looks good to me, but someone else must approve
Mike Percy: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/13415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I4f3731f8df04791727c44af9ade9989f2cc6ef95
Gerrit-Change-Number: 13415
Gerrit-PatchSet: 6
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR] [backup] KUDU-2787 Allow single table failures for backup
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/13415 )
Change subject: [backup] KUDU-2787 Allow single table failures for backup
......................................................................
Patch Set 5: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/13415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f3731f8df04791727c44af9ade9989f2cc6ef95
Gerrit-Change-Number: 13415
Gerrit-PatchSet: 5
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Thu, 30 May 2019 16:20:12 +0000
Gerrit-HasComments: No
[kudu-CR] [backup] KUDU-2787 Allow single table failures for backup and restore
Posted by "Mike Percy (Code Review)" <ge...@cloudera.org>.
Mike Percy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13415 )
Change subject: [backup] KUDU-2787 Allow single table failures for backup and restore
......................................................................
Patch Set 4:
Dup with the other 2787 patch?
--
To view, visit http://gerrit.cloudera.org:8080/13415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f3731f8df04791727c44af9ade9989f2cc6ef95
Gerrit-Change-Number: 13415
Gerrit-PatchSet: 4
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Comment-Date: Tue, 28 May 2019 17:13:11 +0000
Gerrit-HasComments: No
[kudu-CR] [backup] KUDU-2787 Allow single table failures for backup and restore
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/13415 )
Change subject: [backup] KUDU-2787 Allow single table failures for backup and restore
......................................................................
Patch Set 4:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/13415/4/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala
File java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala:
http://gerrit.cloudera.org:8080/#/c/13415/4/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala@140
PS4, Line 140: s"Failed to back up table $tableName: Look back in the logs for the full exception. Error: ${ex.toString}")
Is ex.toString not the full exception? You should be able to use `log.error(msg, ex)` instead of calling ex.tostring.
http://gerrit.cloudera.org:8080/#/c/13415/4/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala@142
PS4, Line 142: if (backupResults.exists(_._2.isFailure))
Maybe return a boolean instead of an int here. That way this is just:
// Return the success of the backup.
backupResults.exists(_._2.isFailure)
http://gerrit.cloudera.org:8080/#/c/13415/4/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala@158
PS4, Line 158: System.exit(run(options, session))
Instead of calling System.exit which is difficult to capture in testing, can you instead throw a RuntimeException here if run returns false.
http://gerrit.cloudera.org:8080/#/c/13415/4/java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
File java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala:
http://gerrit.cloudera.org:8080/#/c/13415/4/java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala@107
PS4, Line 107: // Check that a backup of a missing table fails fast with an exception if the
Is there a test for failOnFirstError = false?
--
To view, visit http://gerrit.cloudera.org:8080/13415
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4f3731f8df04791727c44af9ade9989f2cc6ef95
Gerrit-Change-Number: 13415
Gerrit-PatchSet: 4
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Comment-Date: Wed, 29 May 2019 15:41:13 +0000
Gerrit-HasComments: Yes