You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Grant Henke (Code Review)" <ge...@cloudera.org> on 2019/04/26 17:27:46 UTC
[kudu-CR] [backup] Add more metadata fields
Grant Henke has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13130
Change subject: [backup] Add more metadata fields
......................................................................
[backup] Add more metadata fields
This patch adds more metadata fields to the metdata
file for each backup.
The following was added:
- table_id: Can be used in the future to handle dropped
or renamed tables gracefully.
- comment: Used to backup and restore column coments.
- column_ids: Can be used in the future to handle
dropped or renamed columns gracefully.
Change-Id: I42458f598a523596acb9f18558e6f518719a969b
---
M java/kudu-backup/src/main/protobuf/backup.proto
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/TableMetadata.scala
M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
M java/kudu-client/src/main/java/org/apache/kudu/Schema.java
M java/kudu-client/src/main/java/org/apache/kudu/util/SchemaGenerator.java
5 files changed, 42 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/30/13130/1
--
To view, visit http://gerrit.cloudera.org:8080/13130
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I42458f598a523596acb9f18558e6f518719a969b
Gerrit-Change-Number: 13130
Gerrit-PatchSet: 1
Gerrit-Owner: Grant Henke <gr...@apache.org>
[kudu-CR] [backup] Add more metadata fields
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/13130 )
Change subject: [backup] Add more metadata fields
......................................................................
Patch Set 2: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/13130
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I42458f598a523596acb9f18558e6f518719a969b
Gerrit-Change-Number: 13130
Gerrit-PatchSet: 2
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Comment-Date: Mon, 29 Apr 2019 15:59:30 +0000
Gerrit-HasComments: No
[kudu-CR] [backup] Add more metadata fields
Posted by "Mike Percy (Code Review)" <ge...@cloudera.org>.
Mike Percy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13130 )
Change subject: [backup] Add more metadata fields
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/13130
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I42458f598a523596acb9f18558e6f518719a969b
Gerrit-Change-Number: 13130
Gerrit-PatchSet: 2
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Comment-Date: Wed, 01 May 2019 21:35:11 +0000
Gerrit-HasComments: No
[kudu-CR] [backup] Add more metadata fields
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13130 )
Change subject: [backup] Add more metadata fields
......................................................................
[backup] Add more metadata fields
This patch adds more metadata fields to the metadata
file for each backup.
The following was added:
- table_id: Can be used in the future to handle dropped
or renamed tables gracefully.
- comment: Used to backup and restore column comments.
- column_ids: Can be used in the future to handle
dropped or renamed columns gracefully.
Change-Id: I42458f598a523596acb9f18558e6f518719a969b
Reviewed-on: http://gerrit.cloudera.org:8080/13130
Tested-by: Grant Henke <gr...@apache.org>
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Reviewed-by: Mike Percy <mp...@apache.org>
---
M java/kudu-backup/src/main/protobuf/backup.proto
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/TableMetadata.scala
M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
M java/kudu-client/src/main/java/org/apache/kudu/Schema.java
M java/kudu-client/src/main/java/org/apache/kudu/util/SchemaGenerator.java
5 files changed, 42 insertions(+), 5 deletions(-)
Approvals:
Grant Henke: Verified
Adar Dembo: Looks good to me, but someone else must approve
Mike Percy: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/13130
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I42458f598a523596acb9f18558e6f518719a969b
Gerrit-Change-Number: 13130
Gerrit-PatchSet: 3
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
[kudu-CR] [backup] Add more metadata fields
Posted by "Mike Percy (Code Review)" <ge...@cloudera.org>.
Mike Percy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13130 )
Change subject: [backup] Add more metadata fields
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13130/2/java/kudu-backup/src/main/protobuf/backup.proto
File java/kudu-backup/src/main/protobuf/backup.proto:
http://gerrit.cloudera.org:8080/#/c/13130/2/java/kudu-backup/src/main/protobuf/backup.proto@115
PS2, Line 115: string table_id = 6;
Does it make sense to include table_id, and column_ids in a patch that will include a test for it, as opposed to this standalone thing that is not tested?
--
To view, visit http://gerrit.cloudera.org:8080/13130
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I42458f598a523596acb9f18558e6f518719a969b
Gerrit-Change-Number: 13130
Gerrit-PatchSet: 2
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Comment-Date: Mon, 29 Apr 2019 22:07:16 +0000
Gerrit-HasComments: Yes
[kudu-CR] [backup] Add more metadata fields
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/13130 )
Change subject: [backup] Add more metadata fields
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13130/2/java/kudu-backup/src/main/protobuf/backup.proto
File java/kudu-backup/src/main/protobuf/backup.proto:
http://gerrit.cloudera.org:8080/#/c/13130/2/java/kudu-backup/src/main/protobuf/backup.proto@115
PS2, Line 115: string table_id = 6;
> Does it make sense to include table_id, and column_ids in a patch that will
I added this before the patches to "handle" them because I wasn't sure which I would get to before a release but wanted to be storing the metadata. One of the patches up uses column_ids and another that uses table_id will be up shortly. Unless you feel strongly I would prefer to leave it as is.
--
To view, visit http://gerrit.cloudera.org:8080/13130
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I42458f598a523596acb9f18558e6f518719a969b
Gerrit-Change-Number: 13130
Gerrit-PatchSet: 2
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Comment-Date: Tue, 30 Apr 2019 00:40:14 +0000
Gerrit-HasComments: Yes
[kudu-CR] [backup] Add more metadata fields
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/13130 )
Change subject: [backup] Add more metadata fields
......................................................................
Patch Set 1:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/13130/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13130/1//COMMIT_MSG@9
PS1, Line 9: This patch adds more metadata fields to the metdata
> metadata
Done
http://gerrit.cloudera.org:8080/#/c/13130/1//COMMIT_MSG@13
PS1, Line 13: - table_id: Can be used in the future to handle dropped
> Nit: indentation
Done
http://gerrit.cloudera.org:8080/#/c/13130/1//COMMIT_MSG@15
PS1, Line 15: - comment: Used to backup and restore column coments.
> comments
Done
--
To view, visit http://gerrit.cloudera.org:8080/13130
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I42458f598a523596acb9f18558e6f518719a969b
Gerrit-Change-Number: 13130
Gerrit-PatchSet: 1
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Comment-Date: Mon, 29 Apr 2019 14:17:07 +0000
Gerrit-HasComments: Yes
[kudu-CR] [backup] Add more metadata fields
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/13130 )
Change subject: [backup] Add more metadata fields
......................................................................
Patch Set 1:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/13130/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13130/1//COMMIT_MSG@9
PS1, Line 9: This patch adds more metadata fields to the metdata
metadata
http://gerrit.cloudera.org:8080/#/c/13130/1//COMMIT_MSG@13
PS1, Line 13: - table_id: Can be used in the future to handle dropped
Nit: indentation
http://gerrit.cloudera.org:8080/#/c/13130/1//COMMIT_MSG@15
PS1, Line 15: - comment: Used to backup and restore column coments.
comments
http://gerrit.cloudera.org:8080/#/c/13130/1/java/kudu-backup/src/main/protobuf/backup.proto
File java/kudu-backup/src/main/protobuf/backup.proto:
http://gerrit.cloudera.org:8080/#/c/13130/1/java/kudu-backup/src/main/protobuf/backup.proto@120
PS1, Line 120: // A map of column name to internal column id.
: // This is useful for detecting dropped and added columns.
Can you doc why column IDs are here and not in ColumnMetadataPB? You could always build a map at runtime after loading the protobuf.
Also, should we track all column IDs that ever existed? Or the max column ID? Would either be useful to backup/restore?
http://gerrit.cloudera.org:8080/#/c/13130/1/java/kudu-client/src/main/java/org/apache/kudu/Schema.java
File java/kudu-client/src/main/java/org/apache/kudu/Schema.java:
PS1:
Do you remember off-hand when the master supplies a schema with column IDs vs. without?
http://gerrit.cloudera.org:8080/#/c/13130/1/java/kudu-client/src/main/java/org/apache/kudu/util/SchemaGenerator.java
File java/kudu-client/src/main/java/org/apache/kudu/util/SchemaGenerator.java:
PS1:
Hmm, why isn't this in src/test/java?
--
To view, visit http://gerrit.cloudera.org:8080/13130
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I42458f598a523596acb9f18558e6f518719a969b
Gerrit-Change-Number: 13130
Gerrit-PatchSet: 1
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Comment-Date: Fri, 26 Apr 2019 19:03:40 +0000
Gerrit-HasComments: Yes
[kudu-CR] [backup] Add more metadata fields
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has removed a vote on this change.
Change subject: [backup] Add more metadata fields
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/13130
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I42458f598a523596acb9f18558e6f518719a969b
Gerrit-Change-Number: 13130
Gerrit-PatchSet: 2
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
[kudu-CR] [backup] Add more metadata fields
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/13130 )
Change subject: [backup] Add more metadata fields
......................................................................
Patch Set 2: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/13130
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I42458f598a523596acb9f18558e6f518719a969b
Gerrit-Change-Number: 13130
Gerrit-PatchSet: 2
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Comment-Date: Mon, 29 Apr 2019 22:18:16 +0000
Gerrit-HasComments: No
[kudu-CR] [backup] Add more metadata fields
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Hello Mike Percy, Kudu Jenkins, Adar Dembo,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13130
to look at the new patch set (#2).
Change subject: [backup] Add more metadata fields
......................................................................
[backup] Add more metadata fields
This patch adds more metadata fields to the metadata
file for each backup.
The following was added:
- table_id: Can be used in the future to handle dropped
or renamed tables gracefully.
- comment: Used to backup and restore column comments.
- column_ids: Can be used in the future to handle
dropped or renamed columns gracefully.
Change-Id: I42458f598a523596acb9f18558e6f518719a969b
---
M java/kudu-backup/src/main/protobuf/backup.proto
M java/kudu-backup/src/main/scala/org/apache/kudu/backup/TableMetadata.scala
M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala
M java/kudu-client/src/main/java/org/apache/kudu/Schema.java
M java/kudu-client/src/main/java/org/apache/kudu/util/SchemaGenerator.java
5 files changed, 42 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/30/13130/2
--
To view, visit http://gerrit.cloudera.org:8080/13130
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I42458f598a523596acb9f18558e6f518719a969b
Gerrit-Change-Number: 13130
Gerrit-PatchSet: 2
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
[kudu-CR] [backup] Add more metadata fields
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/13130 )
Change subject: [backup] Add more metadata fields
......................................................................
Patch Set 1:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/13130/1/java/kudu-backup/src/main/protobuf/backup.proto
File java/kudu-backup/src/main/protobuf/backup.proto:
http://gerrit.cloudera.org:8080/#/c/13130/1/java/kudu-backup/src/main/protobuf/backup.proto@120
PS1, Line 120: // A map of column name to internal column id.
: // This is useful for detecting dropped and added columns.
> Can you doc why column IDs are here and not in ColumnMetadataPB? You could always build a map at runtime after loading the protobuf.
I kept them separate because they are for separate things. Once is primarily for recreating the columns, while the other is primarily for validation. If someone were hand crafting a json file for the sake of creating a table, they don't need the ids.
> Also, should we track all column IDs that ever existed? Or the max column ID? Would either be useful to backup/restore?
I don't have a reason for this yet. We can add it if we do.
http://gerrit.cloudera.org:8080/#/c/13130/1/java/kudu-client/src/main/java/org/apache/kudu/Schema.java
File java/kudu-client/src/main/java/org/apache/kudu/Schema.java:
PS1:
> Do you remember off-hand when the master supplies a schema with column IDs
I am not positive off-hand, but I was under the impression Schemas from the server always have Ids. Schemas constructed by users client side don't.
http://gerrit.cloudera.org:8080/#/c/13130/1/java/kudu-client/src/main/java/org/apache/kudu/util/SchemaGenerator.java
File java/kudu-client/src/main/java/org/apache/kudu/util/SchemaGenerator.java:
PS1:
> Hmm, why isn't this in src/test/java?
It's used in application code for non unit testing reasons like the DistributedDataGenerator.
--
To view, visit http://gerrit.cloudera.org:8080/13130
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I42458f598a523596acb9f18558e6f518719a969b
Gerrit-Change-Number: 13130
Gerrit-PatchSet: 1
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Comment-Date: Fri, 26 Apr 2019 20:59:21 +0000
Gerrit-HasComments: Yes