You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Yingchun Lai (Code Review)" <ge...@cloudera.org> on 2019/02/23 04:03:54 UTC
[kudu-CR] [tools] Add tool to copy table data to another table
Yingchun Lai has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12563
Change subject: [tools] Add tool to copy table data to another table
......................................................................
[tools] Add tool to copy table data to another table
Copy table data to another table, the two tables could be in the same
cluster or not. The two tables must have the same table schema, but
could have different partition schemas. And also it's able to create
new table with the same table schema and partition schema with the
source table by this tool.
Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
---
M src/kudu/client/schema.cc
M src/kudu/client/schema.h
M src/kudu/common/partition.h
M src/kudu/common/schema.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/test_workload.cc
M src/kudu/tools/data_gen_util.cc
M src/kudu/tools/kudu-admin-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/table_scanner.cc
M src/kudu/tools/table_scanner.h
M src/kudu/tools/tool_action_table.cc
12 files changed, 925 insertions(+), 43 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/63/12563/1
--
To view, visit http://gerrit.cloudera.org:8080/12563
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
Gerrit-Change-Number: 12563
Gerrit-PatchSet: 1
Gerrit-Owner: Yingchun Lai <40...@qq.com>
[kudu-CR] [tools] Add tool to copy table data to another table
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/12563 )
Change subject: [tools] Add tool to copy table data to another table
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12563/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/12563/1//COMMIT_MSG@7
PS1, Line 7: [tools] Add tool to copy table data to another table
I'm a bit nervous about providing this as a non-distributed CLI tool vs a Spark job or something else that's naturally distributed and fault tolerant.
Can you share the use case here? It seems like this would only work well for very small tables. And it also overlaps quite a bit with the tooling being built for the backup project (based on Java/Spark)
--
To view, visit http://gerrit.cloudera.org:8080/12563
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
Gerrit-Change-Number: 12563
Gerrit-PatchSet: 1
Gerrit-Owner: Yingchun Lai <40...@qq.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Wed, 27 Feb 2019 00:45:12 +0000
Gerrit-HasComments: Yes
[kudu-CR] [tools] Add tool to copy table data to another table
Posted by "Yingchun Lai (Code Review)" <ge...@cloudera.org>.
Yingchun Lai has posted comments on this change. ( http://gerrit.cloudera.org:8080/12563 )
Change subject: [tools] Add tool to copy table data to another table
......................................................................
Patch Set 8:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/12563/7/src/kudu/tools/kudu-tool-test.cc
File src/kudu/tools/kudu-tool-test.cc:
http://gerrit.cloudera.org:8080/#/c/12563/7/src/kudu/tools/kudu-tool-test.cc@2861
PS7, Line 2861: ASSERT_LE(num, kNumTables);
> There's a lot of repetition between these tests. Consider making them param
Good idea!
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc
File src/kudu/tools/table_scanner.cc:
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@321
PS5, Line 321:
> Whoops, I made a typo in my recommendation. Should have been "Destination t
Done
http://gerrit.cloudera.org:8080/#/c/12563/7/src/kudu/tools/table_scanner.cc
File src/kudu/tools/table_scanner.cc:
http://gerrit.cloudera.org:8080/#/c/12563/7/src/kudu/tools/table_scanner.cc@20
PS7, Line 20: #include <cstddef>
: #include <cstdint>
: #include <cstring>
> Nit: we prefer C++ equivalents where they exist (i.e. <cstddef>, <cstdint>,
Done
--
To view, visit http://gerrit.cloudera.org:8080/12563
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
Gerrit-Change-Number: 12563
Gerrit-PatchSet: 8
Gerrit-Owner: Yingchun Lai <40...@qq.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Yingchun Lai <40...@qq.com>
Gerrit-Comment-Date: Fri, 22 Mar 2019 09:09:51 +0000
Gerrit-HasComments: Yes
[kudu-CR] [tools] Add tool to copy table data to another table
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/12563 )
Change subject: [tools] Add tool to copy table data to another table
......................................................................
Patch Set 5:
(36 comments)
Didn't review the new tests yet, had a bunch of comments on the code.
http://gerrit.cloudera.org:8080/#/c/12563/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/12563/5//COMMIT_MSG@9
PS5, Line 9: ,
Nit: use a semicolon (';') here, since the two parts of the sentence are distinct and separate thoughts.
http://gerrit.cloudera.org:8080/#/c/12563/5//COMMIT_MSG@11
PS5, Line 11: And also it's able to create
: new table with the same table schema and partition schema with the
: source table by this tool.
Reword: "Alternatively, the tool can create the new table using the same table and partition schema as the source table."
http://gerrit.cloudera.org:8080/#/c/12563/1/src/kudu/common/partition.h
File src/kudu/common/partition.h:
http://gerrit.cloudera.org:8080/#/c/12563/1/src/kudu/common/partition.h@139
PS1, Line 139: struct RangeSchema {
: std::vector<ColumnId> column_ids;
: };
> Does this need to be public?
Could you answer this question?
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/common/schema.cc
File src/kudu/common/schema.cc:
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/common/schema.cc@a348
PS5, Line 348:
Why do we need to lift this restriction?
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.h
File src/kudu/tools/table_scanner.h:
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.h@55
PS5, Line 55: TableScanner(client::sp::shared_ptr<KuduClient> client, string table_name,
You can avoid the second constructor by adding the two new params into the existing constructor with default values of boost::none (if you wrap them with boost::optional).
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.h@64
PS5, Line 64: // Not thread-safe.
: Status StartScan();
:
: // Not thread-safe.
: Status StartCopy();
I'd document the entire class as not thread-safe, though I don't think anyone would expect it to be.
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.h@78
PS5, Line 78: client::sp::shared_ptr<KuduClient> dst_client_;
: string dst_table_name_;
Should wrap these in boost::optional to emphasize their optionality.
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc
File src/kudu/tools/table_scanner.cc:
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@110
PS5, Line 110: "Table data write type, 'insert', 'upsert' or empty string, while empty string means "
: "not copy data."
Nit: reword as "How data should be copied to the destination table. Valid values are 'insert', 'upsert' or the empty string. If the empty string, data will not be copied (useful when create_table is 'true')."
How do you actually specify the empty string though? Is there a test for this?
Also please add a gflag validator to enforce that the flag's value is one of those accepted values.
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@318
PS5, Line 318: is exist.
Nit: exists
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@321
PS5, Line 321: Destination table is the same with the source.
Nit: "Destination table is as the same as the source table"
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@327
PS5, Line 327: is NOT exist.
Nit: does NOT exist.
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@329
PS5, Line 329: is not
Nit: does not exist
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@339
PS5, Line 339: ColumnNames
Nit: rename to "convert_column_ids_to_names".
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@343
PS5, Line 343: push_back
Nit: emplace_back() for new code.
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@384
PS5, Line 384: auto lower = new KuduPartialRow(&schema_internal);
: auto upper = new KuduPartialRow(&schema_internal);
Store these in unique_ptrs and then release() them into add_range_partition. That'll prevent a memory leak if a DecodeRangeKey call fails.
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@396
PS5, Line 396: create success
Nit: "created successfully"
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@401
PS5, Line 401: Status AddRow(const client::sp::shared_ptr<KuduTable>& table,
Can we do something simpler here where we copy the raw data from 'row' into 'write_row'? Shouldn't need to expand this for every type since this is an internal tool; we can poke holes with friendship in KuduPartialRow and/or KuduScanBatch::RowPtr if need be.
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@403
PS5, Line 403: const KuduScanBatch::RowPtr& row,
Nit: rename to src_row and rename write_row to dst_row.
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@411
PS5, Line 411: return Status::InvalidArgument(Substitute("invalid write_type: $0", FLAGS_write_type));
This can LOG(FATAL) I think; it'd be a programming error to wind up here.
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@546
PS5, Line 546: LOG(ERROR) << error->status().ToString() << endl;
LOG() will automatically terminate messages with endl, so you can remove this.
Elsewhere too.
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@552
PS5, Line 552: KuduValue* ParseValue(KuduColumnSchema::DataType type,
Where is this used?
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@614
PS5, Line 614: void TableScanner::CopyTask(const vector<KuduScanToken*>& tokens) {
I think you can reuse much of the code here. Decompose the bulk of the scan loop into a separate method that takes a functor as an argument. The idea is to call the functor for each row. You could parameterize the "scanned" vs. "copied" string too, if that's important to you.
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@640
PS5, Line 640: for (auto it = batch.begin(); it != batch.end(); ++it) {
Can do a simple for loop:
for (const auto& row : batch)
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@644
PS5, Line 644: Status s = session->Flush();
With AUTO_FLUSH_BACKGROUND why bother with the explicit Flush()? If you want to check for errors, periodically poll the error collector.
You probably want to close the session at the end though.
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@668
PS5, Line 668: Status TableScanner::StartScan() {
Seems like the bulk of this code be reused via decomposition too.
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@726
PS5, Line 726: // Copy schema only.
Nit: "Create table only."
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@749
PS5, Line 749: push_back
Nit: emplace_back for new code.
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/tool_action_table.cc
File src/kudu/tools/tool_action_table.cc:
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/tool_action_table.cc@76
PS5, Line 76: "The name of the destination table the data will be copied to.");
Doc what it means if this is the empty string (i.e. if the option is not set).
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/tool_action_table.cc@139
PS5, Line 139: const char* const kDstMasterAddressesArg = "dst_master_addresses";
Nit: reword as "dest_master_addresses"; the 'e' is only one extra char but helps emphasize that the abbreviation is short for "destination".
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/tool_action_table.cc@152
PS5, Line 152: Status CreateKuduClient(const RunnerContext& context,
: client::sp::shared_ptr<KuduClient>* client) {
: return CreateKuduClient(context, kMasterAddressesArg, client);
: }
:
: Status CreateDstClusterKuduClient(const RunnerContext& context,
: client::sp::shared_ptr<KuduClient>* client) {
: return CreateKuduClient(context, kDstMasterAddressesArg, client);
: }
Doesn't seem like useful decomposotion; just call CreateKuduClient with different args directly.
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/tool_action_table.cc@423
PS5, Line 423:
: client::sp::shared_ptr<KuduClient> dst_client;
: RETURN_NOT_OK(CreateDstClusterKuduClient(context, &dst_client));
Would be nice to avoid this if copying into the same cluster.
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/tool_action_table.cc@512
PS5, Line 512: .ExtraDescription("Copy table data to another table, the two tables could be in the same "
Please update this using the suggestions I left in the commit message.
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/tool_action_table.cc@517
PS5, Line 517: .AddRequiredParameter({ kMasterAddressesArg, kMasterAddressesArgDesc })
Since this tool allows for both a source and a destination cluster, I'd override the description of this parameter with something like: "Comma-separated list of Kudu Master addresses (source) where each address is of form 'hostname:port'".
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/tool_action_table.cc@518
PS5, Line 518: .AddRequiredParameter({ kTableNameArg, "Name of the table copied from." })
Nit: reword as "Name of the source table"
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/tool_action_table.cc@519
PS5, Line 519: .AddRequiredParameter({ kDstMasterAddressesArg,
Shouldn't this parameter be optional? If not provided, we could assume an intra-cluster table copy.
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/tool_action_table.cc@520
PS5, Line 520: "The master addresses of the destination cluster "
Nit: reword as: "Comma-separated list of Kudu Master addresses (destination) where each address is of form 'hostname:port'".
--
To view, visit http://gerrit.cloudera.org:8080/12563
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
Gerrit-Change-Number: 12563
Gerrit-PatchSet: 5
Gerrit-Owner: Yingchun Lai <40...@qq.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Yingchun Lai <40...@qq.com>
Gerrit-Comment-Date: Fri, 15 Mar 2019 19:22:05 +0000
Gerrit-HasComments: Yes
[kudu-CR] [tools] Add tool to copy table data to another table
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/12563 )
Change subject: [tools] Add tool to copy table data to another table
......................................................................
Patch Set 1:
(3 comments)
Just passing through with a few questions.
http://gerrit.cloudera.org:8080/#/c/12563/1/src/kudu/common/partition.h
File src/kudu/common/partition.h:
http://gerrit.cloudera.org:8080/#/c/12563/1/src/kudu/common/partition.h@139
PS1, Line 139: struct RangeSchema {
: std::vector<ColumnId> column_ids;
: };
Does this need to be public?
http://gerrit.cloudera.org:8080/#/c/12563/1/src/kudu/common/schema.h
File src/kudu/common/schema.h:
http://gerrit.cloudera.org:8080/#/c/12563/1/src/kudu/common/schema.h@899
PS1, Line 899: void ToUserSchema() {
Is this necessary? Couldn't you get by with CopyWithoutColumnIds(), which continues to treat the Schema as immutable and returns a copy?
http://gerrit.cloudera.org:8080/#/c/12563/1/src/kudu/tools/table_scanner.cc
File src/kudu/tools/table_scanner.cc:
http://gerrit.cloudera.org:8080/#/c/12563/1/src/kudu/tools/table_scanner.cc@335
PS1, Line 335: dst_table_schema.ToUserKuduSchema();
Curious as to whether we could instead remove column IDs from client schemas, either at the time that we receive them from the server (i.e. processing a GetTableSchemaResponse), at the time that the server sends back schemas (i.e. responding to a GetTableSchema RPC), or both. Then we could transparently feed the schema a client receives into something like CreateTable.
Would that work? Or does the client need column IDs for something?
--
To view, visit http://gerrit.cloudera.org:8080/12563
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
Gerrit-Change-Number: 12563
Gerrit-PatchSet: 1
Gerrit-Owner: Yingchun Lai <40...@qq.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Yingchun Lai <40...@qq.com>
Gerrit-Comment-Date: Wed, 27 Feb 2019 17:59:38 +0000
Gerrit-HasComments: Yes
[kudu-CR] [tools] Add tool to copy table data to another table
Posted by "Yingchun Lai (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Adar Dembo, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12563
to look at the new patch set (#8).
Change subject: [tools] Add tool to copy table data to another table
......................................................................
[tools] Add tool to copy table data to another table
Copy table data to another table; the two tables could be in the same
cluster or not. The two tables must have the same table schema, but
could have different partition schemas. Alternatively, the tool can
create the new table using the same table and partition schema as the
source table.
It's useful to use this tool to copy small tables, or just copy table
schemas in a convenient way, and it's recommend to use Java client's
Backup/Restore features to copy large tables.
Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
---
M src/kudu/client/scan_batch.h
M src/kudu/client/schema.cc
M src/kudu/common/partial_row.h
M src/kudu/common/partition.h
M src/kudu/common/schema.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/test_workload.cc
M src/kudu/tools/data_gen_util.cc
M src/kudu/tools/kudu-admin-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/table_scanner.cc
M src/kudu/tools/table_scanner.h
M src/kudu/tools/tool_action_table.cc
13 files changed, 808 insertions(+), 72 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/63/12563/8
--
To view, visit http://gerrit.cloudera.org:8080/12563
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
Gerrit-Change-Number: 12563
Gerrit-PatchSet: 8
Gerrit-Owner: Yingchun Lai <40...@qq.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Yingchun Lai <40...@qq.com>
[kudu-CR] [tools] Add tool to copy table data to another table
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12563 )
Change subject: [tools] Add tool to copy table data to another table
......................................................................
[tools] Add tool to copy table data to another table
Copy table data to another table; the two tables could be in the same
cluster or not. The two tables must have the same table schema, but
could have different partition schemas. Alternatively, the tool can
create the new table using the same table and partition schema as the
source table.
It's useful to use this tool to copy small tables, or just copy table
schemas in a convenient way, and it's recommend to use Java client's
Backup/Restore features to copy large tables.
Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
Reviewed-on: http://gerrit.cloudera.org:8080/12563
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <ad...@cloudera.com>
---
M src/kudu/client/scan_batch.h
M src/kudu/client/schema.cc
M src/kudu/common/partial_row.h
M src/kudu/common/partition.h
M src/kudu/common/schema.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/test_workload.cc
M src/kudu/tools/data_gen_util.cc
M src/kudu/tools/kudu-admin-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/table_scanner.cc
M src/kudu/tools/table_scanner.h
M src/kudu/tools/tool_action_common.cc
M src/kudu/tools/tool_action_common.h
M src/kudu/tools/tool_action_perf.cc
M src/kudu/tools/tool_action_table.cc
16 files changed, 813 insertions(+), 69 deletions(-)
Approvals:
Kudu Jenkins: Verified
Adar Dembo: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/12563
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
Gerrit-Change-Number: 12563
Gerrit-PatchSet: 12
Gerrit-Owner: Yingchun Lai <40...@qq.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Yingchun Lai <40...@qq.com>
[kudu-CR] [tools] Add tool to copy table data to another table
Posted by "Yingchun Lai (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Adar Dembo, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12563
to look at the new patch set (#4).
Change subject: [tools] Add tool to copy table data to another table
......................................................................
[tools] Add tool to copy table data to another table
Copy table data to another table, the two tables could be in the same
cluster or not. The two tables must have the same table schema, but
could have different partition schemas. And also it's able to create
new table with the same table schema and partition schema with the
source table by this tool.
It's useful to use this tool to copy small tables, or just copy table
schemas in a convenient way, and it's recommend to use Java client's
Backup/Restore features to copy large tables.
Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
---
M src/kudu/client/schema.cc
M src/kudu/common/partition.h
M src/kudu/common/schema.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/test_workload.cc
M src/kudu/tools/data_gen_util.cc
M src/kudu/tools/kudu-admin-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/table_scanner.cc
M src/kudu/tools/table_scanner.h
M src/kudu/tools/tool_action_table.cc
11 files changed, 909 insertions(+), 46 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/63/12563/4
--
To view, visit http://gerrit.cloudera.org:8080/12563
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
Gerrit-Change-Number: 12563
Gerrit-PatchSet: 4
Gerrit-Owner: Yingchun Lai <40...@qq.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Yingchun Lai <40...@qq.com>
[kudu-CR] [tools] Add tool to copy table data to another table
Posted by "Yingchun Lai (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Adar Dembo, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12563
to look at the new patch set (#9).
Change subject: [tools] Add tool to copy table data to another table
......................................................................
[tools] Add tool to copy table data to another table
Copy table data to another table; the two tables could be in the same
cluster or not. The two tables must have the same table schema, but
could have different partition schemas. Alternatively, the tool can
create the new table using the same table and partition schema as the
source table.
It's useful to use this tool to copy small tables, or just copy table
schemas in a convenient way, and it's recommend to use Java client's
Backup/Restore features to copy large tables.
Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
---
M src/kudu/client/scan_batch.h
M src/kudu/client/schema.cc
M src/kudu/common/partial_row.h
M src/kudu/common/partition.h
M src/kudu/common/schema.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/test_workload.cc
M src/kudu/tools/data_gen_util.cc
M src/kudu/tools/kudu-admin-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/table_scanner.cc
M src/kudu/tools/table_scanner.h
M src/kudu/tools/tool_action_table.cc
13 files changed, 800 insertions(+), 73 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/63/12563/9
--
To view, visit http://gerrit.cloudera.org:8080/12563
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
Gerrit-Change-Number: 12563
Gerrit-PatchSet: 9
Gerrit-Owner: Yingchun Lai <40...@qq.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Yingchun Lai <40...@qq.com>
[kudu-CR] [tools] Add tool to copy table data to another table
Posted by "Yingchun Lai (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Adar Dembo, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12563
to look at the new patch set (#3).
Change subject: [tools] Add tool to copy table data to another table
......................................................................
[tools] Add tool to copy table data to another table
Copy table data to another table, the two tables could be in the same
cluster or not. The two tables must have the same table schema, but
could have different partition schemas. And also it's able to create
new table with the same table schema and partition schema with the
source table by this tool.
It's useful to use this tool to copy small tables, or just copy table
schemas in a convenient way, and it's recommend to use Java client's
Backup/Restore features to copy large tables.
Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
---
M src/kudu/client/schema.cc
M src/kudu/common/partition.h
M src/kudu/common/schema.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/test_workload.cc
M src/kudu/tools/data_gen_util.cc
M src/kudu/tools/kudu-admin-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/table_scanner.cc
M src/kudu/tools/table_scanner.h
M src/kudu/tools/tool_action_table.cc
11 files changed, 910 insertions(+), 47 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/63/12563/3
--
To view, visit http://gerrit.cloudera.org:8080/12563
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
Gerrit-Change-Number: 12563
Gerrit-PatchSet: 3
Gerrit-Owner: Yingchun Lai <40...@qq.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Yingchun Lai <40...@qq.com>
[kudu-CR] [tools] Add tool to copy table data to another table
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/12563 )
Change subject: [tools] Add tool to copy table data to another table
......................................................................
Patch Set 10: Code-Review+2
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12563/10/src/kudu/tools/kudu-tool-test.cc
File src/kudu/tools/kudu-tool-test.cc:
http://gerrit.cloudera.org:8080/#/c/12563/10/src/kudu/tools/kudu-tool-test.cc@664
PS10, Line 664: multi_args.emplace_back(args_temp);
For each of these cases, you can std::move(args_temp) into multi_args.
--
To view, visit http://gerrit.cloudera.org:8080/12563
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
Gerrit-Change-Number: 12563
Gerrit-PatchSet: 10
Gerrit-Owner: Yingchun Lai <40...@qq.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Yingchun Lai <40...@qq.com>
Gerrit-Comment-Date: Sat, 23 Mar 2019 23:07:49 +0000
Gerrit-HasComments: Yes
[kudu-CR] [tools] Add tool to copy table data to another table
Posted by "Yingchun Lai (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Adar Dembo, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12563
to look at the new patch set (#11).
Change subject: [tools] Add tool to copy table data to another table
......................................................................
[tools] Add tool to copy table data to another table
Copy table data to another table; the two tables could be in the same
cluster or not. The two tables must have the same table schema, but
could have different partition schemas. Alternatively, the tool can
create the new table using the same table and partition schema as the
source table.
It's useful to use this tool to copy small tables, or just copy table
schemas in a convenient way, and it's recommend to use Java client's
Backup/Restore features to copy large tables.
Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
---
M src/kudu/client/scan_batch.h
M src/kudu/client/schema.cc
M src/kudu/common/partial_row.h
M src/kudu/common/partition.h
M src/kudu/common/schema.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/test_workload.cc
M src/kudu/tools/data_gen_util.cc
M src/kudu/tools/kudu-admin-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/table_scanner.cc
M src/kudu/tools/table_scanner.h
M src/kudu/tools/tool_action_common.cc
M src/kudu/tools/tool_action_common.h
M src/kudu/tools/tool_action_perf.cc
M src/kudu/tools/tool_action_table.cc
16 files changed, 813 insertions(+), 69 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/63/12563/11
--
To view, visit http://gerrit.cloudera.org:8080/12563
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
Gerrit-Change-Number: 12563
Gerrit-PatchSet: 11
Gerrit-Owner: Yingchun Lai <40...@qq.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Yingchun Lai <40...@qq.com>
[kudu-CR] [tools] Add tool to copy table data to another table
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/12563 )
Change subject: [tools] Add tool to copy table data to another table
......................................................................
Patch Set 11: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/12563
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
Gerrit-Change-Number: 12563
Gerrit-PatchSet: 11
Gerrit-Owner: Yingchun Lai <40...@qq.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Yingchun Lai <40...@qq.com>
Gerrit-Comment-Date: Sun, 24 Mar 2019 18:19:03 +0000
Gerrit-HasComments: No
[kudu-CR] [tools] Add tool to copy table data to another table
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/12563 )
Change subject: [tools] Add tool to copy table data to another table
......................................................................
Patch Set 7:
(3 comments)
Nice refactoring.
http://gerrit.cloudera.org:8080/#/c/12563/7/src/kudu/tools/kudu-tool-test.cc
File src/kudu/tools/kudu-tool-test.cc:
http://gerrit.cloudera.org:8080/#/c/12563/7/src/kudu/tools/kudu-tool-test.cc@2861
PS7, Line 2861: TEST_F(ToolTest, TestCopyTableSchemaOnly) {
There's a lot of repetition between these tests. Consider making them parameterized on TableCopyMode. Search for TEST_P in this file to see how this is done.
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc
File src/kudu/tools/table_scanner.cc:
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@321
PS5, Line 321:
> Done
Whoops, I made a typo in my recommendation. Should have been "Destination table is the same as the source table". Sorry!
http://gerrit.cloudera.org:8080/#/c/12563/7/src/kudu/tools/table_scanner.cc
File src/kudu/tools/table_scanner.cc:
http://gerrit.cloudera.org:8080/#/c/12563/7/src/kudu/tools/table_scanner.cc@20
PS7, Line 20: #include <stddef.h>
: #include <stdint.h>
: #include <string.h>
Nit: we prefer C++ equivalents where they exist (i.e. <cstddef>, <cstdint>, and <cstring>).
--
To view, visit http://gerrit.cloudera.org:8080/12563
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
Gerrit-Change-Number: 12563
Gerrit-PatchSet: 7
Gerrit-Owner: Yingchun Lai <40...@qq.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Yingchun Lai <40...@qq.com>
Gerrit-Comment-Date: Thu, 21 Mar 2019 18:58:11 +0000
Gerrit-HasComments: Yes
[kudu-CR] [tools] Add tool to copy table data to another table
Posted by "Yingchun Lai (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Adar Dembo, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12563
to look at the new patch set (#5).
Change subject: [tools] Add tool to copy table data to another table
......................................................................
[tools] Add tool to copy table data to another table
Copy table data to another table, the two tables could be in the same
cluster or not. The two tables must have the same table schema, but
could have different partition schemas. And also it's able to create
new table with the same table schema and partition schema with the
source table by this tool.
It's useful to use this tool to copy small tables, or just copy table
schemas in a convenient way, and it's recommend to use Java client's
Backup/Restore features to copy large tables.
Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
---
M src/kudu/client/schema.cc
M src/kudu/common/partition.h
M src/kudu/common/schema.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/test_workload.cc
M src/kudu/tools/data_gen_util.cc
M src/kudu/tools/kudu-admin-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/table_scanner.cc
M src/kudu/tools/table_scanner.h
M src/kudu/tools/tool_action_table.cc
11 files changed, 912 insertions(+), 46 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/63/12563/5
--
To view, visit http://gerrit.cloudera.org:8080/12563
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
Gerrit-Change-Number: 12563
Gerrit-PatchSet: 5
Gerrit-Owner: Yingchun Lai <40...@qq.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Yingchun Lai <40...@qq.com>
[kudu-CR] [tools] Add tool to copy table data to another table
Posted by "Yingchun Lai (Code Review)" <ge...@cloudera.org>.
Yingchun Lai has posted comments on this change. ( http://gerrit.cloudera.org:8080/12563 )
Change subject: [tools] Add tool to copy table data to another table
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12563/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/12563/1//COMMIT_MSG@7
PS1, Line 7: [tools] Add tool to copy table data to another table
> I'm a bit nervous about providing this as a non-distributed CLI tool vs a S
Yea, in our use case, we only copy some small tables, or only table schemas to a backup cluster when we try to do some dangerous operations on the origin cluster(like masters migration).
Backup/Restore tables seem a new feature, we haven't use it ever.
--
To view, visit http://gerrit.cloudera.org:8080/12563
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
Gerrit-Change-Number: 12563
Gerrit-PatchSet: 1
Gerrit-Owner: Yingchun Lai <40...@qq.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Yingchun Lai <40...@qq.com>
Gerrit-Comment-Date: Wed, 27 Feb 2019 06:43:06 +0000
Gerrit-HasComments: Yes
[kudu-CR] [tools] Add tool to copy table data to another table
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/12563 )
Change subject: [tools] Add tool to copy table data to another table
......................................................................
Patch Set 8:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/12563/8/src/kudu/tools/kudu-tool-test.cc
File src/kudu/tools/kudu-tool-test.cc:
http://gerrit.cloudera.org:8080/#/c/12563/8/src/kudu/tools/kudu-tool-test.cc@635
PS8, Line 635: std::vector<RunCopyTableCheckArgs> GenerateArgs() {
There are more deduplication opportunities here if you're willing to make it less declarative. Like, you can declare a RunCopyTableCheckArgs, populate it with the all of the args that are the same, modify the one arg that's different, add it to a vector, add more entries to the vector for kTestCopyTablePredicates, then return it.
http://gerrit.cloudera.org:8080/#/c/12563/8/src/kudu/tools/kudu-tool-test.cc@741
PS8, Line 741: KuduSchema CreateComplexSchema() {
This function actually does a few things that may fail. Could you change it to return a Status and switch the CHECK_OK macros with RETURN_NOT_OK? You can have the KuduSchema returned via OUT parameter (i.e. KuduSchema*).
http://gerrit.cloudera.org:8080/#/c/12563/8/src/kudu/tools/kudu-tool-test.cc@3047
PS8, Line 3047: RunCopyTableCheck(arg);
Wrap in NO_FATALS().
http://gerrit.cloudera.org:8080/#/c/12563/8/src/kudu/tools/table_scanner.cc
File src/kudu/tools/table_scanner.cc:
http://gerrit.cloudera.org:8080/#/c/12563/8/src/kudu/tools/table_scanner.cc@20
PS8, Line 20: #include <cstddef>
: #include <cstdint>
: #include <cstring>
Needs to be resorted with the includes on L24-29.
--
To view, visit http://gerrit.cloudera.org:8080/12563
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
Gerrit-Change-Number: 12563
Gerrit-PatchSet: 8
Gerrit-Owner: Yingchun Lai <40...@qq.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Yingchun Lai <40...@qq.com>
Gerrit-Comment-Date: Fri, 22 Mar 2019 17:57:39 +0000
Gerrit-HasComments: Yes
[kudu-CR] [tools] Add tool to copy table data to another table
Posted by "Yingchun Lai (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Adar Dembo, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12563
to look at the new patch set (#10).
Change subject: [tools] Add tool to copy table data to another table
......................................................................
[tools] Add tool to copy table data to another table
Copy table data to another table; the two tables could be in the same
cluster or not. The two tables must have the same table schema, but
could have different partition schemas. Alternatively, the tool can
create the new table using the same table and partition schema as the
source table.
It's useful to use this tool to copy small tables, or just copy table
schemas in a convenient way, and it's recommend to use Java client's
Backup/Restore features to copy large tables.
Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
---
M src/kudu/client/scan_batch.h
M src/kudu/client/schema.cc
M src/kudu/common/partial_row.h
M src/kudu/common/partition.h
M src/kudu/common/schema.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/test_workload.cc
M src/kudu/tools/data_gen_util.cc
M src/kudu/tools/kudu-admin-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/table_scanner.cc
M src/kudu/tools/table_scanner.h
M src/kudu/tools/tool_action_table.cc
13 files changed, 800 insertions(+), 73 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/63/12563/10
--
To view, visit http://gerrit.cloudera.org:8080/12563
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
Gerrit-Change-Number: 12563
Gerrit-PatchSet: 10
Gerrit-Owner: Yingchun Lai <40...@qq.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Yingchun Lai <40...@qq.com>
[kudu-CR] [tools] Add tool to copy table data to another table
Posted by "Yingchun Lai (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Adar Dembo, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12563
to look at the new patch set (#2).
Change subject: [tools] Add tool to copy table data to another table
......................................................................
[tools] Add tool to copy table data to another table
Copy table data to another table, the two tables could be in the same
cluster or not. The two tables must have the same table schema, but
could have different partition schemas. And also it's able to create
new table with the same table schema and partition schema with the
source table by this tool.
It's useful to use this tool to copy small tables, or just copy table
schemas in a convenient way, and it's recommend to use Java client's
Backup/Restore features to copy large tables.
Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
---
M src/kudu/client/schema.cc
M src/kudu/client/schema.h
M src/kudu/common/partition.h
M src/kudu/common/schema.cc
M src/kudu/integration-tests/test_workload.cc
M src/kudu/tools/data_gen_util.cc
M src/kudu/tools/kudu-admin-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/table_scanner.cc
M src/kudu/tools/table_scanner.h
M src/kudu/tools/tool_action_table.cc
11 files changed, 922 insertions(+), 43 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/63/12563/2
--
To view, visit http://gerrit.cloudera.org:8080/12563
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
Gerrit-Change-Number: 12563
Gerrit-PatchSet: 2
Gerrit-Owner: Yingchun Lai <40...@qq.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Yingchun Lai <40...@qq.com>
[kudu-CR] [tools] Add tool to copy table data to another table
Posted by "Yingchun Lai (Code Review)" <ge...@cloudera.org>.
Yingchun Lai has posted comments on this change. ( http://gerrit.cloudera.org:8080/12563 )
Change subject: [tools] Add tool to copy table data to another table
......................................................................
Patch Set 6:
(38 comments)
http://gerrit.cloudera.org:8080/#/c/12563/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/12563/5//COMMIT_MSG@9
PS5, Line 9: ;
> Nit: use a semicolon (';') here, since the two parts of the sentence are di
Done
http://gerrit.cloudera.org:8080/#/c/12563/5//COMMIT_MSG@11
PS5, Line 11: Alternatively, the tool can
: create the new table using the same table and partition schema as the
: source table.
> Reword: "Alternatively, the tool can create the new table using the same ta
Done
http://gerrit.cloudera.org:8080/#/c/12563/1/src/kudu/common/partition.h
File src/kudu/common/partition.h:
http://gerrit.cloudera.org:8080/#/c/12563/1/src/kudu/common/partition.h@139
PS1, Line 139: struct RangeSchema {
: std::vector<ColumnId> column_ids;
: };
> Does this need to be public?
The new added range_partition_ids() return RangeSchema now, to keep it the same style with hash_partition_schemas() which return std::vector<HashBucketSchema>, so it's needed to be public now.
http://gerrit.cloudera.org:8080/#/c/12563/1/src/kudu/common/schema.h
File src/kudu/common/schema.h:
http://gerrit.cloudera.org:8080/#/c/12563/1/src/kudu/common/schema.h@899
PS1, Line 899: int find_first_is_deleted_virtual_column() const {
> Is this necessary? Couldn't you get by with CopyWithoutColumnIds(), which c
Done
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/common/schema.cc
File src/kudu/common/schema.cc:
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/common/schema.cc@a348
PS5, Line 348:
> Why do we need to lift this restriction?
Since the function name is 'CopyWithoutColumnIds', I think it's not needed to require column ids.
On the other hand, when the source schema is a client side schema which has no column ids, the CHECK will fail.
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.h
File src/kudu/tools/table_scanner.h:
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.h@55
PS5, Line 55: total_count_(0),
> You can avoid the second constructor by adding the two new params into the
Done
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.h@64
PS5, Line 64:
: private:
: enum class WorkType {
: kScan,
: kCopy
> I'd document the entire class as not thread-safe, though I don't think anyo
Done
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.h@78
PS5, Line 78: Status AddRow(const client::sp::shared_ptr<kudu::client::KuduTable>& table,
: const kud
> Should wrap these in boost::optional to emphasize their optionality.
Done
http://gerrit.cloudera.org:8080/#/c/12563/1/src/kudu/tools/table_scanner.cc
File src/kudu/tools/table_scanner.cc:
http://gerrit.cloudera.org:8080/#/c/12563/1/src/kudu/tools/table_scanner.cc@335
PS1, Line 335: return Status::OK();
> Curious as to whether we could instead remove column IDs from client schema
I searched the whole kudu project, column IDs used by client side KuduSchema seemed only in cases like iterate all columns or key columns, I agree it's reasonable to remove it. Instead, we can add some other functions to achieve this goal.
However, we must modify some client APIs(i.e. remove KuduSchema::Column(idx)), this may introduce some compatibility issues.
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc
File src/kudu/tools/table_scanner.cc:
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@110
PS5, Line 110: "For example,\n"
: R"*( ["AND", [
> Nit: reword as "How data should be copied to the destination table. Valid v
Like 'write_type=' can specify an empty value for the flag.
And there is an unit test for it, 'TEST_F(ToolTest, TestCopyTableSchemaOnly)'
gflag validator: done
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@318
PS5, Line 318: tArray(pr
> Nit: exists
Done
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@321
PS5, Line 321:
> Nit: "Destination table is as the same as the source table"
Done
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@327
PS5, Line 327:
> Nit: does NOT exist.
Done
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@329
PS5, Line 329:
> Nit: does not exist
Done
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@339
PS5, Line 339:
> Nit: rename to "convert_column_ids_to_names".
Done
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@343
PS5, Line 343: () && !s.
> Nit: emplace_back() for new code.
Done
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@384
PS5, Line 384: for (const auto& hash_partition_schema : partition_schema.hash_partition_schemas()) {
: auto hash_columns = convert_column_ids_to_names(ha
> Store these in unique_ptrs and then release() them into add_range_partition
Done
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@396
PS5, Line 396:
> Nit: "created successfully"
Done
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@401
PS5, Line 401: for (const auto& partition : partitions) {
> Can we do something simpler here where we copy the raw data from 'row' into
Done
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@403
PS5, Line 403: const auto& hash_buckets = partition.hash_buckets();
> Nit: rename to src_row and rename write_row to dst_row.
Done
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@411
PS5, Line 411: ScopedDisableRedaction no_redaction;
> This can LOG(FATAL) I think; it'd be a programming error to wind up here.
Done
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@546
PS5, Line 546:
> LOG() will automatically terminate messages with endl, so you can remove th
Done
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@552
PS5, Line 552:
> Where is this used?
outdated code, I'll remove it.
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@614
PS5, Line 614:
> I think you can reuse much of the code here. Decompose the bulk of the scan
Done
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@640
PS5, Line 640:
> Can do a simple for loop:
Done
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@644
PS5, Line 644:
> With AUTO_FLUSH_BACKGROUND why bother with the explicit Flush()? If you wan
The explicit Flush() is aid to make sure all write operations have been sent.
Close(bool force) will not work correctly here, with 'true' it will drop pending operations, while with 'false' it will return an 'IllegalState' status simply, I have to call it several times to check the result util it returns 'OK'.
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@668
PS5, Line 668:
> Seems like the bulk of this code be reused via decomposition too.
Done
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@726
PS5, Line 726:
> Nit: "Create table only."
Done
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/table_scanner.cc@749
PS5, Line 749:
> Nit: emplace_back for new code.
Done
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/tool_action_table.cc
File src/kudu/tools/tool_action_table.cc:
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/tool_action_table.cc@76
PS5, Line 76: DEFINE_string(dst_table, "",
> Doc what it means if this is the empty string (i.e. if the option is not se
Done
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/tool_action_table.cc@139
PS5, Line 139: const char* const kNewColumnNameArg = "new_column_name";
> Nit: reword as "dest_master_addresses"; the 'e' is only one extra char but
Done
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/tool_action_table.cc@152
PS5, Line 152: }
:
: Status CreateKuduClient(const RunnerContext& context,
: client::sp::shared_ptr<KuduClient>* client) {
: return CreateKuduClient(context, kMasterAddressesArg, client);
: }
:
: Status DeleteTable(const RunnerContext& context) {
:
> Doesn't seem like useful decomposotion; just call CreateKuduClient with dif
It's OK for CreateDstClusterKuduClient, however, for CreateKuduClient, I just want to keep compatible with old function parameters which is everywhere in tools code.
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/tool_action_table.cc@423
PS5, Line 423: == FindOrDie(context.required_args, kDestMasterAddressesArg)) {
: dst_client = src_client;
: } else {
> Would be nice to avoid this if copying into the same cluster.
Done
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/tool_action_table.cc@512
PS5, Line 512: unique_ptr<Action> copy_table =
> Please update this using the suggestions I left in the commit message.
Done
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/tool_action_table.cc@517
PS5, Line 517: "could have different partition schemas. Alternatively, the tool can "
> Since this tool allows for both a source and a destination cluster, I'd ove
Done
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/tool_action_table.cc@518
PS5, Line 518: "create the new table using the same table and partition schema as the "
> Nit: reword as "Name of the source table"
Done
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/tool_action_table.cc@519
PS5, Line 519: "source table.")
> Shouldn't this parameter be optional? If not provided, we could assume an i
I think it should be required. Since 'dst_table' is an optional flag, so when both 'dst_table' and kDstMasterAddressesArg are not specified, the action of this tool will copy itself to itself. And the most use case of this tool is to copy data to another cluster in my opinion.
http://gerrit.cloudera.org:8080/#/c/12563/5/src/kudu/tools/tool_action_table.cc@520
PS5, Line 520: .AddRequiredParameter({ kMasterAddressesArg,
> Nit: reword as: "Comma-separated list of Kudu Master addresses (destination
Done
--
To view, visit http://gerrit.cloudera.org:8080/12563
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
Gerrit-Change-Number: 12563
Gerrit-PatchSet: 6
Gerrit-Owner: Yingchun Lai <40...@qq.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Yingchun Lai <40...@qq.com>
Gerrit-Comment-Date: Thu, 21 Mar 2019 05:32:05 +0000
Gerrit-HasComments: Yes
[kudu-CR] [tools] Add tool to copy table data to another table
Posted by "Yingchun Lai (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Adar Dembo, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12563
to look at the new patch set (#6).
Change subject: [tools] Add tool to copy table data to another table
......................................................................
[tools] Add tool to copy table data to another table
Copy table data to another table; the two tables could be in the same
cluster or not. The two tables must have the same table schema, but
could have different partition schemas. Alternatively, the tool can
create the new table using the same table and partition schema as the
source table.
It's useful to use this tool to copy small tables, or just copy table
schemas in a convenient way, and it's recommend to use Java client's
Backup/Restore features to copy large tables.
Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
---
M src/kudu/client/scan_batch.h
M src/kudu/client/schema.cc
M src/kudu/common/partial_row.h
M src/kudu/common/partition.h
M src/kudu/common/schema.cc
M src/kudu/common/schema.h
M src/kudu/integration-tests/test_workload.cc
M src/kudu/tools/data_gen_util.cc
M src/kudu/tools/kudu-admin-test.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/table_scanner.cc
M src/kudu/tools/table_scanner.h
M src/kudu/tools/tool_action_table.cc
13 files changed, 797 insertions(+), 71 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/63/12563/6
--
To view, visit http://gerrit.cloudera.org:8080/12563
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifdec51701ac9ec57739b1a6f7c18786294642a54
Gerrit-Change-Number: 12563
Gerrit-PatchSet: 6
Gerrit-Owner: Yingchun Lai <40...@qq.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Yingchun Lai <40...@qq.com>