You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Wang Xixu (Code Review)" <ge...@cloudera.org> on 2024/06/18 09:01:35 UTC

[kudu-CR] [Tool] Limit table copying speed

Wang Xixu has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21527


Change subject: [Tool] Limit table copying speed
......................................................................

[Tool] Limit table copying speed

Migrating data to another Kudu cluster using CLI command
'kudu table copy' When the data is very large, it may cause
memory pressure or run out the network bandwidth. To reduce
the affect on other service, it is better to limit the copying
speed.

This patch introduces two parameter:
--table_copy_throttler_bytes_per_sec and
--table_copy_throttler_burst_factor to limit the table copying
speed.

Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
---
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/table_scanner.cc
M src/kudu/tools/table_scanner.h
M src/kudu/tools/tool_action_table.cc
4 files changed, 78 insertions(+), 0 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/27/21527/1
-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 1
Gerrit-Owner: Wang Xixu <14...@qq.com>

[kudu-CR] [Tool] Limit table copying speed

Posted by "Yifan Zhang (Code Review)" <ge...@cloudera.org>.
Yifan Zhang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21527 )

Change subject: [Tool] Limit table copying speed
......................................................................

[Tool] Limit table copying speed

Migrating data to another Kudu cluster using the 'kudu table copy'
CLI command when the data is very large may cause memory and/or
network bandwidth perssure. To reduce the effect on other services,
it is better to limit the copying speed.

This patch introduces two parameter:
--table_copy_throttler_bytes_per_sec and
--table_copy_throttler_burst_factor to limit the table copying
speed.

Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Reviewed-on: http://gerrit.cloudera.org:8080/21527
Reviewed-by: Zoltan Chovan <zc...@cloudera.com>
Reviewed-by: Abhishek Chennaka <ac...@cloudera.com>
Tested-by: Yifan Zhang <ch...@163.com>
Reviewed-by: Yifan Zhang <ch...@163.com>
---
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/table_scanner.cc
M src/kudu/tools/table_scanner.h
M src/kudu/tools/tool_action_table.cc
4 files changed, 81 insertions(+), 0 deletions(-)

Approvals:
  Zoltan Chovan: Looks good to me, but someone else must approve
  Abhishek Chennaka: Looks good to me, approved
  Yifan Zhang: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 6
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>

[kudu-CR] [Tool] Limit table copying speed

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 )

Change subject: [Tool] Limit table copying speed
......................................................................


Patch Set 2:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/265/


-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 2
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Fri, 21 Jun 2024 03:16:39 +0000
Gerrit-HasComments: No

[kudu-CR] [Tool] Limit table copying speed

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 )

Change subject: [Tool] Limit table copying speed
......................................................................


Patch Set 4:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/284/


-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 4
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Tue, 02 Jul 2024 06:14:03 +0000
Gerrit-HasComments: No

[kudu-CR] [Tool] Limit table copying speed

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 )

Change subject: [Tool] Limit table copying speed
......................................................................


Patch Set 3: Verified-1

Build Failed 

http://jenkins.kudu.apache.org/job/pre_commit/279/ : FAILURE


-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 3
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Mon, 01 Jul 2024 03:08:02 +0000
Gerrit-HasComments: No

[kudu-CR] [Tool] Limit table copying speed

Posted by "Wang Xixu (Code Review)" <ge...@cloudera.org>.
Hello Marton Greber, Zoltan Chovan, Zoltan Martonka, Attila Bukor, Kudu Jenkins, Abhishek Chennaka, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/21527

to look at the new patch set (#3).

Change subject: [Tool] Limit table copying speed
......................................................................

[Tool] Limit table copying speed

Migrating data to another Kudu cluster using the 'kudu table copy'
CLI command when the data is very large may cause memory and/or
network bandwidth perssure. To reduce the effect on other services,
it is better to limit the copying speed.

This patch introduces two parameter:
--table_copy_throttler_bytes_per_sec and
--table_copy_throttler_burst_factor to limit the table copying
speed.

Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
---
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/table_scanner.cc
M src/kudu/tools/table_scanner.h
M src/kudu/tools/tool_action_table.cc
4 files changed, 79 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/27/21527/3
-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 3
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>

[kudu-CR] [Tool] Limit table copying speed

Posted by "Wang Xixu (Code Review)" <ge...@cloudera.org>.
Wang Xixu has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 )

Change subject: [Tool] Limit table copying speed
......................................................................


Patch Set 2:

(8 comments)

http://gerrit.cloudera.org:8080/#/c/21527/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/21527/1//COMMIT_MSG@10
PS1, Line 10: se
> nit: 'it' not needed
Done


http://gerrit.cloudera.org:8080/#/c/21527/1//COMMIT_MSG@9
PS1, Line 9: using the 'kudu table copy'
           : CLI command when t
> reword: using the 'kudu table copy' CLI command
Done


http://gerrit.cloudera.org:8080/#/c/21527/1//COMMIT_MSG@10
PS1, Line 10: he d
> nit: when
Done


http://gerrit.cloudera.org:8080/#/c/21527/1//COMMIT_MSG@11
PS1, Line 11: network bandwidth perssure. To reduce the effect 
> memory and/or network bandwidth perssure
Done


http://gerrit.cloudera.org:8080/#/c/21527/1//COMMIT_MSG@12
PS1, Line 12: s better to limit the c
> effect on other services
Done


http://gerrit.cloudera.org:8080/#/c/21527/1/src/kudu/tools/kudu-tool-test.cc
File src/kudu/tools/kudu-tool-test.cc:

http://gerrit.cloudera.org:8080/#/c/21527/1/src/kudu/tools/kudu-tool-test.cc@5942
PS1, Line 5942: "--table_copy_throttler_burst_factor=100",
> consider extracting this to it's own variable as it is used multiple times 
Done


http://gerrit.cloudera.org:8080/#/c/21527/1/src/kudu/tools/table_scanner.cc
File src/kudu/tools/table_scanner.cc:

http://gerrit.cloudera.org:8080/#/c/21527/1/src/kudu/tools/table_scanner.cc@150
PS1, Line 150: maximum
> nit:maximum
Done


http://gerrit.cloudera.org:8080/#/c/21527/1/src/kudu/tools/table_scanner.cc@150
PS1, Line 150: equal 
> nit:equal
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 2
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Fri, 21 Jun 2024 03:16:44 +0000
Gerrit-HasComments: Yes

[kudu-CR] [Tool] Limit table copying speed

Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Abhishek Chennaka has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 )

Change subject: [Tool] Limit table copying speed
......................................................................


Patch Set 5: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 5
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Wed, 10 Jul 2024 17:32:51 +0000
Gerrit-HasComments: No

[kudu-CR] [Tool] Limit table copying speed

Posted by "Yifan Zhang (Code Review)" <ge...@cloudera.org>.
Yifan Zhang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 )

Change subject: [Tool] Limit table copying speed
......................................................................


Patch Set 5: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 5
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Jul 2024 12:00:01 +0000
Gerrit-HasComments: No

[kudu-CR] [Tool] Limit table copying speed

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 )

Change subject: [Tool] Limit table copying speed
......................................................................


Patch Set 4: Verified-1

Build Failed 

http://jenkins.kudu.apache.org/job/pre_commit/284/ : FAILURE


-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 4
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Tue, 02 Jul 2024 06:25:22 +0000
Gerrit-HasComments: No

[kudu-CR] [Tool] Limit table copying speed

Posted by "Yifan Zhang (Code Review)" <ge...@cloudera.org>.
Yifan Zhang has removed a vote on this change.

Change subject: [Tool] Limit table copying speed
......................................................................


Removed Verified-1 by Kudu Jenkins (120)
-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 5
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>

[kudu-CR] [Tool] Limit table copying speed

Posted by "Wang Xixu (Code Review)" <ge...@cloudera.org>.
Wang Xixu has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 )

Change subject: [Tool] Limit table copying speed
......................................................................


Patch Set 6:

(9 comments)

http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/kudu-tool-test.cc
File src/kudu/tools/kudu-tool-test.cc:

http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/kudu-tool-test.cc@6011
PS6, Line 6011: 10240
> Are you sure this is small enough to track copying of 20000 rows, especiall
Actually, the data size is 881500. Every batch size is about 5248, total 170 batches is readout. So, every 2 batches will be block as the table_copy_throttler_bytes_per_sec is 10240. That will block for at least 85 seconds. Please see my next patch: https://gerrit.cloudera.org/c/21609


http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/kudu-tool-test.cc@6029
PS6, Line 6029: client->OpenTable(kNewTableName, &table);
> Wrap this into ASSERT_OK()?
Done


http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/kudu-tool-test.cc@6031
PS6, Line 6031: scanner.Open();
> Wrap this into ASSERT_OK()?
Done


http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/kudu-tool-test.cc@6036
PS6, Line 6036: =
> Shouldn't this be '+=' instead of '='?  With the current code, the assertio
Sorry, that is a mistake.


http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/kudu-tool-test.cc@6038
PS6, Line 6038: must
> nit: must be
Done


http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.h
File src/kudu/tools/table_scanner.h:

http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.h@107
PS6, Line 107: std::shared_ptr
> Why shared_ptr? There isn't any shared ownership for this field, this is so
You are right. Thanks.


http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.cc
File src/kudu/tools/table_scanner.cc:

http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.cc@611
PS6, Line 611: copy
> nit: copying
Done


http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.cc@613
PS6, Line 613: SCOPED_LOG_SLOW_EXECUTION
> What's the purpose of this warning?  If the idea was to track cases when it
Done


http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.cc@614
PS6, Line 614: 0
> Ah, this 0 is for op tokens, and the 'bytes' consumed is set as necessary, 
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 6
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Jul 2024 10:17:46 +0000
Gerrit-HasComments: Yes

[kudu-CR] [Tool] Limit table copying speed

Posted by "Zoltan Chovan (Code Review)" <ge...@cloudera.org>.
Zoltan Chovan has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 )

Change subject: [Tool] Limit table copying speed
......................................................................


Patch Set 2: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 2
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Fri, 21 Jun 2024 13:52:25 +0000
Gerrit-HasComments: No

[kudu-CR] [Tool] Limit table copying speed

Posted by "Wang Xixu (Code Review)" <ge...@cloudera.org>.
Hello Zoltan Chovan, Kudu Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/21527

to look at the new patch set (#2).

Change subject: [Tool] Limit table copying speed
......................................................................

[Tool] Limit table copying speed

Migrating data to another Kudu cluster using the 'kudu table copy'
CLI command when the data is very large may cause memory and/or
network bandwidth perssure. To reduce the effect on other services,
it is better to limit the copying speed.

This patch introduces two parameter:
--table_copy_throttler_bytes_per_sec and
--table_copy_throttler_burst_factor to limit the table copying
speed.

Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
---
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/table_scanner.cc
M src/kudu/tools/table_scanner.h
M src/kudu/tools/tool_action_table.cc
4 files changed, 79 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/27/21527/2
-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 2
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>

[kudu-CR] [Tool] Limit table copying speed

Posted by "Wang Xixu (Code Review)" <ge...@cloudera.org>.
Hello Marton Greber, Zoltan Chovan, Zoltan Martonka, Attila Bukor, Kudu Jenkins, Abhishek Chennaka, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/21527

to look at the new patch set (#5).

Change subject: [Tool] Limit table copying speed
......................................................................

[Tool] Limit table copying speed

Migrating data to another Kudu cluster using the 'kudu table copy'
CLI command when the data is very large may cause memory and/or
network bandwidth perssure. To reduce the effect on other services,
it is better to limit the copying speed.

This patch introduces two parameter:
--table_copy_throttler_bytes_per_sec and
--table_copy_throttler_burst_factor to limit the table copying
speed.

Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
---
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/table_scanner.cc
M src/kudu/tools/table_scanner.h
M src/kudu/tools/tool_action_table.cc
4 files changed, 81 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/27/21527/5
-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 5
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>

[kudu-CR] [Tool] Limit table copying speed

Posted by "Zoltan Chovan (Code Review)" <ge...@cloudera.org>.
Zoltan Chovan has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 )

Change subject: [Tool] Limit table copying speed
......................................................................


Patch Set 1:

(8 comments)

http://gerrit.cloudera.org:8080/#/c/21527/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/21527/1//COMMIT_MSG@10
PS1, Line 10: When
nit: when


http://gerrit.cloudera.org:8080/#/c/21527/1//COMMIT_MSG@9
PS1, Line 9: using CLI command
           : 'kudu table copy' 
reword: using the 'kudu table copy' CLI command


http://gerrit.cloudera.org:8080/#/c/21527/1//COMMIT_MSG@10
PS1, Line 10: it
nit: 'it' not needed


http://gerrit.cloudera.org:8080/#/c/21527/1//COMMIT_MSG@11
PS1, Line 11: memory pressure or run out the network bandwidth.
memory and/or network bandwidth perssure


http://gerrit.cloudera.org:8080/#/c/21527/1//COMMIT_MSG@12
PS1, Line 12: affect on other service
effect on other services


http://gerrit.cloudera.org:8080/#/c/21527/1/src/kudu/tools/kudu-tool-test.cc
File src/kudu/tools/kudu-tool-test.cc:

http://gerrit.cloudera.org:8080/#/c/21527/1/src/kudu/tools/kudu-tool-test.cc@5942
PS1, Line 5942: cluster_->master()->bound_rpc_addr().ToString()
consider extracting this to it's own variable as it is used multiple times in the test, e.g. string master_addr = cluster_->master()->bound_rpc_addr().ToString();


http://gerrit.cloudera.org:8080/#/c/21527/1/src/kudu/tools/table_scanner.cc
File src/kudu/tools/table_scanner.cc:

http://gerrit.cloudera.org:8080/#/c/21527/1/src/kudu/tools/table_scanner.cc@150
PS1, Line 150: maximun
nit:maximum


http://gerrit.cloudera.org:8080/#/c/21527/1/src/kudu/tools/table_scanner.cc@150
PS1, Line 150: equals
nit:equal



-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 1
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Thu, 20 Jun 2024 09:56:04 +0000
Gerrit-HasComments: Yes

[kudu-CR] [Tool] Limit table copying speed

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 )

Change subject: [Tool] Limit table copying speed
......................................................................


Patch Set 5: Verified-1

Build Failed 

http://jenkins.kudu.apache.org/job/pre_commit/285/ : FAILURE


-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 5
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Tue, 02 Jul 2024 07:40:05 +0000
Gerrit-HasComments: No

[kudu-CR] [Tool] Limit table copying speed

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 )

Change subject: [Tool] Limit table copying speed
......................................................................


Patch Set 6:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.cc
File src/kudu/tools/table_scanner.cc:

http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.cc@614
PS6, Line 614: 0
> Why 0?  Isn't this a typo?  Would be great to at least add a comment on thi
Ah, this 0 is for op tokens, and the 'bytes' consumed is set as necessary, so this one should be good.  Please ignore this particular comment.



-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 6
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Jul 2024 18:48:17 +0000
Gerrit-HasComments: Yes

[kudu-CR] [Tool] Limit table copying speed

Posted by "Yifan Zhang (Code Review)" <ge...@cloudera.org>.
Yifan Zhang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 )

Change subject: [Tool] Limit table copying speed
......................................................................


Patch Set 5: Verified+1

Unrelated test failures.


-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 5
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Jul 2024 11:59:52 +0000
Gerrit-HasComments: No

[kudu-CR] [Tool] Limit table copying speed

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 )

Change subject: [Tool] Limit table copying speed
......................................................................


Patch Set 1:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/252/


-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 1
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 18 Jun 2024 09:01:45 +0000
Gerrit-HasComments: No

[kudu-CR] [Tool] Limit table copying speed

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 )

Change subject: [Tool] Limit table copying speed
......................................................................


Patch Set 2: Verified-1

Build Failed 

http://jenkins.kudu.apache.org/job/pre_commit/265/ : FAILURE


-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 2
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Comment-Date: Fri, 21 Jun 2024 03:36:12 +0000
Gerrit-HasComments: No

[kudu-CR] [Tool] Limit table copying speed

Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Abhishek Chennaka has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 )

Change subject: [Tool] Limit table copying speed
......................................................................


Patch Set 2:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/21527/2/src/kudu/tools/table_scanner.cc
File src/kudu/tools/table_scanner.cc:

http://gerrit.cloudera.org:8080/#/c/21527/2/src/kudu/tools/table_scanner.cc@146
PS2, Line 146: seconds
nit: second


http://gerrit.cloudera.org:8080/#/c/21527/2/src/kudu/tools/table_scanner.cc@149
PS2, Line 149: multiply
nit: multiplied by


http://gerrit.cloudera.org:8080/#/c/21527/2/src/kudu/tools/table_scanner.cc@150
PS2, Line 150: base rate
nit: base rate (--table_copy_throttler_bytes_per_sec).



-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 2
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Wed, 26 Jun 2024 23:47:45 +0000
Gerrit-HasComments: Yes

[kudu-CR] [Tool] Limit table copying speed

Posted by "Wang Xixu (Code Review)" <ge...@cloudera.org>.
Wang Xixu has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 )

Change subject: [Tool] Limit table copying speed
......................................................................


Patch Set 3:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/21527/2/src/kudu/tools/table_scanner.cc
File src/kudu/tools/table_scanner.cc:

http://gerrit.cloudera.org:8080/#/c/21527/2/src/kudu/tools/table_scanner.cc@146
PS2, Line 146: second"
> nit: second
Done


http://gerrit.cloudera.org:8080/#/c/21527/2/src/kudu/tools/table_scanner.cc@149
PS2, Line 149: multipli
> nit: multiplied by
Done


http://gerrit.cloudera.org:8080/#/c/21527/2/src/kudu/tools/table_scanner.cc@150
PS2, Line 150: base rate
> nit: base rate (--table_copy_throttler_bytes_per_sec).
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 3
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Mon, 01 Jul 2024 02:57:56 +0000
Gerrit-HasComments: Yes

[kudu-CR] [Tool] Limit table copying speed

Posted by "Wang Xixu (Code Review)" <ge...@cloudera.org>.
Hello Marton Greber, Zoltan Chovan, Zoltan Martonka, Attila Bukor, Kudu Jenkins, Abhishek Chennaka, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/21527

to look at the new patch set (#4).

Change subject: [Tool] Limit table copying speed
......................................................................

[Tool] Limit table copying speed

Migrating data to another Kudu cluster using the 'kudu table copy'
CLI command when the data is very large may cause memory and/or
network bandwidth perssure. To reduce the effect on other services,
it is better to limit the copying speed.

This patch introduces two parameter:
--table_copy_throttler_bytes_per_sec and
--table_copy_throttler_burst_factor to limit the table copying
speed.

Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
---
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/table_scanner.cc
M src/kudu/tools/table_scanner.h
M src/kudu/tools/tool_action_table.cc
4 files changed, 79 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/27/21527/4
-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 4
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>

[kudu-CR] [Tool] Limit table copying speed

Posted by "Zoltan Chovan (Code Review)" <ge...@cloudera.org>.
Zoltan Chovan has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 )

Change subject: [Tool] Limit table copying speed
......................................................................


Patch Set 5: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 5
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Tue, 09 Jul 2024 09:02:39 +0000
Gerrit-HasComments: No

[kudu-CR] [Tool] Limit table copying speed

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 )

Change subject: [Tool] Limit table copying speed
......................................................................


Patch Set 1: Verified-1

Build Failed 

http://jenkins.kudu.apache.org/job/pre_commit/252/ : FAILURE


-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 1
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 18 Jun 2024 12:08:41 +0000
Gerrit-HasComments: No

[kudu-CR] [Tool] Limit table copying speed

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 )

Change subject: [Tool] Limit table copying speed
......................................................................


Patch Set 3:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/279/


-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 3
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Mon, 01 Jul 2024 02:57:49 +0000
Gerrit-HasComments: No

[kudu-CR] [Tool] Limit table copying speed

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 )

Change subject: [Tool] Limit table copying speed
......................................................................


Patch Set 5:

Build Started http://jenkins.kudu.apache.org/job/pre_commit/285/


-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 5
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Tue, 02 Jul 2024 06:29:33 +0000
Gerrit-HasComments: No

[kudu-CR] [Tool] Limit table copying speed

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/21527 )

Change subject: [Tool] Limit table copying speed
......................................................................


Patch Set 6:

(9 comments)

I'm not sure this works as expected in PS6.

http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/kudu-tool-test.cc
File src/kudu/tools/kudu-tool-test.cc:

http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/kudu-tool-test.cc@6011
PS6, Line 6011: 10240
Are you sure this is small enough to track copying of 20000 rows, especially of the data is compressed?


http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/kudu-tool-test.cc@6029
PS6, Line 6029: client->OpenTable(kNewTableName, &table);
Wrap this into ASSERT_OK()?


http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/kudu-tool-test.cc@6031
PS6, Line 6031: scanner.Open();
Wrap this into ASSERT_OK()?


http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/kudu-tool-test.cc@6036
PS6, Line 6036: =
Shouldn't this be '+=' instead of '='?  With the current code, the assertion at 6039 doesn't make much sense if 'data_size' just tracks the size of the very last batch.


http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/kudu-tool-test.cc@6038
PS6, Line 6038: must
nit: must be


http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.h
File src/kudu/tools/table_scanner.h:

http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.h@107
PS6, Line 107: std::shared_ptr
Why shared_ptr? There isn't any shared ownership for this field, this is solely own by TableScanner instance.

Should this be std::unique_ptr instead?


http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.cc
File src/kudu/tools/table_scanner.cc:

http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.cc@611
PS6, Line 611: copy
nit: copying


http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.cc@613
PS6, Line 613: SCOPED_LOG_SLOW_EXECUTION
What's the purpose of this warning?  If the idea was to track cases when it takes a long time to consume a throttler's token, semantically that seems closer to INFO, not WARNING since it's expected to throttle as configured.


http://gerrit.cloudera.org:8080/#/c/21527/6/src/kudu/tools/table_scanner.cc@614
PS6, Line 614: 0
Why 0?  Isn't this a typo?  Would be great to at least add a comment on this -- I'd expect this to be at least 1.



-- 
To view, visit http://gerrit.cloudera.org:8080/21527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I37d23f6f5158618f91b67528e152cf2ff4cf38f3
Gerrit-Change-Number: 21527
Gerrit-PatchSet: 6
Gerrit-Owner: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Reviewer: Wang Xixu <14...@qq.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Jul 2024 18:18:27 +0000
Gerrit-HasComments: Yes