You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Todd Lipcon (Code Review)" <ge...@cloudera.org> on 2016/11/23 01:10:37 UTC

[kudu-CR] bshuf block: some low-hanging fruit optimizations on write path 

Todd Lipcon has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/5195

Change subject: bshuf_block: some low-hanging fruit optimizations on write path
......................................................................

bshuf_block: some low-hanging fruit optimizations on write path

Rather than adding an element at a time and calling the virtual
'IsBlockFull()' function, we predetermine how many elements we will
accept. This allows a much simpler batched 'Add()' implementation.

This sped up the write side of cfile-test's 100M-integer test about 2x.

Change-Id: Ia895f7731e5371967782ef9cb176a9d493894a83
---
M src/kudu/cfile/bshuf_block.h
1 file changed, 11 insertions(+), 15 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/95/5195/1
-- 
To view, visit http://gerrit.cloudera.org:8080/5195
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia895f7731e5371967782ef9cb176a9d493894a83
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>

[kudu-CR] bshuf block: some low-hanging-fruit optimizations on write path

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has submitted this change and it was merged.

Change subject: bshuf_block: some low-hanging-fruit optimizations on write path
......................................................................


bshuf_block: some low-hanging-fruit optimizations on write path

Rather than adding an element at a time and calling the virtual
'IsBlockFull()' function, we predetermine how many elements we will
accept. This allows a much simpler batched 'Add()' implementation.

This sped up the write side of cfile-test's 100M-integer test about 2x.

Change-Id: Ia895f7731e5371967782ef9cb176a9d493894a83
Reviewed-on: http://gerrit.cloudera.org:8080/5195
Reviewed-by: Dan Burkert <da...@apache.org>
Tested-by: Kudu Jenkins
---
M src/kudu/cfile/bshuf_block.h
1 file changed, 10 insertions(+), 26 deletions(-)

Approvals:
  Dan Burkert: Looks good to me, approved
  Kudu Jenkins: Verified



-- 
To view, visit http://gerrit.cloudera.org:8080/5195
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ia895f7731e5371967782ef9cb176a9d493894a83
Gerrit-PatchSet: 7
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[kudu-CR] bshuf block: some low-hanging-fruit optimizations on write path

Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has posted comments on this change.

Change subject: bshuf_block: some low-hanging-fruit optimizations on write path
......................................................................


Patch Set 5:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/5195/5//COMMIT_MSG
Commit Message:

Line 9: Rather than adding an element at a time and calling the virtual
gvint_block.cc is doing something very similar.  Perhaps it would be good to keep the implementation in-sync WRT this behavior?


http://gerrit.cloudera.org:8080/#/c/5195/5/src/kudu/cfile/bshuf_block.h
File src/kudu/cfile/bshuf_block.h:

Line 165:   size_t EstimateEncodedSize() const {
This is no longer used.


-- 
To view, visit http://gerrit.cloudera.org:8080/5195
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ia895f7731e5371967782ef9cb176a9d493894a83
Gerrit-PatchSet: 5
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-HasComments: Yes

[kudu-CR] bshuf block: some low-hanging-fruit optimizations on write path

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Hello Dan Burkert, Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/5195

to look at the new patch set (#6).

Change subject: bshuf_block: some low-hanging-fruit optimizations on write path
......................................................................

bshuf_block: some low-hanging-fruit optimizations on write path

Rather than adding an element at a time and calling the virtual
'IsBlockFull()' function, we predetermine how many elements we will
accept. This allows a much simpler batched 'Add()' implementation.

This sped up the write side of cfile-test's 100M-integer test about 2x.

Change-Id: Ia895f7731e5371967782ef9cb176a9d493894a83
---
M src/kudu/cfile/bshuf_block.h
1 file changed, 10 insertions(+), 26 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/95/5195/6
-- 
To view, visit http://gerrit.cloudera.org:8080/5195
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia895f7731e5371967782ef9cb176a9d493894a83
Gerrit-PatchSet: 6
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[kudu-CR] bshuf block: some low-hanging-fruit optimizations on write path

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Hello Dan Burkert, Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/5195

to look at the new patch set (#5).

Change subject: bshuf_block: some low-hanging-fruit optimizations on write path
......................................................................

bshuf_block: some low-hanging-fruit optimizations on write path

Rather than adding an element at a time and calling the virtual
'IsBlockFull()' function, we predetermine how many elements we will
accept. This allows a much simpler batched 'Add()' implementation.

This sped up the write side of cfile-test's 100M-integer test about 2x.

Change-Id: Ia895f7731e5371967782ef9cb176a9d493894a83
---
M src/kudu/cfile/bshuf_block.h
1 file changed, 11 insertions(+), 15 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/95/5195/5
-- 
To view, visit http://gerrit.cloudera.org:8080/5195
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia895f7731e5371967782ef9cb176a9d493894a83
Gerrit-PatchSet: 5
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot

[kudu-CR] bshuf block: some low-hanging-fruit optimizations on write path

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change.

Change subject: bshuf_block: some low-hanging-fruit optimizations on write path
......................................................................


Patch Set 5:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/5195/5/src/kudu/cfile/bshuf_block.h
File src/kudu/cfile/bshuf_block.h:

PS5, Line 93: OVERRIDE
nit: here and elsewhere in this file: consider replacing the macro with 'override'


-- 
To view, visit http://gerrit.cloudera.org:8080/5195
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ia895f7731e5371967782ef9cb176a9d493894a83
Gerrit-PatchSet: 5
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-HasComments: Yes

[kudu-CR] bshuf block: some low-hanging-fruit optimizations on write path

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/5195

to look at the new patch set (#3).

Change subject: bshuf_block: some low-hanging-fruit optimizations on write path
......................................................................

bshuf_block: some low-hanging-fruit optimizations on write path

Rather than adding an element at a time and calling the virtual
'IsBlockFull()' function, we predetermine how many elements we will
accept. This allows a much simpler batched 'Add()' implementation.

This sped up the write side of cfile-test's 100M-integer test about 2x.

Change-Id: Ia895f7731e5371967782ef9cb176a9d493894a83
---
M src/kudu/cfile/bshuf_block.h
1 file changed, 11 insertions(+), 15 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/95/5195/3
-- 
To view, visit http://gerrit.cloudera.org:8080/5195
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia895f7731e5371967782ef9cb176a9d493894a83
Gerrit-PatchSet: 3
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot

[kudu-CR] bshuf block: some low-hanging-fruit optimizations on write path

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change.

Change subject: bshuf_block: some low-hanging-fruit optimizations on write path
......................................................................


Patch Set 5:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/5195/5/src/kudu/cfile/bshuf_block.h
File src/kudu/cfile/bshuf_block.h:

Line 174:     return kHeaderSize + count_ * size_of_type;
Why don't we round up this estimate anymore?


-- 
To view, visit http://gerrit.cloudera.org:8080/5195
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ia895f7731e5371967782ef9cb176a9d493894a83
Gerrit-PatchSet: 5
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-HasComments: Yes

[kudu-CR] bshuf block: some low-hanging-fruit optimizations on write path

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change.

Change subject: bshuf_block: some low-hanging-fruit optimizations on write path
......................................................................


Patch Set 5:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/5195/5//COMMIT_MSG
Commit Message:

Line 9: Rather than adding an element at a time and calling the virtual
> gvint_block.cc is doing something very similar.  Perhaps it would be good t
I think gvint is basically dead except for test codepaths, since it's only usable for UINT32, and users can't specify uint32 anymore


http://gerrit.cloudera.org:8080/#/c/5195/5/src/kudu/cfile/bshuf_block.h
File src/kudu/cfile/bshuf_block.h:

PS5, Line 93: OVERRIDE
> nit: here and elsewhere in this file: consider replacing the macro with 'ov
would rather do this separately


Line 174:     return kHeaderSize + count_ * size_of_type;
> Why don't we round up this estimate anymore?
not worth it to improve estimate by ~30 bytes max


-- 
To view, visit http://gerrit.cloudera.org:8080/5195
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ia895f7731e5371967782ef9cb176a9d493894a83
Gerrit-PatchSet: 5
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes

[kudu-CR] bshuf block: some low-hanging-fruit optimizations on write path

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/5195

to look at the new patch set (#2).

Change subject: bshuf_block: some low-hanging-fruit optimizations on write path
......................................................................

bshuf_block: some low-hanging-fruit optimizations on write path

Rather than adding an element at a time and calling the virtual
'IsBlockFull()' function, we predetermine how many elements we will
accept. This allows a much simpler batched 'Add()' implementation.

This sped up the write side of cfile-test's 100M-integer test about 2x.

Change-Id: Ia895f7731e5371967782ef9cb176a9d493894a83
---
M src/kudu/cfile/bshuf_block.h
1 file changed, 11 insertions(+), 15 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/95/5195/2
-- 
To view, visit http://gerrit.cloudera.org:8080/5195
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia895f7731e5371967782ef9cb176a9d493894a83
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot

[kudu-CR] bshuf block: some low-hanging-fruit optimizations on write path

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Hello David Ribeiro Alves, Adar Dembo, Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/5195

to look at the new patch set (#4).

Change subject: bshuf_block: some low-hanging-fruit optimizations on write path
......................................................................

bshuf_block: some low-hanging-fruit optimizations on write path

Rather than adding an element at a time and calling the virtual
'IsBlockFull()' function, we predetermine how many elements we will
accept. This allows a much simpler batched 'Add()' implementation.

This sped up the write side of cfile-test's 100M-integer test about 2x.

Change-Id: Ia895f7731e5371967782ef9cb176a9d493894a83
---
M src/kudu/cfile/bshuf_block.h
1 file changed, 11 insertions(+), 15 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/95/5195/4
-- 
To view, visit http://gerrit.cloudera.org:8080/5195
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia895f7731e5371967782ef9cb176a9d493894a83
Gerrit-PatchSet: 4
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot

[kudu-CR] bshuf block: some low-hanging-fruit optimizations on write path

Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has posted comments on this change.

Change subject: bshuf_block: some low-hanging-fruit optimizations on write path
......................................................................


Patch Set 6: Code-Review+2

-- 
To view, visit http://gerrit.cloudera.org:8080/5195
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ia895f7731e5371967782ef9cb176a9d493894a83
Gerrit-PatchSet: 6
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: No