You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Todd Lipcon (Code Review)" <ge...@cloudera.org> on 2017/09/07 20:53:27 UTC
[kudu-CR] block id: use a better hash function
Hello Dan Burkert, Adar Dembo,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/8001
to review the following change.
Change subject: block_id: use a better hash function
......................................................................
block_id: use a better hash function
In looking at LBM startup time I noticed that the insertion into the block
hashmap was taking substantially longer than advertised in various benchmarks.
That led me to think that our block ID patterns were causing high collision rates.
Swapping out the hash function for block IDs reduced the startup time
substantially.
Tested on a host with ~11M blocks across 14 drives using 'fs check'.
Before:
I0907 13:01:46.997755 21274 fs_manager.cc:335] Time spent opening block manager: real 114.501s user 0.000s sys 0.002s
After:
I0907 12:58:42.863929 20320 fs_manager.cc:335] Time spent opening block manager: real 69.951s user 0.001s sys 0.001s
Change-Id: I30717955f962957d109a6403b55d59ab6c446a87
---
M src/kudu/fs/block_id.h
1 file changed, 7 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/01/8001/1
--
To view, visit http://gerrit.cloudera.org:8080/8001
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I30717955f962957d109a6403b55d59ab6c446a87
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
[kudu-CR] block id: use a better hash function
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change.
Change subject: block_id: use a better hash function
......................................................................
Patch Set 1:
Would be great if you could synthesize a unit test that exhibits the improvement too, so that if we wanted to reevaluate the choice of hash function in the future, we wouldn't need a full cluster to do so.
--
To view, visit http://gerrit.cloudera.org:8080/8001
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I30717955f962957d109a6403b55d59ab6c446a87
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-HasComments: No
[kudu-CR] block id: use a better hash function
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has abandoned this change.
Change subject: block_id: use a better hash function
......................................................................
Abandoned
--
To view, visit http://gerrit.cloudera.org:8080/8001
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: abandon
Gerrit-Change-Id: I30717955f962957d109a6403b55d59ab6c446a87
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] block id: use a better hash function
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change.
Change subject: block_id: use a better hash function
......................................................................
Patch Set 1:
I tried making a unit test and weirdly in my unit test environment, the new hash was actually worse than the old one.
I also tried reverting this but leaving my other optimizations, and it turns out in the presence of the other fixes around refcounting, etc, and using sparsepp instead of sparsehash, this doesn't make a difference. So, I'll abandon for now.
--
To view, visit http://gerrit.cloudera.org:8080/8001
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I30717955f962957d109a6403b55d59ab6c446a87
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: No