You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Todd Lipcon (Code Review)" <ge...@cloudera.org> on 2016/06/28 01:02:59 UTC
[kudu-CR] tablet: change default bloom filter FP rate to 0.01%
Hello Jean-Daniel Cryans,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/3517
to review the following change.
Change subject: tablet: change default bloom filter FP rate to 0.01%
......................................................................
tablet: change default bloom filter FP rate to 0.01%
The old default, 1%, was high enough that in a uniform random write workload,
we ended up needing to read in most of the key blocks even with bloom filters
enabled. On a 5 node cluster, after inserting a few billion rows, the write
throughput dropped dramatically as every batch of writes was seeking and
reading keys off disk.
In testing on the same cluster, changing the FP rate to 0.01% improved the
throughput dramatically by reducing the random reads coming off disk. The
cost is a 2x increase in bloom filter size (20 bits per key vs 10) but
20 bits is still a small percentage compared to typical row key sizes
in target applications.
Of course if an application has no random write characteristics and really
cares about disk space, this can always be flipped back.
Change-Id: I99bdd6298349a5be5f1fc3a666fe04305699e293
---
M docs/release_notes.adoc
M src/kudu/tablet/tablet.cc
2 files changed, 10 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/17/3517/1
--
To view, visit http://gerrit.cloudera.org:8080/3517
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I99bdd6298349a5be5f1fc3a666fe04305699e293
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
[kudu-CR] tablet: change default bloom filter FP rate to 0.01%
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change.
Change subject: tablet: change default bloom filter FP rate to 0.01%
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/3517/2//COMMIT_MSG
Commit Message:
PS2, Line 25: https://gist.github.com/toddlipcon/1ab9b36b7fbae10b635d3a905e1fe55a
> interesting how the changed graph seems to have a second life towards the e
yea, guess I should have mentioned that -- that was me hacking out block cache memory tracking (KUDU-1502)
--
To view, visit http://gerrit.cloudera.org:8080/3517
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I99bdd6298349a5be5f1fc3a666fe04305699e293
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes
[kudu-CR] tablet: change default bloom filter FP rate to 0.01%
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has submitted this change and it was merged.
Change subject: tablet: change default bloom filter FP rate to 0.01%
......................................................................
tablet: change default bloom filter FP rate to 0.01%
The old default, 1%, was high enough that in a uniform random write workload,
we ended up needing to read in most of the key blocks even with bloom filters
enabled. On a 5 node cluster, after inserting a few billion rows, the write
throughput dropped dramatically as every batch of writes was seeking and
reading keys off disk.
In testing on the same cluster, changing the FP rate to 0.01% improved the
throughput dramatically (>2x) by reducing the random reads coming off disk. The
cost is a 2x increase in bloom filter size (20 bits per key vs 10) but
20 bits is still a small percentage compared to typical row key sizes
in target applications.
Of course if an application has no random write characteristics and really
cares about disk space, this can always be flipped back.
Screenshots of the inserts/second graph (1hr rolling average) for these tests
are at: https://gist.github.com/toddlipcon/1ab9b36b7fbae10b635d3a905e1fe55a
Change-Id: I99bdd6298349a5be5f1fc3a666fe04305699e293
Reviewed-on: http://gerrit.cloudera.org:8080/3517
Tested-by: Kudu Jenkins
Reviewed-by: Jean-Daniel Cryans <jd...@apache.org>
Reviewed-by: David Ribeiro Alves <dr...@apache.org>
---
M docs/release_notes.adoc
M src/kudu/tablet/tablet.cc
2 files changed, 10 insertions(+), 1 deletion(-)
Approvals:
David Ribeiro Alves: Looks good to me, approved
Jean-Daniel Cryans: Looks good to me, approved
Kudu Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/3517
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I99bdd6298349a5be5f1fc3a666fe04305699e293
Gerrit-PatchSet: 3
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <dr...@apache.org>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] tablet: change default bloom filter FP rate to 0.01%
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change.
Change subject: tablet: change default bloom filter FP rate to 0.01%
......................................................................
Patch Set 1:
Build Started http://104.196.14.100/job/kudu-gerrit/2077/
--
To view, visit http://gerrit.cloudera.org:8080/3517
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I99bdd6298349a5be5f1fc3a666fe04305699e293
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-HasComments: No
[kudu-CR] tablet: change default bloom filter FP rate to 0.01%
Posted by "Jean-Daniel Cryans (Code Review)" <ge...@cloudera.org>.
Jean-Daniel Cryans has posted comments on this change.
Change subject: tablet: change default bloom filter FP rate to 0.01%
......................................................................
Patch Set 1: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/3517
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I99bdd6298349a5be5f1fc3a666fe04305699e293
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-HasComments: No