You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Todd Lipcon (Code Review)" <ge...@cloudera.org> on 2018/04/30 17:24:13 UTC
[kudu-CR] benchmarks: update parameters for mt-bloomfile benchmark
Todd Lipcon has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10242
Change subject: benchmarks: update parameters for mt-bloomfile benchmark
......................................................................
benchmarks: update parameters for mt-bloomfile benchmark
The old parameters were somewhat unrealistic:
- The default bloom block size is 4kb, but it was using 32kb.
- The cache capacity was set to 1MB, which meant that each shard on a
multicore system was only 50KB or less. With the new optimization that
allows 1% slack in MemTracker accounting, that slack was only
500bytes or less. With 4kb bloom blocks, that optimization was
effectively disabled.
The new parameters are slightly more realistic while keeping the
performance reasonable: 100MB cache capacity (>1MB per shard even on 88
core system) and 4kb blocks. Runtime is still reasonable - 4-5 seconds
for the query phase. The number of unique keys is bumped up to 150M to
ensure cache churn.
Change-Id: I0a85fe222285ee20d1e21ddf260cdb50c4994e96
---
M src/kudu/scripts/benchmarks.sh
1 file changed, 5 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/42/10242/1
--
To view, visit http://gerrit.cloudera.org:8080/10242
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I0a85fe222285ee20d1e21ddf260cdb50c4994e96
Gerrit-Change-Number: 10242
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon <to...@apache.org>
[kudu-CR] benchmarks: update parameters for mt-bloomfile benchmark
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10242 )
Change subject: benchmarks: update parameters for mt-bloomfile benchmark
......................................................................
benchmarks: update parameters for mt-bloomfile benchmark
The old parameters were somewhat unrealistic:
- The default bloom block size is 4kb, but it was using 32kb.
- The cache capacity was set to 1MB, which meant that each shard on a
multicore system was only 50KB or less. With the new optimization that
allows 1% slack in MemTracker accounting, that slack was only
500bytes or less. With 4kb bloom blocks, that optimization was
effectively disabled.
The new parameters are slightly more realistic while keeping the
performance reasonable: 100MB cache capacity (>1MB per shard even on 88
core system) and 4kb blocks. Runtime is still reasonable - 4-5 seconds
for the query phase. The number of unique keys is bumped up to 150M to
ensure cache churn.
Change-Id: I0a85fe222285ee20d1e21ddf260cdb50c4994e96
Reviewed-on: http://gerrit.cloudera.org:8080/10242
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Tested-by: Kudu Jenkins
---
M src/kudu/scripts/benchmarks.sh
1 file changed, 5 insertions(+), 3 deletions(-)
Approvals:
Adar Dembo: Looks good to me, approved
Kudu Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/10242
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I0a85fe222285ee20d1e21ddf260cdb50c4994e96
Gerrit-Change-Number: 10242
Gerrit-PatchSet: 2
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] benchmarks: update parameters for mt-bloomfile benchmark
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/10242 )
Change subject: benchmarks: update parameters for mt-bloomfile benchmark
......................................................................
Patch Set 1: Code-Review+2
What tipped you off that the parameters needed to be tweaked?
--
To view, visit http://gerrit.cloudera.org:8080/10242
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0a85fe222285ee20d1e21ddf260cdb50c4994e96
Gerrit-Change-Number: 10242
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Comment-Date: Mon, 30 Apr 2018 18:08:46 +0000
Gerrit-HasComments: No
[kudu-CR] benchmarks: update parameters for mt-bloomfile benchmark
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10242 )
Change subject: benchmarks: update parameters for mt-bloomfile benchmark
......................................................................
Patch Set 1:
The performance of the benchmark got slower when I pushed the improvement for memtracker contention in the LRU cache :)
--
To view, visit http://gerrit.cloudera.org:8080/10242
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0a85fe222285ee20d1e21ddf260cdb50c4994e96
Gerrit-Change-Number: 10242
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Mon, 30 Apr 2018 18:58:18 +0000
Gerrit-HasComments: No