You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "shauryachats (via GitHub)" <gi...@apache.org> on 2025/02/05 04:40:07 UTC
[PR] Adding perf benchmark logic for GroupIdGenerator hash map [pinot]
shauryachats opened a new pull request, #14992:
URL: https://github.com/apache/pinot/pull/14992
This PR adds performance benchmarking logic to identify and measure the improvement of different strategies for hash map selection, to make a data-driven choice on the hash map used to power `GroupIdGenerator`, which has been described in the issue https://github.com/apache/pinot/issues/14685.
The results of this benchmark are:
```
Benchmark (_cardinality) Mode Cnt Score Error Units
BenchmarkObjectOpenHashMap.object2IntOpenHashMap 500000 avgt 20 111.262 ± 5.448 ms/op
BenchmarkObjectOpenHashMap.object2IntOpenHashMap 1000000 avgt 20 299.255 ± 9.814 ms/op
BenchmarkObjectOpenHashMap.object2IntOpenHashMap 5000000 avgt 20 1859.503 ± 58.990 ms/op
BenchmarkObjectOpenHashMap.object2IntOpenHashMap 20000000 avgt 20 8236.525 ± 170.751 ms/op
BenchmarkObjectOpenHashMap.object2IntReservedOpenHashMap 500000 avgt 20 79.908 ± 4.715 ms/op
BenchmarkObjectOpenHashMap.object2IntReservedOpenHashMap 1000000 avgt 20 180.827 ± 19.987 ms/op
BenchmarkObjectOpenHashMap.object2IntReservedOpenHashMap 5000000 avgt 20 1051.368 ± 49.204 ms/op
BenchmarkObjectOpenHashMap.object2IntReservedOpenHashMap 20000000 avgt 20 3340.668 ± 106.874 ms/op
BenchmarkObjectOpenHashMap.vanillaHashMap 500000 avgt 20 109.589 ± 2.836 ms/op
BenchmarkObjectOpenHashMap.vanillaHashMap 1000000 avgt 20 265.262 ± 5.215 ms/op
BenchmarkObjectOpenHashMap.vanillaHashMap 5000000 avgt 20 1556.399 ± 49.787 ms/op
BenchmarkObjectOpenHashMap.vanillaHashMap 20000000 avgt 20 6757.234 ± 314.138 ms/op
BenchmarkObjectOpenHashMap.vanillaReservedHashMap 500000 avgt 20 98.798 ± 4.344 ms/op
BenchmarkObjectOpenHashMap.vanillaReservedHashMap 1000000 avgt 20 228.480 ± 6.570 ms/op
BenchmarkObjectOpenHashMap.vanillaReservedHashMap 5000000 avgt 20 1067.580 ± 48.764 ms/op
BenchmarkObjectOpenHashMap.vanillaReservedHashMap 20000000 avgt 20 4725.897 ± 284.449 ms/op
```
which yields that the reserved hashmap performs ~2.5x better than the current unreserved hashmap, which led to the following PR: https://github.com/apache/pinot/pull/14981.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Adding perf benchmark logic for GroupIdGenerator hash map [pinot]
Posted by "shauryachats (via GitHub)" <gi...@apache.org>.
shauryachats commented on code in PR #14992:
URL: https://github.com/apache/pinot/pull/14992#discussion_r1978123127
##########
pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkObjectOpenHashMap.java:
##########
@@ -0,0 +1,130 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.perf;
+
+import it.unimi.dsi.fastutil.objects.Object2IntOpenHashMap;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.Warmup;
+import org.openjdk.jmh.runner.Runner;
+import org.openjdk.jmh.runner.options.OptionsBuilder;
+
+
+@OutputTimeUnit(TimeUnit.MILLISECONDS)
+@Fork(1)
+@Warmup(iterations = 5, time = 10)
+@Measurement(iterations = 20, time = 10)
+@State(Scope.Benchmark)
+public class BenchmarkObjectOpenHashMap {
Review Comment:
Added context and instructions to run the benchmark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Adding perf benchmark logic for GroupIdGenerator hash map [pinot]
Posted by "ankitsultana (via GitHub)" <gi...@apache.org>.
ankitsultana merged PR #14992:
URL: https://github.com/apache/pinot/pull/14992
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Adding perf benchmark logic for GroupIdGenerator hash map [pinot]
Posted by "ankitsultana (via GitHub)" <gi...@apache.org>.
ankitsultana commented on code in PR #14992:
URL: https://github.com/apache/pinot/pull/14992#discussion_r1978093749
##########
pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkObjectOpenHashMap.java:
##########
@@ -0,0 +1,130 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.perf;
+
+import it.unimi.dsi.fastutil.objects.Object2IntOpenHashMap;
+import java.util.HashMap;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.Warmup;
+import org.openjdk.jmh.runner.Runner;
+import org.openjdk.jmh.runner.options.OptionsBuilder;
+
+
+@OutputTimeUnit(TimeUnit.MILLISECONDS)
+@Fork(1)
+@Warmup(iterations = 5, time = 10)
+@Measurement(iterations = 20, time = 10)
+@State(Scope.Benchmark)
+public class BenchmarkObjectOpenHashMap {
Review Comment:
can you add some context about this benchmark in the Javadoc? Otherwise lgtm.
Also link to instructions on how to run this.. or perhaps briefly mention how to do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Adding perf benchmark logic for GroupIdGenerator hash map [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #14992:
URL: https://github.com/apache/pinot/pull/14992#issuecomment-2635740181
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/14992?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 63.55%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`6e70a53`)](https://app.codecov.io/gh/apache/pinot/commit/6e70a53bfddc4c09e2e8b7a8851c9745173b526f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 1673 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #14992 +/- ##
============================================
+ Coverage 61.75% 63.55% +1.80%
- Complexity 207 1388 +1181
============================================
Files 2436 2713 +277
Lines 133233 152169 +18936
Branches 20636 23526 +2890
============================================
+ Hits 82274 96718 +14444
- Misses 44911 48151 +3240
- Partials 6048 7300 +1252
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/14992/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/14992/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/14992/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/14992/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/14992/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/14992/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/14992/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.55% <ø> (+1.93%)` | :arrow_up: |
| [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/14992/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.01% <ø> (-5.73%)` | :arrow_down: |
| [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/14992/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.54% <ø> (+35.82%)` | :arrow_up: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/14992/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.55% <ø> (+1.80%)` | :arrow_up: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/14992/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.55% <ø> (+1.81%)` | :arrow_up: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/14992/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.06% <ø> (+9.17%)` | :arrow_up: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/14992/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.01% <ø> (+6.28%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/14992?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org