You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/12/04 22:43:47 UTC
[GitHub] [incubator-pinot] richardstartin opened a new pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator
richardstartin opened a new pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320
## Description
This PR uses some more efficient APIs from RoaringBitmap.
- `RoaringBitmapWriter` is optimised for writing row ids into the bitmap sequentially. This class has a few options to control memory usage, but I've used its defaults.
- A `RoaringBitmap` can be serialized to a `ByteBuffer`, which is generally a lot faster than a `DataOutput`. However, this requires mapping a file outside of the control of `PinotByteBuffer`. The buffer also has to be big endian for backward compatibility with `DataOutput`.
I haven't run any benchmarks on this and am judging this on having used (and created) the same APIs for a similar problem. I tried to run the pinot-perf benchmarks but ran into errors with missing files. I would appreciate some help validating the benefits this change brings.
## Upgrade Notes
Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)
* [No]
Does this PR fix a zero-downtime upgrade introduced earlier?
* [ No]
Does this PR otherwise need attention when creating release notes?
* [ No]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#issuecomment-739128938
Can you follow one of the perf test classes (module pinot-perf , e.g BenchmarkPinotDataBuffer, BenchmarkDictionary etc) and get some numbers?
Also, you can write a simple test function that can be called while simultaneously profiling with YourKit/VisualVM. This can help reveal how the cpu and memory usage changes with this code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-io edited a comment on pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#issuecomment-739096511
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=h1) Report
> Merging [#6320](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=desc) (e3ec220) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) (1beaab5) will **increase** coverage by `7.67%`.
> The diff coverage is `68.40%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6320 +/- ##
==========================================
+ Coverage 66.44% 74.12% +7.67%
==========================================
Files 1075 1281 +206
Lines 54773 61823 +7050
Branches 8168 8986 +818
==========================================
+ Hits 36396 45825 +9429
+ Misses 15700 13036 -2664
- Partials 2677 2962 +285
```
| Flag | Coverage Δ | |
|---|---|---|
| integration | `45.71% <52.01%> (?)` | |
| unittests | `65.36% <45.12%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `100.00% <ø> (ø)` | |
| [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `52.83% <0.00%> (-13.84%)` | :arrow_down: |
| [...ker/routing/instanceselector/InstanceSelector.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9pbnN0YW5jZXNlbGVjdG9yL0luc3RhbmNlU2VsZWN0b3IuamF2YQ==) | `100.00% <ø> (ø)` | |
| [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `44.44% <0.00%> (-4.40%)` | :arrow_down: |
| [...not/common/assignment/InstancePartitionsUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnNVdGlscy5qYXZh) | `78.57% <ø> (+5.40%)` | :arrow_up: |
| [.../apache/pinot/common/exception/QueryException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZXhjZXB0aW9uL1F1ZXJ5RXhjZXB0aW9uLmphdmE=) | `90.27% <ø> (+5.55%)` | :arrow_up: |
| [...pinot/common/function/AggregationFunctionType.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vQWdncmVnYXRpb25GdW5jdGlvblR5cGUuamF2YQ==) | `98.27% <ø> (-1.73%)` | :arrow_down: |
| [.../pinot/common/function/DateTimePatternHandler.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRGF0ZVRpbWVQYXR0ZXJuSGFuZGxlci5qYXZh) | `83.33% <ø> (ø)` | |
| [...ot/common/function/FunctionDefinitionRegistry.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRnVuY3Rpb25EZWZpbml0aW9uUmVnaXN0cnkuamF2YQ==) | `88.88% <ø> (+44.44%)` | :arrow_up: |
| [...org/apache/pinot/common/function/FunctionInfo.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRnVuY3Rpb25JbmZvLmphdmE=) | `100.00% <ø> (ø)` | |
| ... and [1067 more](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=footer). Last update [41a7722...e3ec220](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] richardstartin commented on pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
richardstartin commented on pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#issuecomment-739241103
This PR allocates about the same amount as before
before
```
Benchmark (assignment) (cardinality) (numDocs) Mode Cnt Score Error Units
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 1000000 ss 5 0.031 ± 0.009 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10 1000000 ss 5 11.935 ± 1.702 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10 1000000 ss 5 7164704.000 ± 355.929 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 10000000 ss 5 0.408 ± 0.019 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10 10000000 ss 5 62.589 ± 5.249 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10 10000000 ss 5 71375131.200 ± 56512.643 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10 10000000 ss 5 70.263 ± 370.500 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10 10000000 ss 5 81788928.000 ± 431258859.992 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 10 10000000 ss 5 0.190 ± 1.632 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 10 10000000 ss 5 217683.200 ± 1874318.001 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10 10000000 ss 5 2.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10 10000000 ss 5 16.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 100000000 ss 5 4.193 ± 0.372 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10 100000000 ss 5 122.075 ± 12.717 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10 100000000 ss 5 712415345.600 ± 55691.481 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10 100000000 ss 5 120.923 ± 70.361 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10 100000000 ss 5 706320793.600 ± 422412916.188 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 10 100000000 ss 5 1.682 ± 0.744 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 10 100000000 ss 5 9832512.000 ± 4758932.013 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10 100000000 ss 5 18.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10 100000000 ss 5 35.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 1000000 ss 5 0.092 ± 0.014 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 1000 1000000 ss 5 22.304 ± 0.194 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 1000 1000000 ss 5 14236214.400 ± 1270.764 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 1000 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 10000000 ss 5 1.010 ± 0.033 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 1000 10000000 ss 5 76.099 ± 1.422 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 1000 10000000 ss 5 139322380.800 ± 46579.496 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 1000 10000000 ss 5 83.271 ± 179.254 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 1000 10000000 ss 5 152672665.600 ± 328639423.512 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 1000 10000000 ss 5 0.046 ± 0.399 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 1000 10000000 ss 5 84628.800 ± 728679.490 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 1000 10000000 ss 5 4.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 1000 10000000 ss 5 29.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 100000000 ss 5 11.633 ± 0.451 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 1000 100000000 ss 5 101.363 ± 2.794 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 1000 100000000 ss 5 1385939961.600 ± 55294.403 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 1000 100000000 ss 5 102.172 ± 30.699 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 1000 100000000 ss 5 1396703232.000 ± 398076377.621 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 1000 100000000 ss 5 0.011 ± 0.052 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 1000 100000000 ss 5 145384.000 ± 710672.251 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 1000 100000000 ss 5 37.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 1000 100000000 ss 5 37.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 1000000 ss 5 0.150 ± 0.045 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10000 1000000 ss 5 46.513 ± 4.210 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10000 1000000 ss 5 33470916.800 ± 45678.008 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10000 1000000 ss 5 41.258 ± 355.241 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10000 1000000 ss 5 29779558.400 ± 256410978.784 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10000 1000000 ss 5 1.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10000 1000000 ss 5 10.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 10000000 ss 5 1.738 ± 0.386 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10000 10000000 ss 5 125.776 ± 12.578 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10000 10000000 ss 5 317429830.400 ± 64562.976 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10000 10000000 ss 5 110.973 ± 164.658 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10000 10000000 ss 5 279340646.400 ± 400867868.240 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 10000 10000000 ss 5 0.143 ± 0.755 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 10000 10000000 ss 5 354760.000 ± 1874217.875 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10000 10000000 ss 5 6.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10000 10000000 ss 5 57.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 100000000 ss 5 17.156 ± 0.148 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10000 100000000 ss 5 160.044 ± 1.436 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10000 100000000 ss 5 3116823352.000 ± 45545.903 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10000 100000000 ss 5 159.645 ± 26.123 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10000 100000000 ss 5 3108818124.800 ± 482114724.008 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 10000 100000000 ss 5 0.026 ± 0.009 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 10000 100000000 ss 5 497857.600 ± 172824.399 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10000 100000000 ss 5 68.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10000 100000000 ss 5 73.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 1000000 ss 5 0.020 ± 0.002 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10 1000000 ss 5 2.138 ± 0.034 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10 1000000 ss 5 1184286.400 ± 378.164 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 10000000 ss 5 0.211 ± 0.001 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10 10000000 ss 5 9.535 ± 0.231 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10 10000000 ss 5 7565811.200 ± 502.228 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10 10000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 100000000 ss 5 2.326 ± 0.034 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10 100000000 ss 5 20.862 ± 0.237 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10 100000000 ss 5 71689880.000 ± 56609.543 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 10 100000000 ss 5 20.966 ± 110.550 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 10 100000000 ss 5 72142028.800 ± 380384094.171 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space SORTED_UNIFORM 10 100000000 ss 5 0.015 ± 0.127 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm SORTED_UNIFORM 10 100000000 ss 5 50654.400 ± 436149.660 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10 100000000 ss 5 2.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 10 100000000 ss 5 15.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 1000000 ss 5 0.074 ± 0.004 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 1000 1000000 ss 5 12.163 ± 0.138 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 1000 1000000 ss 5 7452222.400 ± 1019.460 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 1000 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 10000000 ss 5 0.275 ± 0.013 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 1000 10000000 ss 5 52.103 ± 1.439 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 1000 10000000 ss 5 50582414.400 ± 45991.882 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 1000 10000000 ss 5 42.028 ± 361.875 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 1000 10000000 ss 5 41104179.200 ± 353919379.167 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 1000 10000000 ss 5 1.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 1000 10000000 ss 5 7.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 100000000 ss 5 2.099 ± 0.251 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 1000 100000000 ss 5 35.099 ± 0.579 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 1000 100000000 ss 5 114648668.800 ± 55423.632 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 1000 100000000 ss 5 33.538 ± 117.892 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 1000 100000000 ss 5 109471334.400 ± 384807165.034 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space SORTED_UNIFORM 1000 100000000 ss 5 0.026 ± 0.223 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm SORTED_UNIFORM 1000 100000000 ss 5 83841.600 ± 721901.461 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 1000 100000000 ss 5 3.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 1000 100000000 ss 5 22.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 1000000 ss 5 0.093 ± 0.011 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10000 1000000 ss 5 20.878 ± 1.225 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10000 1000000 ss 5 13544881.600 ± 191.637 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10000 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 10000000 ss 5 0.886 ± 0.244 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10000 10000000 ss 5 46.010 ± 1.042 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10000 10000000 ss 5 74304364.800 ± 46037.614 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 10000 10000000 ss 5 24.963 ± 214.936 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 10000 10000000 ss 5 40684748.800 ± 350307956.931 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10000 10000000 ss 5 1.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 10000 10000000 ss 5 9.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 100000000 ss 5 3.826 ± 0.052 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10000 100000000 ss 5 100.180 ± 1.713 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10000 100000000 ss 5 505538409.600 ± 96799.515 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 10000 100000000 ss 5 99.990 ± 79.926 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 10000 100000000 ss 5 504574771.200 ± 402985207.107 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space SORTED_UNIFORM 10000 100000000 ss 5 0.329 ± 2.728 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm SORTED_UNIFORM 10000 100000000 ss 5 1669273.600 ± 13844467.735 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10000 100000000 ss 5 13.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 10000 100000000 ss 5 28.000 ms
```
after
```
Benchmark (assignment) (cardinality) (numDocs) Mode Cnt Score Error Units
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 1000000 ss 5 0.029 ± 0.022 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10 1000000 ss 5 12.403 ± 1.117 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10 1000000 ss 5 7146006.400 ± 30985.905 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 10000000 ss 5 0.188 ± 0.026 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10 10000000 ss 5 87.108 ± 4.467 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10 10000000 ss 5 71108766.400 ± 78261.812 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10 10000000 ss 5 95.241 ± 502.203 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10 10000000 ss 5 78014054.400 ± 411385221.551 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 10 10000000 ss 5 1.101 ± 9.482 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 10 10000000 ss 5 899424.000 ± 7744311.890 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10 10000000 ss 5 2.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10 10000000 ss 5 25.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 100000000 ss 5 1.947 ± 0.116 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10 100000000 ss 5 208.005 ± 7.371 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10 100000000 ss 5 709829240.000 ± 51497.263 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10 100000000 ss 5 201.399 ± 76.815 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10 100000000 ss 5 687026995.200 ± 247098882.679 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Old_Gen ROUND_ROBIN 10 100000000 ss 5 2.224 ± 17.872 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Old_Gen.norm ROUND_ROBIN 10 100000000 ss 5 7549747.200 ± 60592304.093 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 10 100000000 ss 5 4.788 ± 16.708 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 10 100000000 ss 5 16357785.600 ± 57215688.521 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10 100000000 ss 5 17.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10 100000000 ss 5 256.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 1000000 ss 5 0.040 ± 0.023 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 1000 1000000 ss 5 20.222 ± 1.683 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 1000 1000000 ss 5 11859302.400 ± 30898.399 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 1000 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 10000000 ss 5 0.293 ± 0.080 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 1000 10000000 ss 5 123.802 ± 11.035 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 1000 10000000 ss 5 113905795.200 ± 74048.230 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 1000 10000000 ss 5 108.134 ± 386.562 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 1000 10000000 ss 5 100663296.000 ± 360192977.778 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 1000 10000000 ss 5 6.363 ± 34.666 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 1000 10000000 ss 5 5872025.600 ± 32124407.719 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 1000 10000000 ss 5 3.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 1000 10000000 ss 5 35.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 100000000 ss 5 2.898 ± 0.114 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 1000 100000000 ss 5 255.212 ± 6.003 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 1000 100000000 ss 5 1129770475.200 ± 124788.799 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 1000 100000000 ss 5 257.389 ± 62.089 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 1000 100000000 ss 5 1139592396.800 ± 287647003.971 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Old_Gen ROUND_ROBIN 1000 100000000 ss 5 6.228 ± 32.860 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Old_Gen.norm ROUND_ROBIN 1000 100000000 ss 5 27472691.200 ± 144918864.230 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 1000 100000000 ss 5 7.259 ± 12.981 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 1000 100000000 ss 5 32086425.600 ± 56930036.769 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 1000 100000000 ss 5 24.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 1000 100000000 ss 5 1023.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 1000000 ss 5 0.066 ± 0.037 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10000 1000000 ss 5 72.601 ± 4.420 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10000 1000000 ss 5 45097956.800 ± 48245.104 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10000 1000000 ss 5 74.815 ± 644.181 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10000 1000000 ss 5 47395635.200 ± 408090712.713 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10000 1000000 ss 5 1.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10000 1000000 ss 5 14.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 10000000 ss 5 0.557 ± 0.267 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10000 10000000 ss 5 186.074 ± 47.551 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10000 10000000 ss 5 223440820.800 ± 78325.490 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10000 10000000 ss 5 138.524 ± 337.002 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10000 10000000 ss 5 171127603.200 ± 416904247.274 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 10000 10000000 ss 5 1.756 ± 15.119 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 10000 10000000 ss 5 2097152.000 ± 18057111.182 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10000 10000000 ss 5 5.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10000 10000000 ss 5 298.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 100000000 ss 5 5.260 ± 0.325 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10000 100000000 ss 5 310.016 ± 14.954 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10000 100000000 ss 5 2143068185.600 ± 64550.566 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10000 100000000 ss 5 306.636 ± 42.750 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10000 100000000 ss 5 2120220672.000 ± 339646533.455 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 10000 100000000 ss 5 8.017 ± 4.673 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 10000 100000000 ss 5 55364812.800 ± 30537345.918 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10000 100000000 ss 5 39.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10000 100000000 ss 5 4087.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 1000000 ss 5 0.017 ± 0.006 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10 1000000 ss 5 2.146 ± 0.059 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10 1000000 ss 5 1175939.200 ± 30372.447 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 10000000 ss 5 0.155 ± 0.001 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10 10000000 ss 5 10.358 ± 0.065 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10 10000000 ss 5 7563580.800 ± 30697.759 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10 10000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 100000000 ss 5 1.548 ± 0.023 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10 100000000 ss 5 28.135 ± 0.251 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10 100000000 ss 5 71579300.800 ± 78187.548 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 10 100000000 ss 5 28.278 ± 149.101 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 10 100000000 ss 5 72142028.800 ± 380384094.171 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space SORTED_UNIFORM 10 100000000 ss 5 0.028 ± 0.239 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm SORTED_UNIFORM 10 100000000 ss 5 70888.000 ± 610367.059 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10 100000000 ss 5 2.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 10 100000000 ss 5 17.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 1000000 ss 5 0.023 ± 0.032 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 1000 1000000 ss 5 13.814 ± 1.193 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 1000 1000000 ss 5 7669910.400 ± 300898.435 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 1000 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 10000000 ss 5 0.181 ± 0.037 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 1000 10000000 ss 5 66.605 ± 4.336 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 1000 10000000 ss 5 50803308.800 ± 76359.327 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 1000 10000000 ss 5 62.270 ± 536.163 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 1000 10000000 ss 5 47395635.200 ± 408090712.713 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 1000 10000000 ss 5 1.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 1000 10000000 ss 5 8.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 100000000 ss 5 1.568 ± 0.020 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 1000 100000000 ss 5 44.731 ± 0.340 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 1000 100000000 ss 5 114796513.600 ± 73178.185 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 1000 100000000 ss 5 43.084 ± 151.447 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 1000 100000000 ss 5 110729625.600 ± 389230235.896 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space SORTED_UNIFORM 1000 100000000 ss 5 0.032 ± 0.275 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm SORTED_UNIFORM 1000 100000000 ss 5 82238.400 ± 708097.426 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 1000 100000000 ss 5 3.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 1000 100000000 ss 5 24.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 1000000 ss 5 0.034 ± 0.037 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10000 1000000 ss 5 27.006 ± 2.034 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10000 1000000 ss 5 15331587.200 ± 30511.819 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10000 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 10000000 ss 5 0.186 ± 0.016 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10000 10000000 ss 5 99.771 ± 3.199 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10000 10000000 ss 5 76239598.400 ± 76412.193 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 10000 10000000 ss 5 53.525 ± 460.870 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 10000 10000000 ss 5 41523609.600 ± 357530801.403 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space SORTED_UNIFORM 10000 10000000 ss 5 0.062 ± 0.531 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm SORTED_UNIFORM 10000 10000000 ss 5 47827.200 ± 411806.616 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10000 10000000 ss 5 1.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 10000 10000000 ss 5 9.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 100000000 ss 5 1.736 ± 0.045 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10000 100000000 ss 5 184.429 ± 5.794 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10000 100000000 ss 5 507845612.800 ± 59214.207 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 10000 100000000 ss 5 171.683 ± 16.667 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 10000 100000000 ss 5 472698060.800 ± 32099022.960 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space SORTED_UNIFORM 10000 100000000 ss 5 0.901 ± 4.227 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm SORTED_UNIFORM 10000 100000000 ss 5 2492544.000 ± 11744141.868 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10000 100000000 ss 5 10.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 10000 100000000 ss 5 49.000 ms
```
Allocation can be reduced and get a speed gain by guessing the approximate sizes of the bitmaps, and by reusing a single writer.
latest commit
```
Benchmark (assignment) (cardinality) (numDocs) Mode Cnt Score Error Units
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 1000000 ss 5 0.021 ± 0.038 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 10000000 ss 5 0.175 ± 0.015 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 100000000 ss 5 1.780 ± 0.117 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 1000000 ss 5 0.024 ± 0.003 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 10000000 ss 5 0.243 ± 0.035 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 100000000 ss 5 2.711 ± 0.219 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 1000000 ss 5 0.052 ± 0.024 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 10000000 ss 5 0.539 ± 0.107 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 100000000 ss 5 4.500 ± 0.246 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 1000000 ss 5 0.016 ± 0.003 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 10000000 ss 5 0.148 ± 0.005 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 100000000 ss 5 1.412 ± 0.026 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 1000000 ss 5 0.021 ± 0.026 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 10000000 ss 5 0.165 ± 0.019 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 100000000 ss 5 1.437 ± 0.021 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 1000000 ss 5 0.022 ± 0.005 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 10000000 ss 5 0.172 ± 0.022 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 100000000 ss 5 1.581 ± 0.095 s/op
```
```
Benchmark (assignment) (cardinality) (numDocs) Mode Cnt Score Error Units
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 1000000 ss 5 0.017 ± 0.002 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10 1000000 ss 5 12.633 ± 0.155 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10 1000000 ss 5 6986539.200 ± 30442.010 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 10000000 ss 5 0.176 ± 0.012 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10 10000000 ss 5 87.350 ± 1.202 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10 10000000 ss 5 69659446.400 ± 52302.495 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10 10000000 ss 5 110.528 ± 582.795 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10 10000000 ss 5 88499814.400 ± 466642710.250 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10 10000000 ss 5 2.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10 10000000 ss 5 12.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 100000000 ss 5 1.783 ± 0.104 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10 100000000 ss 5 214.505 ± 7.603 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10 100000000 ss 5 695378540.800 ± 139270.627 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10 100000000 ss 5 214.143 ± 104.774 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10 100000000 ss 5 694576742.400 ± 352155335.840 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Old_Gen ROUND_ROBIN 10 100000000 ss 5 1.321 ± 11.370 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Old_Gen.norm ROUND_ROBIN 10 100000000 ss 5 4256358.400 ± 36648529.462 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 10 100000000 ss 5 3.111 ± 7.794 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 10 100000000 ss 5 10067219.200 ± 25155045.841 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10 100000000 ss 5 23.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10 100000000 ss 5 275.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 1000000 ss 5 0.025 ± 0.022 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 1000 1000000 ss 5 14.770 ± 1.072 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 1000 1000000 ss 5 8325694.400 ± 30481.121 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 1000 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 10000000 ss 5 0.244 ± 0.040 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 1000 10000000 ss 5 92.415 ± 4.679 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 1000 10000000 ss 5 80292582.400 ± 51274.216 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 1000 10000000 ss 5 69.550 ± 366.944 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 1000 10000000 ss 5 60817408.000 ± 320787006.897 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 1000 10000000 ss 5 2.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 1000 10000000 ss 5 22.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 100000000 ss 5 2.429 ± 0.248 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 1000 100000000 ss 5 200.418 ± 13.277 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 1000 100000000 ss 5 787876115.200 ± 256741.280 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 1000 100000000 ss 5 193.777 ± 130.177 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 1000 100000000 ss 5 762943897.600 ± 544097645.991 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Old_Gen ROUND_ROBIN 1000 100000000 ss 5 1.467 ± 12.630 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Old_Gen.norm ROUND_ROBIN 1000 100000000 ss 5 5872025.600 ± 50559911.310 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 1000 100000000 ss 5 4.343 ± 9.498 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 1000 100000000 ss 5 17196646.400 ± 38474784.725 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 1000 100000000 ss 5 23.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 1000 100000000 ss 5 746.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 1000000 ss 5 0.054 ± 0.025 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10000 1000000 ss 5 29.477 ± 2.118 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10000 1000000 ss 5 17577993.600 ± 48406.649 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10000 1000000 ss 5 30.144 ± 259.553 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10000 1000000 ss 5 18035507.200 ± 155291156.165 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10000 1000000 ss 5 1.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10000 1000000 ss 5 12.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 10000000 ss 5 0.481 ± 0.050 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10000 10000000 ss 5 205.551 ± 10.013 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10000 10000000 ss 5 230199414.400 ± 124257.456 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10000 10000000 ss 5 200.550 ± 150.026 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10000 10000000 ss 5 224395264.000 ± 162413652.372 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Old_Gen ROUND_ROBIN 10000 10000000 ss 5 1.853 ± 15.957 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Old_Gen.norm ROUND_ROBIN 10000 10000000 ss 5 2097152.000 ± 18057111.182 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 10000 10000000 ss 5 11.056 ± 41.589 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 10000 10000000 ss 5 12379680.000 ± 46553344.432 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10000 10000000 ss 5 11.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10000 10000000 ss 5 304.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 100000000 ss 5 4.505 ± 0.168 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10000 100000000 ss 5 220.361 ± 6.174 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10000 100000000 ss 5 1348977696.000 ± 48374.015 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10000 100000000 ss 5 220.473 ± 89.136 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10000 100000000 ss 5 1350146457.600 ± 565202002.425 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Old_Gen ROUND_ROBIN 10000 100000000 ss 5 0.719 ± 6.193 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Old_Gen.norm ROUND_ROBIN 10000 100000000 ss 5 4404019.200 ± 37919933.482 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 10000 100000000 ss 5 5.615 ± 5.115 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 10000 100000000 ss 5 34393292.800 ± 31587043.999 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10000 100000000 ss 5 36.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10000 100000000 ss 5 2728.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 1000000 ss 5 0.022 ± 0.037 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10 1000000 ss 5 2.066 ± 0.233 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10 1000000 ss 5 1148363.200 ± 30594.344 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 10000000 ss 5 0.148 ± 0.003 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10 10000000 ss 5 10.223 ± 0.105 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10 10000000 ss 5 7390347.200 ± 30613.010 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10 10000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 100000000 ss 5 1.410 ± 0.015 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10 100000000 ss 5 29.172 ± 0.183 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10 100000000 ss 5 70052542.400 ± 137755.129 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 10 100000000 ss 5 29.999 ± 158.176 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 10 100000000 ss 5 72142028.800 ± 380384094.171 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space SORTED_UNIFORM 10 100000000 ss 5 0.034 ± 0.293 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm SORTED_UNIFORM 10 100000000 ss 5 81944.000 ± 705562.553 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10 100000000 ss 5 2.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 10 100000000 ss 5 16.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 1000000 ss 5 0.025 ± 0.040 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 1000 1000000 ss 5 12.373 ± 1.158 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 1000 1000000 ss 5 6932864.000 ± 30927.743 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 1000 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 10000000 ss 5 0.164 ± 0.015 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 1000 10000000 ss 5 66.973 ± 1.648 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 1000 10000000 ss 5 49529336.000 ± 76261.103 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 1000 10000000 ss 5 55.023 ± 473.763 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 1000 10000000 ss 5 41104179.200 ± 353919379.167 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 1000 10000000 ss 5 1.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 1000 10000000 ss 5 8.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 100000000 ss 5 1.434 ± 0.018 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 1000 100000000 ss 5 46.177 ± 0.422 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 1000 100000000 ss 5 112162459.200 ± 73253.706 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 1000 100000000 ss 5 44.479 ± 156.352 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 1000 100000000 ss 5 108213043.200 ± 380384094.171 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space SORTED_UNIFORM 1000 100000000 ss 5 0.033 ± 0.282 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm SORTED_UNIFORM 1000 100000000 ss 5 80251.200 ± 686689.981 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 1000 100000000 ss 5 3.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 1000 100000000 ss 5 23.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 1000000 ss 5 0.023 ± 0.011 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10000 1000000 ss 5 20.115 ± 0.796 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10000 1000000 ss 5 11170392.000 ± 30879.694 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10000 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 10000000 ss 5 0.167 ± 0.016 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10000 10000000 ss 5 91.921 ± 7.404 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10000 10000000 ss 5 69227684.800 ± 52043.344 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 10000 10000000 ss 5 114.528 ± 604.014 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 10000 10000000 ss 5 87241523.200 ± 459999369.696 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10000 10000000 ss 5 2.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 10000 10000000 ss 5 17.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 100000000 ss 5 1.603 ± 0.123 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10000 100000000 ss 5 190.115 ± 9.171 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10000 100000000 ss 5 495109988.800 ± 101261.939 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 10000 100000000 ss 5 197.630 ± 142.159 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 10000 100000000 ss 5 514641100.800 ± 368630518.221 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space SORTED_UNIFORM 10000 100000000 ss 5 0.842 ± 4.357 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm SORTED_UNIFORM 10000 100000000 ss 5 2221792.000 ± 11637986.706 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10000 100000000 ss 5 11.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 10000 100000000 ss 5 49.000 ms
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-io commented on pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#issuecomment-739096511
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=h1) Report
> Merging [#6320](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=desc) (0a7af36) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) (1beaab5) will **decrease** coverage by `1.10%`.
> The diff coverage is `45.12%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6320 +/- ##
==========================================
- Coverage 66.44% 65.34% -1.11%
==========================================
Files 1075 1281 +206
Lines 54773 61813 +7040
Branches 8168 8986 +818
==========================================
+ Hits 36396 40393 +3997
- Misses 15700 18532 +2832
- Partials 2677 2888 +211
```
| Flag | Coverage Δ | |
|---|---|---|
| unittests | `65.34% <45.12%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...e/pinot/broker/api/resources/PinotBrokerDebug.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYXBpL3Jlc291cmNlcy9QaW5vdEJyb2tlckRlYnVnLmphdmE=) | `0.00% <0.00%> (-79.32%)` | :arrow_down: |
| [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `71.42% <ø> (-28.58%)` | :arrow_down: |
| [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `33.96% <0.00%> (-32.71%)` | :arrow_down: |
| [...ker/routing/instanceselector/InstanceSelector.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9pbnN0YW5jZXNlbGVjdG9yL0luc3RhbmNlU2VsZWN0b3IuamF2YQ==) | `100.00% <ø> (ø)` | |
| [...ava/org/apache/pinot/client/AbstractResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Fic3RyYWN0UmVzdWx0U2V0LmphdmE=) | `66.66% <0.00%> (+9.52%)` | :arrow_up: |
| [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `35.55% <0.00%> (-13.29%)` | :arrow_down: |
| [...inot/client/JsonAsyncHttpPinotClientTransport.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0pzb25Bc3luY0h0dHBQaW5vdENsaWVudFRyYW5zcG9ydC5qYXZh) | `10.90% <0.00%> (-51.10%)` | :arrow_down: |
| [...not/common/assignment/InstancePartitionsUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnNVdGlscy5qYXZh) | `73.80% <ø> (+0.63%)` | :arrow_up: |
| [.../apache/pinot/common/exception/QueryException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZXhjZXB0aW9uL1F1ZXJ5RXhjZXB0aW9uLmphdmE=) | `90.27% <ø> (+5.55%)` | :arrow_up: |
| [...pinot/common/function/AggregationFunctionType.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vQWdncmVnYXRpb25GdW5jdGlvblR5cGUuamF2YQ==) | `98.27% <ø> (-1.73%)` | :arrow_down: |
| ... and [1128 more](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=footer). Last update [41a7722...0a7af36](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-io edited a comment on pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#issuecomment-739096511
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=h1) Report
> Merging [#6320](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=desc) (ba99103) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) (1beaab5) will **decrease** coverage by `1.10%`.
> The diff coverage is `45.12%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6320 +/- ##
==========================================
- Coverage 66.44% 65.34% -1.11%
==========================================
Files 1075 1281 +206
Lines 54773 61813 +7040
Branches 8168 8986 +818
==========================================
+ Hits 36396 40392 +3996
- Misses 15700 18529 +2829
- Partials 2677 2892 +215
```
| Flag | Coverage Δ | |
|---|---|---|
| unittests | `65.34% <45.12%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...e/pinot/broker/api/resources/PinotBrokerDebug.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYXBpL3Jlc291cmNlcy9QaW5vdEJyb2tlckRlYnVnLmphdmE=) | `0.00% <0.00%> (-79.32%)` | :arrow_down: |
| [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `71.42% <ø> (-28.58%)` | :arrow_down: |
| [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `33.96% <0.00%> (-32.71%)` | :arrow_down: |
| [...ker/routing/instanceselector/InstanceSelector.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9pbnN0YW5jZXNlbGVjdG9yL0luc3RhbmNlU2VsZWN0b3IuamF2YQ==) | `100.00% <ø> (ø)` | |
| [...ava/org/apache/pinot/client/AbstractResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Fic3RyYWN0UmVzdWx0U2V0LmphdmE=) | `66.66% <0.00%> (+9.52%)` | :arrow_up: |
| [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `35.55% <0.00%> (-13.29%)` | :arrow_down: |
| [...inot/client/JsonAsyncHttpPinotClientTransport.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0pzb25Bc3luY0h0dHBQaW5vdENsaWVudFRyYW5zcG9ydC5qYXZh) | `10.90% <0.00%> (-51.10%)` | :arrow_down: |
| [...not/common/assignment/InstancePartitionsUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnNVdGlscy5qYXZh) | `73.80% <ø> (+0.63%)` | :arrow_up: |
| [.../apache/pinot/common/exception/QueryException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZXhjZXB0aW9uL1F1ZXJ5RXhjZXB0aW9uLmphdmE=) | `90.27% <ø> (+5.55%)` | :arrow_up: |
| [...pinot/common/function/AggregationFunctionType.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vQWdncmVnYXRpb25GdW5jdGlvblR5cGUuamF2YQ==) | `98.27% <ø> (-1.73%)` | :arrow_down: |
| ... and [1129 more](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=footer). Last update [41a7722...18d06e3](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r536890866
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
##########
@@ -181,33 +188,49 @@ public void seal()
}
// Create bitmaps from inverted index buffers and serialize them to file
- try (DataOutputStream offsetDataStream = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)));
- FileOutputStream bitmapFileStream = new FileOutputStream(_invertedIndexFile);
- DataOutputStream bitmapDataStream = new DataOutputStream(new BufferedOutputStream(bitmapFileStream))) {
- int bitmapOffset = (_cardinality + 1) * Integer.BYTES;
- offsetDataStream.writeInt(bitmapOffset);
- bitmapFileStream.getChannel().position(bitmapOffset);
-
+ ByteBuffer offsetBuffer = null;
+ ByteBuffer bitmapBuffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
Review comment:
From the code, I feel the benefit of serializing to `ByteBuffer` is to avoid the reverseBytes caused by BE to LE. How could it cause 20x performance difference? (Or the output stream is not buffered maybe?)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] richardstartin commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r536458126
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
##########
@@ -181,33 +188,49 @@ public void seal()
}
// Create bitmaps from inverted index buffers and serialize them to file
- try (DataOutputStream offsetDataStream = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)));
- FileOutputStream bitmapFileStream = new FileOutputStream(_invertedIndexFile);
- DataOutputStream bitmapDataStream = new DataOutputStream(new BufferedOutputStream(bitmapFileStream))) {
- int bitmapOffset = (_cardinality + 1) * Integer.BYTES;
- offsetDataStream.writeInt(bitmapOffset);
- bitmapFileStream.getChannel().position(bitmapOffset);
-
+ ByteBuffer offsetBuffer = null;
+ ByteBuffer bitmapBuffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ // map the offsets buffer
+ final int startOfBitmaps = (_cardinality + 1) * Integer.BYTES;
+ int bitmapOffset = startOfBitmaps;
+ offsetBuffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, bitmapOffset).order(LITTLE_ENDIAN);
+ offsetBuffer.putInt(reverseBytes(bitmapOffset));
+ MutableRoaringBitmap[] bitmaps = new MutableRoaringBitmap[_cardinality];
Review comment:
This is a allocation, but it's offset against the 16KB in stream buffers no longer allocated, and is a lot smaller than the bitmaps being materialised here anyway. This pattern where bitmaps are allocated just to obtain their serialized bytes could probably be supported by roaringbitmap without actually creating the intermediate bitmaps without much complication.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] richardstartin commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r536911160
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
##########
@@ -181,33 +188,51 @@ public void seal()
}
// Create bitmaps from inverted index buffers and serialize them to file
- try (DataOutputStream offsetDataStream = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)));
- FileOutputStream bitmapFileStream = new FileOutputStream(_invertedIndexFile);
- DataOutputStream bitmapDataStream = new DataOutputStream(new BufferedOutputStream(bitmapFileStream))) {
- int bitmapOffset = (_cardinality + 1) * Integer.BYTES;
- offsetDataStream.writeInt(bitmapOffset);
- bitmapFileStream.getChannel().position(bitmapOffset);
-
+ ByteBuffer offsetBuffer = null;
+ ByteBuffer bitmapBuffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ // map the offsets buffer
+ final int startOfBitmaps = (_cardinality + 1) * Integer.BYTES;
+ int bitmapOffset = startOfBitmaps;
+ offsetBuffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, bitmapOffset).order(LITTLE_ENDIAN);
+ offsetBuffer.putInt(reverseBytes(bitmapOffset));
+ RoaringBitmap[] bitmaps = new RoaringBitmap[_cardinality];
+ RoaringBitmapWriter<RoaringBitmap> writer = RoaringBitmapWriter.writer()
+ .initialCapacity(((_nextDocId - 1) >>> 16) / _cardinality).get();
Review comment:
This is just a heuristic that the keys of the bitmap will be evenly distributed across the dictionary ids. It won't always be perfect, but is never that bad. I can provide more justification.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] richardstartin commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r537835291
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OnHeapBitmapInvertedIndexCreator.java
##########
@@ -18,69 +18,85 @@
*/
package org.apache.pinot.core.segment.creator.impl.inv;
-import com.google.common.base.Preconditions;
-import java.io.BufferedOutputStream;
-import java.io.DataOutputStream;
+
import java.io.File;
-import java.io.FileOutputStream;
import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.channels.FileChannel;
+
import org.apache.commons.io.FileUtils;
import org.apache.pinot.core.segment.creator.DictionaryBasedInvertedIndexCreator;
import org.apache.pinot.core.segment.creator.impl.V1Constants;
-import org.roaringbitmap.buffer.MutableRoaringBitmap;
+import org.apache.pinot.core.util.CleanerUtil;
+import org.roaringbitmap.RoaringBitmap;
+import org.roaringbitmap.RoaringBitmapWriter;
/**
* Implementation of {@link DictionaryBasedInvertedIndexCreator} that uses on-heap memory.
*/
public final class OnHeapBitmapInvertedIndexCreator implements DictionaryBasedInvertedIndexCreator {
private final File _invertedIndexFile;
- private final MutableRoaringBitmap[] _bitmaps;
+ private final RoaringBitmapWriter<RoaringBitmap>[] _bitmapWriters;
private int _nextDocId;
+ @SuppressWarnings("unchecked")
public OnHeapBitmapInvertedIndexCreator(File indexDir, String columnName, int cardinality) {
_invertedIndexFile = new File(indexDir, columnName + V1Constants.Indexes.BITMAP_INVERTED_INDEX_FILE_EXTENSION);
- _bitmaps = new MutableRoaringBitmap[cardinality];
+ _bitmapWriters = new RoaringBitmapWriter[cardinality];
for (int i = 0; i < cardinality; i++) {
- _bitmaps[i] = new MutableRoaringBitmap();
+ _bitmapWriters[i] = RoaringBitmapWriter.writer().runCompress(false).get();
}
}
@Override
public void add(int dictId) {
- _bitmaps[dictId].add(_nextDocId++);
+ _bitmapWriters[dictId].add(_nextDocId++);
}
@Override
public void add(int[] dictIds, int length) {
for (int i = 0; i < length; i++) {
- _bitmaps[dictIds[i]].add(_nextDocId);
+ _bitmapWriters[dictIds[i]].add(_nextDocId);
}
_nextDocId++;
}
@Override
public void seal()
throws IOException {
- try (DataOutputStream out = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)))) {
+ // calculate file size
Review comment:
this comment is now misleading, will remove it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg commented on pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
kishoreg commented on pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#issuecomment-739308749
This looks promising both from performance and memory allocation. @Jackie-Jiang can you please look into this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#issuecomment-740193290
I'm working on a feature which requires extracting the common part of the on-heap and off-heap inverted index creator. Can I add a commit to this PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r537847693
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
##########
@@ -181,33 +184,45 @@ public void seal()
}
// Create bitmaps from inverted index buffers and serialize them to file
- try (DataOutputStream offsetDataStream = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)));
- FileOutputStream bitmapFileStream = new FileOutputStream(_invertedIndexFile);
- DataOutputStream bitmapDataStream = new DataOutputStream(new BufferedOutputStream(bitmapFileStream))) {
- int bitmapOffset = (_cardinality + 1) * Integer.BYTES;
- offsetDataStream.writeInt(bitmapOffset);
- bitmapFileStream.getChannel().position(bitmapOffset);
-
+ ByteBuffer offsetBuffer = null;
+ ByteBuffer bitmapBuffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ // map the offsets buffer
+ int startOfBitmaps = (_cardinality + 1) * Integer.BYTES;
+ offsetBuffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, startOfBitmaps)
+ .order(ByteOrder.BIG_ENDIAN);
+ offsetBuffer.putInt(startOfBitmaps);
+ bitmapBuffer = channel.map(FileChannel.MapMode.READ_WRITE, startOfBitmaps, Integer.MAX_VALUE - startOfBitmaps)
+ .order(ByteOrder.LITTLE_ENDIAN);
+ RoaringBitmapWriter<RoaringBitmap> writer = RoaringBitmapWriter.writer().get();
Review comment:
The bitmap could be dense (when cardinality is very low), but won't be contiguous. What is the old behavior? We don't explicitly call `runOptimize()` on the `MutableRoaringBitmap`, will it perform RLE automatically? If not, we can disable it here to keep the same behavior and have better performance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r536915214
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OnHeapBitmapInvertedIndexCreator.java
##########
@@ -19,68 +19,85 @@
package org.apache.pinot.core.segment.creator.impl.inv;
import com.google.common.base.Preconditions;
-import java.io.BufferedOutputStream;
-import java.io.DataOutputStream;
+
+
import java.io.File;
-import java.io.FileOutputStream;
import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+
import org.apache.commons.io.FileUtils;
import org.apache.pinot.core.segment.creator.DictionaryBasedInvertedIndexCreator;
import org.apache.pinot.core.segment.creator.impl.V1Constants;
+import org.apache.pinot.core.util.CleanerUtil;
+import org.roaringbitmap.RoaringBitmapWriter;
import org.roaringbitmap.buffer.MutableRoaringBitmap;
+import static java.nio.ByteOrder.LITTLE_ENDIAN;
+
/**
* Implementation of {@link DictionaryBasedInvertedIndexCreator} that uses on-heap memory.
*/
public final class OnHeapBitmapInvertedIndexCreator implements DictionaryBasedInvertedIndexCreator {
private final File _invertedIndexFile;
- private final MutableRoaringBitmap[] _bitmaps;
+ private final RoaringBitmapWriter<MutableRoaringBitmap>[] _bitmapWriters;
private int _nextDocId;
+ @SuppressWarnings("unchecked")
public OnHeapBitmapInvertedIndexCreator(File indexDir, String columnName, int cardinality) {
_invertedIndexFile = new File(indexDir, columnName + V1Constants.Indexes.BITMAP_INVERTED_INDEX_FILE_EXTENSION);
- _bitmaps = new MutableRoaringBitmap[cardinality];
+ _bitmapWriters = new RoaringBitmapWriter[cardinality];
for (int i = 0; i < cardinality; i++) {
- _bitmaps[i] = new MutableRoaringBitmap();
+ _bitmapWriters[i] = RoaringBitmapWriter.bufferWriter().get();
}
}
@Override
public void add(int dictId) {
- _bitmaps[dictId].add(_nextDocId++);
+ _bitmapWriters[dictId].add(_nextDocId++);
}
@Override
public void add(int[] dictIds, int length) {
for (int i = 0; i < length; i++) {
- _bitmaps[dictIds[i]].add(_nextDocId);
+ _bitmapWriters[dictIds[i]].add(_nextDocId);
}
_nextDocId++;
}
@Override
public void seal()
throws IOException {
- try (DataOutputStream out = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)))) {
+ // calculate file size
+ int size = (_bitmapWriters.length + 1) * Integer.BYTES;
+ for (RoaringBitmapWriter<MutableRoaringBitmap> writer : _bitmapWriters) {
+ size += writer.get().serializedSizeInBytes();
+ // Check for int overflow
+ Preconditions.checkState(size > 0, "Inverted index file: %s exceeds 2GB limit", _invertedIndexFile);
+ }
+ ByteBuffer buffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ buffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, size).order(LITTLE_ENDIAN);
Review comment:
Not sure I completely follow. You are writing big endian bytes by doing reverseBytes() anyway. So why not just choose the byte order as BE.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] richardstartin commented on pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
richardstartin commented on pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#issuecomment-740213456
> I'm working on a feature which requires extracting the common part of the on-heap and off-heap inverted index creator. Can I add a commit to this PR?
Since this is on a fork, I'm not sure if that's possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-io edited a comment on pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#issuecomment-739096511
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=h1) Report
> Merging [#6320](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=desc) (03de308) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) (1beaab5) will **decrease** coverage by `1.07%`.
> The diff coverage is `45.12%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6320 +/- ##
==========================================
- Coverage 66.44% 65.37% -1.08%
==========================================
Files 1075 1281 +206
Lines 54773 61823 +7050
Branches 8168 8986 +818
==========================================
+ Hits 36396 40417 +4021
- Misses 15700 18515 +2815
- Partials 2677 2891 +214
```
| Flag | Coverage Δ | |
|---|---|---|
| unittests | `65.37% <45.12%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...e/pinot/broker/api/resources/PinotBrokerDebug.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYXBpL3Jlc291cmNlcy9QaW5vdEJyb2tlckRlYnVnLmphdmE=) | `0.00% <0.00%> (-79.32%)` | :arrow_down: |
| [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `71.42% <ø> (-28.58%)` | :arrow_down: |
| [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `33.96% <0.00%> (-32.71%)` | :arrow_down: |
| [...ker/routing/instanceselector/InstanceSelector.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9pbnN0YW5jZXNlbGVjdG9yL0luc3RhbmNlU2VsZWN0b3IuamF2YQ==) | `100.00% <ø> (ø)` | |
| [...ava/org/apache/pinot/client/AbstractResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Fic3RyYWN0UmVzdWx0U2V0LmphdmE=) | `66.66% <0.00%> (+9.52%)` | :arrow_up: |
| [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `35.55% <0.00%> (-13.29%)` | :arrow_down: |
| [...inot/client/JsonAsyncHttpPinotClientTransport.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0pzb25Bc3luY0h0dHBQaW5vdENsaWVudFRyYW5zcG9ydC5qYXZh) | `10.90% <0.00%> (-51.10%)` | :arrow_down: |
| [...not/common/assignment/InstancePartitionsUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnNVdGlscy5qYXZh) | `73.80% <ø> (+0.63%)` | :arrow_up: |
| [.../apache/pinot/common/exception/QueryException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZXhjZXB0aW9uL1F1ZXJ5RXhjZXB0aW9uLmphdmE=) | `90.27% <ø> (+5.55%)` | :arrow_up: |
| [...pinot/common/function/AggregationFunctionType.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vQWdncmVnYXRpb25GdW5jdGlvblR5cGUuamF2YQ==) | `98.27% <ø> (-1.73%)` | :arrow_down: |
| ... and [1128 more](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=footer). Last update [41a7722...e3ec220](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] richardstartin commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r536565284
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
##########
@@ -181,33 +188,49 @@ public void seal()
}
// Create bitmaps from inverted index buffers and serialize them to file
- try (DataOutputStream offsetDataStream = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)));
- FileOutputStream bitmapFileStream = new FileOutputStream(_invertedIndexFile);
- DataOutputStream bitmapDataStream = new DataOutputStream(new BufferedOutputStream(bitmapFileStream))) {
- int bitmapOffset = (_cardinality + 1) * Integer.BYTES;
- offsetDataStream.writeInt(bitmapOffset);
- bitmapFileStream.getChannel().position(bitmapOffset);
-
+ ByteBuffer offsetBuffer = null;
+ ByteBuffer bitmapBuffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
Review comment:
Two places.
- We found that serializing to `ByteBuffer` could be up to 20x faster than to `DataOutput`, which was why the API was added.
- If the bitmaps being materialised prior to serialisation below get reasonably large, using `RoaringBitmapWriter` improves the efficiency of construction. This is because when you call `RoaringBitmap.add`, it does a binary search in the high 16 bits of the value to figure out where to put the low 16 bits, but `RoaringBitmapWriter` assumes all adds are appends and optimises for it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] richardstartin commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r536917433
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OnHeapBitmapInvertedIndexCreator.java
##########
@@ -19,68 +19,85 @@
package org.apache.pinot.core.segment.creator.impl.inv;
import com.google.common.base.Preconditions;
-import java.io.BufferedOutputStream;
-import java.io.DataOutputStream;
+
+
import java.io.File;
-import java.io.FileOutputStream;
import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+
import org.apache.commons.io.FileUtils;
import org.apache.pinot.core.segment.creator.DictionaryBasedInvertedIndexCreator;
import org.apache.pinot.core.segment.creator.impl.V1Constants;
+import org.apache.pinot.core.util.CleanerUtil;
+import org.roaringbitmap.RoaringBitmapWriter;
import org.roaringbitmap.buffer.MutableRoaringBitmap;
+import static java.nio.ByteOrder.LITTLE_ENDIAN;
+
/**
* Implementation of {@link DictionaryBasedInvertedIndexCreator} that uses on-heap memory.
*/
public final class OnHeapBitmapInvertedIndexCreator implements DictionaryBasedInvertedIndexCreator {
private final File _invertedIndexFile;
- private final MutableRoaringBitmap[] _bitmaps;
+ private final RoaringBitmapWriter<MutableRoaringBitmap>[] _bitmapWriters;
private int _nextDocId;
+ @SuppressWarnings("unchecked")
public OnHeapBitmapInvertedIndexCreator(File indexDir, String columnName, int cardinality) {
_invertedIndexFile = new File(indexDir, columnName + V1Constants.Indexes.BITMAP_INVERTED_INDEX_FILE_EXTENSION);
- _bitmaps = new MutableRoaringBitmap[cardinality];
+ _bitmapWriters = new RoaringBitmapWriter[cardinality];
for (int i = 0; i < cardinality; i++) {
- _bitmaps[i] = new MutableRoaringBitmap();
+ _bitmapWriters[i] = RoaringBitmapWriter.bufferWriter().get();
}
}
@Override
public void add(int dictId) {
- _bitmaps[dictId].add(_nextDocId++);
+ _bitmapWriters[dictId].add(_nextDocId++);
}
@Override
public void add(int[] dictIds, int length) {
for (int i = 0; i < length; i++) {
- _bitmaps[dictIds[i]].add(_nextDocId);
+ _bitmapWriters[dictIds[i]].add(_nextDocId);
}
_nextDocId++;
}
@Override
public void seal()
throws IOException {
- try (DataOutputStream out = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)))) {
+ // calculate file size
+ int size = (_bitmapWriters.length + 1) * Integer.BYTES;
+ for (RoaringBitmapWriter<MutableRoaringBitmap> writer : _bitmapWriters) {
+ size += writer.get().serializedSizeInBytes();
+ // Check for int overflow
+ Preconditions.checkState(size > 0, "Inverted index file: %s exceeds 2GB limit", _invertedIndexFile);
+ }
+ ByteBuffer buffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ buffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, size).order(LITTLE_ENDIAN);
Review comment:
In any case, the code in question was changed so that the offsets part of the file is big endian anyway, and a little endian buffer is provided to `RoaringBitmap.serialize`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] richardstartin commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r537811099
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
##########
@@ -181,33 +184,45 @@ public void seal()
}
// Create bitmaps from inverted index buffers and serialize them to file
- try (DataOutputStream offsetDataStream = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)));
- FileOutputStream bitmapFileStream = new FileOutputStream(_invertedIndexFile);
- DataOutputStream bitmapDataStream = new DataOutputStream(new BufferedOutputStream(bitmapFileStream))) {
- int bitmapOffset = (_cardinality + 1) * Integer.BYTES;
- offsetDataStream.writeInt(bitmapOffset);
- bitmapFileStream.getChannel().position(bitmapOffset);
-
+ ByteBuffer offsetBuffer = null;
+ ByteBuffer bitmapBuffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ // map the offsets buffer
+ int startOfBitmaps = (_cardinality + 1) * Integer.BYTES;
+ offsetBuffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, startOfBitmaps)
+ .order(ByteOrder.BIG_ENDIAN);
+ offsetBuffer.putInt(startOfBitmaps);
+ bitmapBuffer = channel.map(FileChannel.MapMode.READ_WRITE, startOfBitmaps, Integer.MAX_VALUE - startOfBitmaps)
+ .order(ByteOrder.LITTLE_ENDIAN);
+ RoaringBitmapWriter<RoaringBitmap> writer = RoaringBitmapWriter.writer().get();
Review comment:
If that's the case, currently this will try to RLE each container when it's appended to the bitmap. Would you prefer I remove that when I reformat the file?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r536520136
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OnHeapBitmapInvertedIndexCreator.java
##########
@@ -19,68 +19,85 @@
package org.apache.pinot.core.segment.creator.impl.inv;
import com.google.common.base.Preconditions;
-import java.io.BufferedOutputStream;
-import java.io.DataOutputStream;
+
+
import java.io.File;
-import java.io.FileOutputStream;
import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+
import org.apache.commons.io.FileUtils;
import org.apache.pinot.core.segment.creator.DictionaryBasedInvertedIndexCreator;
import org.apache.pinot.core.segment.creator.impl.V1Constants;
+import org.apache.pinot.core.util.CleanerUtil;
+import org.roaringbitmap.RoaringBitmapWriter;
import org.roaringbitmap.buffer.MutableRoaringBitmap;
+import static java.nio.ByteOrder.LITTLE_ENDIAN;
+
/**
* Implementation of {@link DictionaryBasedInvertedIndexCreator} that uses on-heap memory.
*/
public final class OnHeapBitmapInvertedIndexCreator implements DictionaryBasedInvertedIndexCreator {
private final File _invertedIndexFile;
- private final MutableRoaringBitmap[] _bitmaps;
+ private final RoaringBitmapWriter<MutableRoaringBitmap>[] _bitmapWriters;
private int _nextDocId;
+ @SuppressWarnings("unchecked")
public OnHeapBitmapInvertedIndexCreator(File indexDir, String columnName, int cardinality) {
_invertedIndexFile = new File(indexDir, columnName + V1Constants.Indexes.BITMAP_INVERTED_INDEX_FILE_EXTENSION);
- _bitmaps = new MutableRoaringBitmap[cardinality];
+ _bitmapWriters = new RoaringBitmapWriter[cardinality];
for (int i = 0; i < cardinality; i++) {
- _bitmaps[i] = new MutableRoaringBitmap();
+ _bitmapWriters[i] = RoaringBitmapWriter.bufferWriter().get();
}
}
@Override
public void add(int dictId) {
- _bitmaps[dictId].add(_nextDocId++);
+ _bitmapWriters[dictId].add(_nextDocId++);
}
@Override
public void add(int[] dictIds, int length) {
for (int i = 0; i < length; i++) {
- _bitmaps[dictIds[i]].add(_nextDocId);
+ _bitmapWriters[dictIds[i]].add(_nextDocId);
}
_nextDocId++;
}
@Override
public void seal()
throws IOException {
- try (DataOutputStream out = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)))) {
+ // calculate file size
+ int size = (_bitmapWriters.length + 1) * Integer.BYTES;
+ for (RoaringBitmapWriter<MutableRoaringBitmap> writer : _bitmapWriters) {
+ size += writer.get().serializedSizeInBytes();
+ // Check for int overflow
+ Preconditions.checkState(size > 0, "Inverted index file: %s exceeds 2GB limit", _invertedIndexFile);
+ }
+ ByteBuffer buffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ buffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, size).order(LITTLE_ENDIAN);
Review comment:
Ok, I see you are swapping the bytes below before writing. So, this is writing BE bytes in LE file so that later during index load, read in BE format will work. In this case, why not just choose the BE format for writing as we do today. Does the usage of RoaringBitmapWriter enforce LE format?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] richardstartin commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r536458126
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
##########
@@ -181,33 +188,49 @@ public void seal()
}
// Create bitmaps from inverted index buffers and serialize them to file
- try (DataOutputStream offsetDataStream = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)));
- FileOutputStream bitmapFileStream = new FileOutputStream(_invertedIndexFile);
- DataOutputStream bitmapDataStream = new DataOutputStream(new BufferedOutputStream(bitmapFileStream))) {
- int bitmapOffset = (_cardinality + 1) * Integer.BYTES;
- offsetDataStream.writeInt(bitmapOffset);
- bitmapFileStream.getChannel().position(bitmapOffset);
-
+ ByteBuffer offsetBuffer = null;
+ ByteBuffer bitmapBuffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ // map the offsets buffer
+ final int startOfBitmaps = (_cardinality + 1) * Integer.BYTES;
+ int bitmapOffset = startOfBitmaps;
+ offsetBuffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, bitmapOffset).order(LITTLE_ENDIAN);
+ offsetBuffer.putInt(reverseBytes(bitmapOffset));
+ MutableRoaringBitmap[] bitmaps = new MutableRoaringBitmap[_cardinality];
Review comment:
This is a new allocation, but it's offset against the 16KB in stream buffers no longer allocated, and is a lot smaller than the bitmaps being materialised here anyway. This pattern where bitmaps are allocated just to obtain their serialized bytes could probably be supported by roaringbitmap without actually creating the intermediate bitmaps without much complication.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r536520136
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OnHeapBitmapInvertedIndexCreator.java
##########
@@ -19,68 +19,85 @@
package org.apache.pinot.core.segment.creator.impl.inv;
import com.google.common.base.Preconditions;
-import java.io.BufferedOutputStream;
-import java.io.DataOutputStream;
+
+
import java.io.File;
-import java.io.FileOutputStream;
import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+
import org.apache.commons.io.FileUtils;
import org.apache.pinot.core.segment.creator.DictionaryBasedInvertedIndexCreator;
import org.apache.pinot.core.segment.creator.impl.V1Constants;
+import org.apache.pinot.core.util.CleanerUtil;
+import org.roaringbitmap.RoaringBitmapWriter;
import org.roaringbitmap.buffer.MutableRoaringBitmap;
+import static java.nio.ByteOrder.LITTLE_ENDIAN;
+
/**
* Implementation of {@link DictionaryBasedInvertedIndexCreator} that uses on-heap memory.
*/
public final class OnHeapBitmapInvertedIndexCreator implements DictionaryBasedInvertedIndexCreator {
private final File _invertedIndexFile;
- private final MutableRoaringBitmap[] _bitmaps;
+ private final RoaringBitmapWriter<MutableRoaringBitmap>[] _bitmapWriters;
private int _nextDocId;
+ @SuppressWarnings("unchecked")
public OnHeapBitmapInvertedIndexCreator(File indexDir, String columnName, int cardinality) {
_invertedIndexFile = new File(indexDir, columnName + V1Constants.Indexes.BITMAP_INVERTED_INDEX_FILE_EXTENSION);
- _bitmaps = new MutableRoaringBitmap[cardinality];
+ _bitmapWriters = new RoaringBitmapWriter[cardinality];
for (int i = 0; i < cardinality; i++) {
- _bitmaps[i] = new MutableRoaringBitmap();
+ _bitmapWriters[i] = RoaringBitmapWriter.bufferWriter().get();
}
}
@Override
public void add(int dictId) {
- _bitmaps[dictId].add(_nextDocId++);
+ _bitmapWriters[dictId].add(_nextDocId++);
}
@Override
public void add(int[] dictIds, int length) {
for (int i = 0; i < length; i++) {
- _bitmaps[dictIds[i]].add(_nextDocId);
+ _bitmapWriters[dictIds[i]].add(_nextDocId);
}
_nextDocId++;
}
@Override
public void seal()
throws IOException {
- try (DataOutputStream out = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)))) {
+ // calculate file size
+ int size = (_bitmapWriters.length + 1) * Integer.BYTES;
+ for (RoaringBitmapWriter<MutableRoaringBitmap> writer : _bitmapWriters) {
+ size += writer.get().serializedSizeInBytes();
+ // Check for int overflow
+ Preconditions.checkState(size > 0, "Inverted index file: %s exceeds 2GB limit", _invertedIndexFile);
+ }
+ ByteBuffer buffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ buffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, size).order(LITTLE_ENDIAN);
Review comment:
Ok, I see you are swapping the bytes below before writing. So, this is writing BE bytes in LE file so that later during read in BE format will work. In this case, why not choose the BE format for writing as we do today. Does the usage of RoaringBitmapWriter enforce LE format?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r537124621
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OnHeapBitmapInvertedIndexCreator.java
##########
@@ -19,68 +19,96 @@
package org.apache.pinot.core.segment.creator.impl.inv;
import com.google.common.base.Preconditions;
-import java.io.BufferedOutputStream;
-import java.io.DataOutputStream;
+
+
import java.io.File;
-import java.io.FileOutputStream;
import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.channels.FileChannel;
+
import org.apache.commons.io.FileUtils;
import org.apache.pinot.core.segment.creator.DictionaryBasedInvertedIndexCreator;
import org.apache.pinot.core.segment.creator.impl.V1Constants;
-import org.roaringbitmap.buffer.MutableRoaringBitmap;
+import org.apache.pinot.core.util.CleanerUtil;
+import org.roaringbitmap.RoaringBitmap;
+import org.roaringbitmap.RoaringBitmapWriter;
/**
* Implementation of {@link DictionaryBasedInvertedIndexCreator} that uses on-heap memory.
*/
public final class OnHeapBitmapInvertedIndexCreator implements DictionaryBasedInvertedIndexCreator {
private final File _invertedIndexFile;
- private final MutableRoaringBitmap[] _bitmaps;
+ private final RoaringBitmapWriter<RoaringBitmap>[] _bitmapWriters;
private int _nextDocId;
+ @SuppressWarnings("unchecked")
public OnHeapBitmapInvertedIndexCreator(File indexDir, String columnName, int cardinality) {
_invertedIndexFile = new File(indexDir, columnName + V1Constants.Indexes.BITMAP_INVERTED_INDEX_FILE_EXTENSION);
- _bitmaps = new MutableRoaringBitmap[cardinality];
+ _bitmapWriters = new RoaringBitmapWriter[cardinality];
for (int i = 0; i < cardinality; i++) {
- _bitmaps[i] = new MutableRoaringBitmap();
+ _bitmapWriters[i] = RoaringBitmapWriter.writer().get();
}
}
@Override
public void add(int dictId) {
- _bitmaps[dictId].add(_nextDocId++);
+ _bitmapWriters[dictId].add(_nextDocId++);
}
@Override
public void add(int[] dictIds, int length) {
for (int i = 0; i < length; i++) {
- _bitmaps[dictIds[i]].add(_nextDocId);
+ _bitmapWriters[dictIds[i]].add(_nextDocId);
}
_nextDocId++;
}
@Override
public void seal()
throws IOException {
- try (DataOutputStream out = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)))) {
+ // calculate file size
+ int startOfBitmaps = (_bitmapWriters.length + 1) * Integer.BYTES;
+ int size = 0;
+ for (RoaringBitmapWriter<RoaringBitmap> writer : _bitmapWriters) {
+ size += writer.get().serializedSizeInBytes();
Review comment:
Maybe we can use the same way as the off-heap creator to preallocate a huge buffer then truncate to same the calculation of the serialized size?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] richardstartin edited a comment on pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
richardstartin edited a comment on pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#issuecomment-739241103
This PR allocates about the same amount as before. TL;DR: as of the [last commit](https://github.com/apache/incubator-pinot/pull/6320/commits/60861bc25110b94011091332a70a5e955e38405b) this is faster on this benchmark, and allocates between 40% and 100% (i.e. the same) as on the main branch.
before
```
Benchmark (assignment) (cardinality) (numDocs) Mode Cnt Score Error Units
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 1000000 ss 5 0.031 ± 0.009 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10 1000000 ss 5 11.935 ± 1.702 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10 1000000 ss 5 7164704.000 ± 355.929 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 10000000 ss 5 0.408 ± 0.019 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10 10000000 ss 5 62.589 ± 5.249 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10 10000000 ss 5 71375131.200 ± 56512.643 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10 10000000 ss 5 70.263 ± 370.500 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10 10000000 ss 5 81788928.000 ± 431258859.992 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 10 10000000 ss 5 0.190 ± 1.632 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 10 10000000 ss 5 217683.200 ± 1874318.001 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10 10000000 ss 5 2.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10 10000000 ss 5 16.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 100000000 ss 5 4.193 ± 0.372 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10 100000000 ss 5 122.075 ± 12.717 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10 100000000 ss 5 712415345.600 ± 55691.481 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10 100000000 ss 5 120.923 ± 70.361 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10 100000000 ss 5 706320793.600 ± 422412916.188 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 10 100000000 ss 5 1.682 ± 0.744 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 10 100000000 ss 5 9832512.000 ± 4758932.013 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10 100000000 ss 5 18.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10 100000000 ss 5 35.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 1000000 ss 5 0.092 ± 0.014 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 1000 1000000 ss 5 22.304 ± 0.194 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 1000 1000000 ss 5 14236214.400 ± 1270.764 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 1000 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 10000000 ss 5 1.010 ± 0.033 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 1000 10000000 ss 5 76.099 ± 1.422 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 1000 10000000 ss 5 139322380.800 ± 46579.496 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 1000 10000000 ss 5 83.271 ± 179.254 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 1000 10000000 ss 5 152672665.600 ± 328639423.512 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 1000 10000000 ss 5 0.046 ± 0.399 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 1000 10000000 ss 5 84628.800 ± 728679.490 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 1000 10000000 ss 5 4.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 1000 10000000 ss 5 29.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 100000000 ss 5 11.633 ± 0.451 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 1000 100000000 ss 5 101.363 ± 2.794 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 1000 100000000 ss 5 1385939961.600 ± 55294.403 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 1000 100000000 ss 5 102.172 ± 30.699 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 1000 100000000 ss 5 1396703232.000 ± 398076377.621 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 1000 100000000 ss 5 0.011 ± 0.052 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 1000 100000000 ss 5 145384.000 ± 710672.251 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 1000 100000000 ss 5 37.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 1000 100000000 ss 5 37.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 1000000 ss 5 0.150 ± 0.045 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10000 1000000 ss 5 46.513 ± 4.210 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10000 1000000 ss 5 33470916.800 ± 45678.008 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10000 1000000 ss 5 41.258 ± 355.241 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10000 1000000 ss 5 29779558.400 ± 256410978.784 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10000 1000000 ss 5 1.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10000 1000000 ss 5 10.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 10000000 ss 5 1.738 ± 0.386 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10000 10000000 ss 5 125.776 ± 12.578 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10000 10000000 ss 5 317429830.400 ± 64562.976 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10000 10000000 ss 5 110.973 ± 164.658 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10000 10000000 ss 5 279340646.400 ± 400867868.240 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 10000 10000000 ss 5 0.143 ± 0.755 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 10000 10000000 ss 5 354760.000 ± 1874217.875 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10000 10000000 ss 5 6.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10000 10000000 ss 5 57.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 100000000 ss 5 17.156 ± 0.148 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10000 100000000 ss 5 160.044 ± 1.436 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10000 100000000 ss 5 3116823352.000 ± 45545.903 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10000 100000000 ss 5 159.645 ± 26.123 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10000 100000000 ss 5 3108818124.800 ± 482114724.008 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 10000 100000000 ss 5 0.026 ± 0.009 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 10000 100000000 ss 5 497857.600 ± 172824.399 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10000 100000000 ss 5 68.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10000 100000000 ss 5 73.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 1000000 ss 5 0.020 ± 0.002 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10 1000000 ss 5 2.138 ± 0.034 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10 1000000 ss 5 1184286.400 ± 378.164 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 10000000 ss 5 0.211 ± 0.001 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10 10000000 ss 5 9.535 ± 0.231 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10 10000000 ss 5 7565811.200 ± 502.228 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10 10000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 100000000 ss 5 2.326 ± 0.034 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10 100000000 ss 5 20.862 ± 0.237 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10 100000000 ss 5 71689880.000 ± 56609.543 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 10 100000000 ss 5 20.966 ± 110.550 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 10 100000000 ss 5 72142028.800 ± 380384094.171 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space SORTED_UNIFORM 10 100000000 ss 5 0.015 ± 0.127 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm SORTED_UNIFORM 10 100000000 ss 5 50654.400 ± 436149.660 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10 100000000 ss 5 2.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 10 100000000 ss 5 15.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 1000000 ss 5 0.074 ± 0.004 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 1000 1000000 ss 5 12.163 ± 0.138 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 1000 1000000 ss 5 7452222.400 ± 1019.460 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 1000 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 10000000 ss 5 0.275 ± 0.013 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 1000 10000000 ss 5 52.103 ± 1.439 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 1000 10000000 ss 5 50582414.400 ± 45991.882 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 1000 10000000 ss 5 42.028 ± 361.875 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 1000 10000000 ss 5 41104179.200 ± 353919379.167 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 1000 10000000 ss 5 1.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 1000 10000000 ss 5 7.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 100000000 ss 5 2.099 ± 0.251 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 1000 100000000 ss 5 35.099 ± 0.579 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 1000 100000000 ss 5 114648668.800 ± 55423.632 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 1000 100000000 ss 5 33.538 ± 117.892 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 1000 100000000 ss 5 109471334.400 ± 384807165.034 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space SORTED_UNIFORM 1000 100000000 ss 5 0.026 ± 0.223 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm SORTED_UNIFORM 1000 100000000 ss 5 83841.600 ± 721901.461 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 1000 100000000 ss 5 3.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 1000 100000000 ss 5 22.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 1000000 ss 5 0.093 ± 0.011 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10000 1000000 ss 5 20.878 ± 1.225 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10000 1000000 ss 5 13544881.600 ± 191.637 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10000 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 10000000 ss 5 0.886 ± 0.244 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10000 10000000 ss 5 46.010 ± 1.042 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10000 10000000 ss 5 74304364.800 ± 46037.614 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 10000 10000000 ss 5 24.963 ± 214.936 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 10000 10000000 ss 5 40684748.800 ± 350307956.931 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10000 10000000 ss 5 1.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 10000 10000000 ss 5 9.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 100000000 ss 5 3.826 ± 0.052 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10000 100000000 ss 5 100.180 ± 1.713 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10000 100000000 ss 5 505538409.600 ± 96799.515 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 10000 100000000 ss 5 99.990 ± 79.926 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 10000 100000000 ss 5 504574771.200 ± 402985207.107 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space SORTED_UNIFORM 10000 100000000 ss 5 0.329 ± 2.728 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm SORTED_UNIFORM 10000 100000000 ss 5 1669273.600 ± 13844467.735 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10000 100000000 ss 5 13.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 10000 100000000 ss 5 28.000 ms
```
after
```
Benchmark (assignment) (cardinality) (numDocs) Mode Cnt Score Error Units
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 1000000 ss 5 0.029 ± 0.022 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10 1000000 ss 5 12.403 ± 1.117 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10 1000000 ss 5 7146006.400 ± 30985.905 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 10000000 ss 5 0.188 ± 0.026 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10 10000000 ss 5 87.108 ± 4.467 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10 10000000 ss 5 71108766.400 ± 78261.812 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10 10000000 ss 5 95.241 ± 502.203 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10 10000000 ss 5 78014054.400 ± 411385221.551 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 10 10000000 ss 5 1.101 ± 9.482 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 10 10000000 ss 5 899424.000 ± 7744311.890 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10 10000000 ss 5 2.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10 10000000 ss 5 25.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 100000000 ss 5 1.947 ± 0.116 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10 100000000 ss 5 208.005 ± 7.371 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10 100000000 ss 5 709829240.000 ± 51497.263 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10 100000000 ss 5 201.399 ± 76.815 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10 100000000 ss 5 687026995.200 ± 247098882.679 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Old_Gen ROUND_ROBIN 10 100000000 ss 5 2.224 ± 17.872 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Old_Gen.norm ROUND_ROBIN 10 100000000 ss 5 7549747.200 ± 60592304.093 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 10 100000000 ss 5 4.788 ± 16.708 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 10 100000000 ss 5 16357785.600 ± 57215688.521 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10 100000000 ss 5 17.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10 100000000 ss 5 256.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 1000000 ss 5 0.040 ± 0.023 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 1000 1000000 ss 5 20.222 ± 1.683 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 1000 1000000 ss 5 11859302.400 ± 30898.399 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 1000 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 10000000 ss 5 0.293 ± 0.080 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 1000 10000000 ss 5 123.802 ± 11.035 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 1000 10000000 ss 5 113905795.200 ± 74048.230 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 1000 10000000 ss 5 108.134 ± 386.562 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 1000 10000000 ss 5 100663296.000 ± 360192977.778 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 1000 10000000 ss 5 6.363 ± 34.666 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 1000 10000000 ss 5 5872025.600 ± 32124407.719 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 1000 10000000 ss 5 3.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 1000 10000000 ss 5 35.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 100000000 ss 5 2.898 ± 0.114 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 1000 100000000 ss 5 255.212 ± 6.003 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 1000 100000000 ss 5 1129770475.200 ± 124788.799 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 1000 100000000 ss 5 257.389 ± 62.089 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 1000 100000000 ss 5 1139592396.800 ± 287647003.971 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Old_Gen ROUND_ROBIN 1000 100000000 ss 5 6.228 ± 32.860 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Old_Gen.norm ROUND_ROBIN 1000 100000000 ss 5 27472691.200 ± 144918864.230 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 1000 100000000 ss 5 7.259 ± 12.981 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 1000 100000000 ss 5 32086425.600 ± 56930036.769 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 1000 100000000 ss 5 24.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 1000 100000000 ss 5 1023.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 1000000 ss 5 0.066 ± 0.037 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10000 1000000 ss 5 72.601 ± 4.420 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10000 1000000 ss 5 45097956.800 ± 48245.104 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10000 1000000 ss 5 74.815 ± 644.181 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10000 1000000 ss 5 47395635.200 ± 408090712.713 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10000 1000000 ss 5 1.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10000 1000000 ss 5 14.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 10000000 ss 5 0.557 ± 0.267 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10000 10000000 ss 5 186.074 ± 47.551 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10000 10000000 ss 5 223440820.800 ± 78325.490 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10000 10000000 ss 5 138.524 ± 337.002 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10000 10000000 ss 5 171127603.200 ± 416904247.274 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 10000 10000000 ss 5 1.756 ± 15.119 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 10000 10000000 ss 5 2097152.000 ± 18057111.182 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10000 10000000 ss 5 5.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10000 10000000 ss 5 298.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 100000000 ss 5 5.260 ± 0.325 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10000 100000000 ss 5 310.016 ± 14.954 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10000 100000000 ss 5 2143068185.600 ± 64550.566 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10000 100000000 ss 5 306.636 ± 42.750 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10000 100000000 ss 5 2120220672.000 ± 339646533.455 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 10000 100000000 ss 5 8.017 ± 4.673 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 10000 100000000 ss 5 55364812.800 ± 30537345.918 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10000 100000000 ss 5 39.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10000 100000000 ss 5 4087.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 1000000 ss 5 0.017 ± 0.006 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10 1000000 ss 5 2.146 ± 0.059 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10 1000000 ss 5 1175939.200 ± 30372.447 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 10000000 ss 5 0.155 ± 0.001 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10 10000000 ss 5 10.358 ± 0.065 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10 10000000 ss 5 7563580.800 ± 30697.759 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10 10000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 100000000 ss 5 1.548 ± 0.023 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10 100000000 ss 5 28.135 ± 0.251 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10 100000000 ss 5 71579300.800 ± 78187.548 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 10 100000000 ss 5 28.278 ± 149.101 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 10 100000000 ss 5 72142028.800 ± 380384094.171 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space SORTED_UNIFORM 10 100000000 ss 5 0.028 ± 0.239 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm SORTED_UNIFORM 10 100000000 ss 5 70888.000 ± 610367.059 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10 100000000 ss 5 2.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 10 100000000 ss 5 17.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 1000000 ss 5 0.023 ± 0.032 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 1000 1000000 ss 5 13.814 ± 1.193 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 1000 1000000 ss 5 7669910.400 ± 300898.435 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 1000 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 10000000 ss 5 0.181 ± 0.037 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 1000 10000000 ss 5 66.605 ± 4.336 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 1000 10000000 ss 5 50803308.800 ± 76359.327 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 1000 10000000 ss 5 62.270 ± 536.163 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 1000 10000000 ss 5 47395635.200 ± 408090712.713 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 1000 10000000 ss 5 1.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 1000 10000000 ss 5 8.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 100000000 ss 5 1.568 ± 0.020 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 1000 100000000 ss 5 44.731 ± 0.340 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 1000 100000000 ss 5 114796513.600 ± 73178.185 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 1000 100000000 ss 5 43.084 ± 151.447 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 1000 100000000 ss 5 110729625.600 ± 389230235.896 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space SORTED_UNIFORM 1000 100000000 ss 5 0.032 ± 0.275 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm SORTED_UNIFORM 1000 100000000 ss 5 82238.400 ± 708097.426 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 1000 100000000 ss 5 3.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 1000 100000000 ss 5 24.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 1000000 ss 5 0.034 ± 0.037 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10000 1000000 ss 5 27.006 ± 2.034 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10000 1000000 ss 5 15331587.200 ± 30511.819 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10000 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 10000000 ss 5 0.186 ± 0.016 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10000 10000000 ss 5 99.771 ± 3.199 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10000 10000000 ss 5 76239598.400 ± 76412.193 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 10000 10000000 ss 5 53.525 ± 460.870 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 10000 10000000 ss 5 41523609.600 ± 357530801.403 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space SORTED_UNIFORM 10000 10000000 ss 5 0.062 ± 0.531 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm SORTED_UNIFORM 10000 10000000 ss 5 47827.200 ± 411806.616 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10000 10000000 ss 5 1.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 10000 10000000 ss 5 9.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 100000000 ss 5 1.736 ± 0.045 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10000 100000000 ss 5 184.429 ± 5.794 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10000 100000000 ss 5 507845612.800 ± 59214.207 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 10000 100000000 ss 5 171.683 ± 16.667 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 10000 100000000 ss 5 472698060.800 ± 32099022.960 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space SORTED_UNIFORM 10000 100000000 ss 5 0.901 ± 4.227 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm SORTED_UNIFORM 10000 100000000 ss 5 2492544.000 ± 11744141.868 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10000 100000000 ss 5 10.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 10000 100000000 ss 5 49.000 ms
```
Allocation can be reduced and get a speed gain by guessing the approximate sizes of the bitmaps, and by reusing a single writer.
latest commit
```
Benchmark (assignment) (cardinality) (numDocs) Mode Cnt Score Error Units
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 1000000 ss 5 0.021 ± 0.038 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 10000000 ss 5 0.175 ± 0.015 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 100000000 ss 5 1.780 ± 0.117 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 1000000 ss 5 0.024 ± 0.003 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 10000000 ss 5 0.243 ± 0.035 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 100000000 ss 5 2.711 ± 0.219 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 1000000 ss 5 0.052 ± 0.024 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 10000000 ss 5 0.539 ± 0.107 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 100000000 ss 5 4.500 ± 0.246 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 1000000 ss 5 0.016 ± 0.003 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 10000000 ss 5 0.148 ± 0.005 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 100000000 ss 5 1.412 ± 0.026 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 1000000 ss 5 0.021 ± 0.026 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 10000000 ss 5 0.165 ± 0.019 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 100000000 ss 5 1.437 ± 0.021 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 1000000 ss 5 0.022 ± 0.005 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 10000000 ss 5 0.172 ± 0.022 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 100000000 ss 5 1.581 ± 0.095 s/op
```
```
Benchmark (assignment) (cardinality) (numDocs) Mode Cnt Score Error Units
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 1000000 ss 5 0.017 ± 0.002 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10 1000000 ss 5 12.633 ± 0.155 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10 1000000 ss 5 6986539.200 ± 30442.010 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 10000000 ss 5 0.176 ± 0.012 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10 10000000 ss 5 87.350 ± 1.202 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10 10000000 ss 5 69659446.400 ± 52302.495 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10 10000000 ss 5 110.528 ± 582.795 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10 10000000 ss 5 88499814.400 ± 466642710.250 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10 10000000 ss 5 2.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10 10000000 ss 5 12.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 100000000 ss 5 1.783 ± 0.104 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10 100000000 ss 5 214.505 ± 7.603 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10 100000000 ss 5 695378540.800 ± 139270.627 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10 100000000 ss 5 214.143 ± 104.774 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10 100000000 ss 5 694576742.400 ± 352155335.840 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Old_Gen ROUND_ROBIN 10 100000000 ss 5 1.321 ± 11.370 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Old_Gen.norm ROUND_ROBIN 10 100000000 ss 5 4256358.400 ± 36648529.462 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 10 100000000 ss 5 3.111 ± 7.794 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 10 100000000 ss 5 10067219.200 ± 25155045.841 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10 100000000 ss 5 23.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10 100000000 ss 5 275.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 1000000 ss 5 0.025 ± 0.022 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 1000 1000000 ss 5 14.770 ± 1.072 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 1000 1000000 ss 5 8325694.400 ± 30481.121 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 1000 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 10000000 ss 5 0.244 ± 0.040 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 1000 10000000 ss 5 92.415 ± 4.679 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 1000 10000000 ss 5 80292582.400 ± 51274.216 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 1000 10000000 ss 5 69.550 ± 366.944 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 1000 10000000 ss 5 60817408.000 ± 320787006.897 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 1000 10000000 ss 5 2.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 1000 10000000 ss 5 22.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 100000000 ss 5 2.429 ± 0.248 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 1000 100000000 ss 5 200.418 ± 13.277 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 1000 100000000 ss 5 787876115.200 ± 256741.280 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 1000 100000000 ss 5 193.777 ± 130.177 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 1000 100000000 ss 5 762943897.600 ± 544097645.991 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Old_Gen ROUND_ROBIN 1000 100000000 ss 5 1.467 ± 12.630 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Old_Gen.norm ROUND_ROBIN 1000 100000000 ss 5 5872025.600 ± 50559911.310 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 1000 100000000 ss 5 4.343 ± 9.498 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 1000 100000000 ss 5 17196646.400 ± 38474784.725 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 1000 100000000 ss 5 23.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 1000 100000000 ss 5 746.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 1000000 ss 5 0.054 ± 0.025 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10000 1000000 ss 5 29.477 ± 2.118 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10000 1000000 ss 5 17577993.600 ± 48406.649 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10000 1000000 ss 5 30.144 ± 259.553 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10000 1000000 ss 5 18035507.200 ± 155291156.165 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10000 1000000 ss 5 1.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10000 1000000 ss 5 12.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 10000000 ss 5 0.481 ± 0.050 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10000 10000000 ss 5 205.551 ± 10.013 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10000 10000000 ss 5 230199414.400 ± 124257.456 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10000 10000000 ss 5 200.550 ± 150.026 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10000 10000000 ss 5 224395264.000 ± 162413652.372 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Old_Gen ROUND_ROBIN 10000 10000000 ss 5 1.853 ± 15.957 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Old_Gen.norm ROUND_ROBIN 10000 10000000 ss 5 2097152.000 ± 18057111.182 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 10000 10000000 ss 5 11.056 ± 41.589 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 10000 10000000 ss 5 12379680.000 ± 46553344.432 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10000 10000000 ss 5 11.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10000 10000000 ss 5 304.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 100000000 ss 5 4.505 ± 0.168 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate ROUND_ROBIN 10000 100000000 ss 5 220.361 ± 6.174 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm ROUND_ROBIN 10000 100000000 ss 5 1348977696.000 ± 48374.015 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space ROUND_ROBIN 10000 100000000 ss 5 220.473 ± 89.136 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm ROUND_ROBIN 10000 100000000 ss 5 1350146457.600 ± 565202002.425 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Old_Gen ROUND_ROBIN 10000 100000000 ss 5 0.719 ± 6.193 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Old_Gen.norm ROUND_ROBIN 10000 100000000 ss 5 4404019.200 ± 37919933.482 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space ROUND_ROBIN 10000 100000000 ss 5 5.615 ± 5.115 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm ROUND_ROBIN 10000 100000000 ss 5 34393292.800 ± 31587043.999 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count ROUND_ROBIN 10000 100000000 ss 5 36.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time ROUND_ROBIN 10000 100000000 ss 5 2728.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 1000000 ss 5 0.022 ± 0.037 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10 1000000 ss 5 2.066 ± 0.233 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10 1000000 ss 5 1148363.200 ± 30594.344 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 10000000 ss 5 0.148 ± 0.003 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10 10000000 ss 5 10.223 ± 0.105 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10 10000000 ss 5 7390347.200 ± 30613.010 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10 10000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 100000000 ss 5 1.410 ± 0.015 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10 100000000 ss 5 29.172 ± 0.183 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10 100000000 ss 5 70052542.400 ± 137755.129 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 10 100000000 ss 5 29.999 ± 158.176 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 10 100000000 ss 5 72142028.800 ± 380384094.171 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space SORTED_UNIFORM 10 100000000 ss 5 0.034 ± 0.293 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm SORTED_UNIFORM 10 100000000 ss 5 81944.000 ± 705562.553 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10 100000000 ss 5 2.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 10 100000000 ss 5 16.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 1000000 ss 5 0.025 ± 0.040 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 1000 1000000 ss 5 12.373 ± 1.158 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 1000 1000000 ss 5 6932864.000 ± 30927.743 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 1000 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 10000000 ss 5 0.164 ± 0.015 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 1000 10000000 ss 5 66.973 ± 1.648 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 1000 10000000 ss 5 49529336.000 ± 76261.103 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 1000 10000000 ss 5 55.023 ± 473.763 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 1000 10000000 ss 5 41104179.200 ± 353919379.167 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 1000 10000000 ss 5 1.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 1000 10000000 ss 5 8.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 100000000 ss 5 1.434 ± 0.018 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 1000 100000000 ss 5 46.177 ± 0.422 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 1000 100000000 ss 5 112162459.200 ± 73253.706 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 1000 100000000 ss 5 44.479 ± 156.352 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 1000 100000000 ss 5 108213043.200 ± 380384094.171 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space SORTED_UNIFORM 1000 100000000 ss 5 0.033 ± 0.282 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm SORTED_UNIFORM 1000 100000000 ss 5 80251.200 ± 686689.981 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 1000 100000000 ss 5 3.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 1000 100000000 ss 5 23.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 1000000 ss 5 0.023 ± 0.011 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10000 1000000 ss 5 20.115 ± 0.796 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10000 1000000 ss 5 11170392.000 ± 30879.694 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10000 1000000 ss 5 ≈ 0 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 10000000 ss 5 0.167 ± 0.016 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10000 10000000 ss 5 91.921 ± 7.404 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10000 10000000 ss 5 69227684.800 ± 52043.344 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 10000 10000000 ss 5 114.528 ± 604.014 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 10000 10000000 ss 5 87241523.200 ± 459999369.696 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10000 10000000 ss 5 2.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 10000 10000000 ss 5 17.000 ms
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 100000000 ss 5 1.603 ± 0.123 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate SORTED_UNIFORM 10000 100000000 ss 5 190.115 ± 9.171 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.alloc.rate.norm SORTED_UNIFORM 10000 100000000 ss 5 495109988.800 ± 101261.939 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space SORTED_UNIFORM 10000 100000000 ss 5 197.630 ± 142.159 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Eden_Space.norm SORTED_UNIFORM 10000 100000000 ss 5 514641100.800 ± 368630518.221 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space SORTED_UNIFORM 10000 100000000 ss 5 0.842 ± 4.357 MB/sec
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.churn.G1_Survivor_Space.norm SORTED_UNIFORM 10000 100000000 ss 5 2221792.000 ± 11637986.706 B/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.count SORTED_UNIFORM 10000 100000000 ss 5 11.000 counts
BenchmarkOffheapBitmapInvertedIndexCreator.seal:·gc.time SORTED_UNIFORM 10000 100000000 ss 5 49.000 ms
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] richardstartin commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r537132715
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
##########
@@ -181,33 +188,51 @@ public void seal()
}
// Create bitmaps from inverted index buffers and serialize them to file
- try (DataOutputStream offsetDataStream = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)));
- FileOutputStream bitmapFileStream = new FileOutputStream(_invertedIndexFile);
- DataOutputStream bitmapDataStream = new DataOutputStream(new BufferedOutputStream(bitmapFileStream))) {
- int bitmapOffset = (_cardinality + 1) * Integer.BYTES;
- offsetDataStream.writeInt(bitmapOffset);
- bitmapFileStream.getChannel().position(bitmapOffset);
-
+ ByteBuffer offsetBuffer = null;
+ ByteBuffer bitmapBuffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ // map the offsets buffer
+ final int startOfBitmaps = (_cardinality + 1) * Integer.BYTES;
+ int bitmapOffset = startOfBitmaps;
+ offsetBuffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, bitmapOffset).order(LITTLE_ENDIAN);
+ offsetBuffer.putInt(reverseBytes(bitmapOffset));
+ RoaringBitmap[] bitmaps = new RoaringBitmap[_cardinality];
+ RoaringBitmapWriter<RoaringBitmap> writer = RoaringBitmapWriter.writer()
+ .initialCapacity(((_nextDocId - 1) >>> 16) / _cardinality).get();
Review comment:
I felt this heuristic was distracting so removed it.
In case this comes up in the future, expecting the range to be `[0, _nextDocId)` would allocate far too much memory for datasets sorted by the indexed dimension, but would be the right choice if the association between row id and dictionary id were uniformly random. Some metadata could probably be associated with each dictionary id (number of documents, min, max, density?) in memory to use these methods could likely be used more effectively, and not in a way that overfits to a benchmark.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-io edited a comment on pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#issuecomment-739096511
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=h1) Report
> Merging [#6320](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=desc) (03de308) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) (1beaab5) will **increase** coverage by `7.71%`.
> The diff coverage is `68.40%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6320 +/- ##
==========================================
+ Coverage 66.44% 74.16% +7.71%
==========================================
Files 1075 1281 +206
Lines 54773 61823 +7050
Branches 8168 8986 +818
==========================================
+ Hits 36396 45852 +9456
+ Misses 15700 13007 -2693
- Partials 2677 2964 +287
```
| Flag | Coverage Δ | |
|---|---|---|
| integration | `45.79% <52.01%> (?)` | |
| unittests | `65.37% <45.12%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `100.00% <ø> (ø)` | |
| [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `52.83% <0.00%> (-13.84%)` | :arrow_down: |
| [...ker/routing/instanceselector/InstanceSelector.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9pbnN0YW5jZXNlbGVjdG9yL0luc3RhbmNlU2VsZWN0b3IuamF2YQ==) | `100.00% <ø> (ø)` | |
| [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `44.44% <0.00%> (-4.40%)` | :arrow_down: |
| [...not/common/assignment/InstancePartitionsUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnNVdGlscy5qYXZh) | `78.57% <ø> (+5.40%)` | :arrow_up: |
| [.../apache/pinot/common/exception/QueryException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZXhjZXB0aW9uL1F1ZXJ5RXhjZXB0aW9uLmphdmE=) | `90.27% <ø> (+5.55%)` | :arrow_up: |
| [...pinot/common/function/AggregationFunctionType.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vQWdncmVnYXRpb25GdW5jdGlvblR5cGUuamF2YQ==) | `98.27% <ø> (-1.73%)` | :arrow_down: |
| [.../pinot/common/function/DateTimePatternHandler.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRGF0ZVRpbWVQYXR0ZXJuSGFuZGxlci5qYXZh) | `83.33% <ø> (ø)` | |
| [...ot/common/function/FunctionDefinitionRegistry.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRnVuY3Rpb25EZWZpbml0aW9uUmVnaXN0cnkuamF2YQ==) | `88.88% <ø> (+44.44%)` | :arrow_up: |
| [...org/apache/pinot/common/function/FunctionInfo.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRnVuY3Rpb25JbmZvLmphdmE=) | `100.00% <ø> (ø)` | |
| ... and [1066 more](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=footer). Last update [41a7722...e3ec220](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] richardstartin commented on pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
richardstartin commented on pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#issuecomment-739429849
@Jackie-Jiang with the last commit there was another performance improvement and a reduction in allocation rate following your suggestions. I have also tried to adhere to the style guidance.
before
```
Benchmark (assignment) (cardinality) (numDocs) Mode Cnt Score Error Units
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 1000000 ss 5 0.030 ± 0.003 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 10000000 ss 5 0.374 ± 0.265 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 100000000 ss 5 4.077 ± 0.191 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 1000000 ss 5 0.090 ± 0.012 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 10000000 ss 5 1.070 ± 0.304 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 100000000 ss 5 11.465 ± 0.308 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 1000000 ss 5 0.152 ± 0.040 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 10000000 ss 5 1.639 ± 0.067 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 100000000 ss 5 17.002 ± 0.217 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 1000000 ss 5 0.021 ± 0.002 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 10000000 ss 5 0.212 ± 0.004 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 100000000 ss 5 2.283 ± 0.109 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 1000000 ss 5 0.075 ± 0.002 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 10000000 ss 5 0.296 ± 0.189 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 100000000 ss 5 2.094 ± 0.284 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 1000000 ss 5 0.093 ± 0.014 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 10000000 ss 5 0.891 ± 0.327 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 100000000 ss 5 3.736 ± 0.067 s/op
```
after
```
Benchmark (assignment) (cardinality) (numDocs) Mode Cnt Score Error Units
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 1000000 ss 5 0.016 ± 0.004 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 10000000 ss 5 0.179 ± 0.021 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 100000000 ss 5 1.711 ± 0.014 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 1000000 ss 5 0.024 ± 0.002 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 10000000 ss 5 0.233 ± 0.034 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 100000000 ss 5 2.118 ± 0.007 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 1000000 ss 5 0.048 ± 0.025 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 10000000 ss 5 0.401 ± 0.090 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 100000000 ss 5 3.575 ± 0.007 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 1000000 ss 5 0.014 ± 0.002 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 10000000 ss 5 0.123 ± 0.004 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 100000000 ss 5 1.444 ± 0.101 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 1000000 ss 5 0.018 ± 0.002 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 10000000 ss 5 0.147 ± 0.029 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 100000000 ss 5 1.513 ± 0.082 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 1000000 ss 5 0.022 ± 0.007 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 10000000 ss 5 0.175 ± 0.022 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 100000000 ss 5 1.350 ± 0.040 s/op
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] richardstartin commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r537828464
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
##########
@@ -181,33 +184,45 @@ public void seal()
}
// Create bitmaps from inverted index buffers and serialize them to file
- try (DataOutputStream offsetDataStream = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)));
- FileOutputStream bitmapFileStream = new FileOutputStream(_invertedIndexFile);
- DataOutputStream bitmapDataStream = new DataOutputStream(new BufferedOutputStream(bitmapFileStream))) {
- int bitmapOffset = (_cardinality + 1) * Integer.BYTES;
- offsetDataStream.writeInt(bitmapOffset);
- bitmapFileStream.getChannel().position(bitmapOffset);
-
+ ByteBuffer offsetBuffer = null;
+ ByteBuffer bitmapBuffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ // map the offsets buffer
+ int startOfBitmaps = (_cardinality + 1) * Integer.BYTES;
+ offsetBuffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, startOfBitmaps)
+ .order(ByteOrder.BIG_ENDIAN);
+ offsetBuffer.putInt(startOfBitmaps);
+ bitmapBuffer = channel.map(FileChannel.MapMode.READ_WRITE, startOfBitmaps, Integer.MAX_VALUE - startOfBitmaps)
+ .order(ByteOrder.LITTLE_ENDIAN);
+ RoaringBitmapWriter<RoaringBitmap> writer = RoaringBitmapWriter.writer().get();
Review comment:
I disabled run compression, which will save some cycles if this is only used for sparse bitmaps.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r537124621
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OnHeapBitmapInvertedIndexCreator.java
##########
@@ -19,68 +19,96 @@
package org.apache.pinot.core.segment.creator.impl.inv;
import com.google.common.base.Preconditions;
-import java.io.BufferedOutputStream;
-import java.io.DataOutputStream;
+
+
import java.io.File;
-import java.io.FileOutputStream;
import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.channels.FileChannel;
+
import org.apache.commons.io.FileUtils;
import org.apache.pinot.core.segment.creator.DictionaryBasedInvertedIndexCreator;
import org.apache.pinot.core.segment.creator.impl.V1Constants;
-import org.roaringbitmap.buffer.MutableRoaringBitmap;
+import org.apache.pinot.core.util.CleanerUtil;
+import org.roaringbitmap.RoaringBitmap;
+import org.roaringbitmap.RoaringBitmapWriter;
/**
* Implementation of {@link DictionaryBasedInvertedIndexCreator} that uses on-heap memory.
*/
public final class OnHeapBitmapInvertedIndexCreator implements DictionaryBasedInvertedIndexCreator {
private final File _invertedIndexFile;
- private final MutableRoaringBitmap[] _bitmaps;
+ private final RoaringBitmapWriter<RoaringBitmap>[] _bitmapWriters;
private int _nextDocId;
+ @SuppressWarnings("unchecked")
public OnHeapBitmapInvertedIndexCreator(File indexDir, String columnName, int cardinality) {
_invertedIndexFile = new File(indexDir, columnName + V1Constants.Indexes.BITMAP_INVERTED_INDEX_FILE_EXTENSION);
- _bitmaps = new MutableRoaringBitmap[cardinality];
+ _bitmapWriters = new RoaringBitmapWriter[cardinality];
for (int i = 0; i < cardinality; i++) {
- _bitmaps[i] = new MutableRoaringBitmap();
+ _bitmapWriters[i] = RoaringBitmapWriter.writer().get();
}
}
@Override
public void add(int dictId) {
- _bitmaps[dictId].add(_nextDocId++);
+ _bitmapWriters[dictId].add(_nextDocId++);
}
@Override
public void add(int[] dictIds, int length) {
for (int i = 0; i < length; i++) {
- _bitmaps[dictIds[i]].add(_nextDocId);
+ _bitmapWriters[dictIds[i]].add(_nextDocId);
}
_nextDocId++;
}
@Override
public void seal()
throws IOException {
- try (DataOutputStream out = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)))) {
+ // calculate file size
+ int startOfBitmaps = (_bitmapWriters.length + 1) * Integer.BYTES;
+ int size = 0;
+ for (RoaringBitmapWriter<RoaringBitmap> writer : _bitmapWriters) {
+ size += writer.get().serializedSizeInBytes();
Review comment:
Maybe we can use the same way as the off-heap creator to preallocate a huge buffer then truncate to save the calculation of the serialized size (we calculate that twice for each bitmap)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg commented on pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
kishoreg commented on pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#issuecomment-739781501
> @Jackie-Jiang since the branch is churning somewhat do you mind if I squash the commits to what we have so far and force push?
it's ok to do that if it helps you. Note that, we automatically perform squash and merge during the final merge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-io edited a comment on pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#issuecomment-739096511
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=h1) Report
> Merging [#6320](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=desc) (e3ec220) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) (1beaab5) will **decrease** coverage by `1.08%`.
> The diff coverage is `45.12%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6320 +/- ##
==========================================
- Coverage 66.44% 65.36% -1.09%
==========================================
Files 1075 1281 +206
Lines 54773 61823 +7050
Branches 8168 8986 +818
==========================================
+ Hits 36396 40412 +4016
- Misses 15700 18521 +2821
- Partials 2677 2890 +213
```
| Flag | Coverage Δ | |
|---|---|---|
| unittests | `65.36% <45.12%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...e/pinot/broker/api/resources/PinotBrokerDebug.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYXBpL3Jlc291cmNlcy9QaW5vdEJyb2tlckRlYnVnLmphdmE=) | `0.00% <0.00%> (-79.32%)` | :arrow_down: |
| [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `71.42% <ø> (-28.58%)` | :arrow_down: |
| [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `33.96% <0.00%> (-32.71%)` | :arrow_down: |
| [...ker/routing/instanceselector/InstanceSelector.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9pbnN0YW5jZXNlbGVjdG9yL0luc3RhbmNlU2VsZWN0b3IuamF2YQ==) | `100.00% <ø> (ø)` | |
| [...ava/org/apache/pinot/client/AbstractResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Fic3RyYWN0UmVzdWx0U2V0LmphdmE=) | `66.66% <0.00%> (+9.52%)` | :arrow_up: |
| [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `35.55% <0.00%> (-13.29%)` | :arrow_down: |
| [...inot/client/JsonAsyncHttpPinotClientTransport.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0pzb25Bc3luY0h0dHBQaW5vdENsaWVudFRyYW5zcG9ydC5qYXZh) | `10.90% <0.00%> (-51.10%)` | :arrow_down: |
| [...not/common/assignment/InstancePartitionsUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnNVdGlscy5qYXZh) | `73.80% <ø> (+0.63%)` | :arrow_up: |
| [.../apache/pinot/common/exception/QueryException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZXhjZXB0aW9uL1F1ZXJ5RXhjZXB0aW9uLmphdmE=) | `90.27% <ø> (+5.55%)` | :arrow_up: |
| [...pinot/common/function/AggregationFunctionType.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vQWdncmVnYXRpb25GdW5jdGlvblR5cGUuamF2YQ==) | `98.27% <ø> (-1.73%)` | :arrow_down: |
| ... and [1128 more](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=footer). Last update [41a7722...e3ec220](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r537789457
##########
File path: pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkOffheapBitmapInvertedIndexCreator.java
##########
@@ -0,0 +1,105 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pinot.perf;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.core.segment.creator.impl.inv.OffHeapBitmapInvertedIndexCreator;
+import org.apache.pinot.spi.data.DimensionFieldSpec;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.openjdk.jmh.annotations.*;
+import org.openjdk.jmh.runner.Runner;
+import org.openjdk.jmh.runner.options.ChainedOptionsBuilder;
+import org.openjdk.jmh.runner.options.OptionsBuilder;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+
+@State(Scope.Benchmark)
+public class BenchmarkOffheapBitmapInvertedIndexCreator {
Review comment:
Please reformat this file to fix the indentation (should be 2 spaces) for this file
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
##########
@@ -181,33 +184,45 @@ public void seal()
}
// Create bitmaps from inverted index buffers and serialize them to file
- try (DataOutputStream offsetDataStream = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)));
- FileOutputStream bitmapFileStream = new FileOutputStream(_invertedIndexFile);
- DataOutputStream bitmapDataStream = new DataOutputStream(new BufferedOutputStream(bitmapFileStream))) {
- int bitmapOffset = (_cardinality + 1) * Integer.BYTES;
- offsetDataStream.writeInt(bitmapOffset);
- bitmapFileStream.getChannel().position(bitmapOffset);
-
+ ByteBuffer offsetBuffer = null;
+ ByteBuffer bitmapBuffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ // map the offsets buffer
+ int startOfBitmaps = (_cardinality + 1) * Integer.BYTES;
+ offsetBuffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, startOfBitmaps)
+ .order(ByteOrder.BIG_ENDIAN);
+ offsetBuffer.putInt(startOfBitmaps);
+ bitmapBuffer = channel.map(FileChannel.MapMode.READ_WRITE, startOfBitmaps, Integer.MAX_VALUE - startOfBitmaps)
+ .order(ByteOrder.LITTLE_ENDIAN);
+ RoaringBitmapWriter<RoaringBitmap> writer = RoaringBitmapWriter.writer().get();
Review comment:
I see your point. We won't create bitmap inverted index if the values are sorted (we have another RLE format), so in most cases, the docIds will spread over all the arrays. But for high cardinality case where each bitmap only contain very few values, pre-allocation might end up allocating too many arrays. So agree with letting it grow automatically.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r537123298
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
##########
@@ -181,33 +188,51 @@ public void seal()
}
// Create bitmaps from inverted index buffers and serialize them to file
- try (DataOutputStream offsetDataStream = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)));
- FileOutputStream bitmapFileStream = new FileOutputStream(_invertedIndexFile);
- DataOutputStream bitmapDataStream = new DataOutputStream(new BufferedOutputStream(bitmapFileStream))) {
- int bitmapOffset = (_cardinality + 1) * Integer.BYTES;
- offsetDataStream.writeInt(bitmapOffset);
- bitmapFileStream.getChannel().position(bitmapOffset);
-
+ ByteBuffer offsetBuffer = null;
+ ByteBuffer bitmapBuffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ // map the offsets buffer
+ final int startOfBitmaps = (_cardinality + 1) * Integer.BYTES;
+ int bitmapOffset = startOfBitmaps;
+ offsetBuffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, bitmapOffset).order(LITTLE_ENDIAN);
+ offsetBuffer.putInt(reverseBytes(bitmapOffset));
+ RoaringBitmap[] bitmaps = new RoaringBitmap[_cardinality];
+ RoaringBitmapWriter<RoaringBitmap> writer = RoaringBitmapWriter.writer()
+ .initialCapacity(((_nextDocId - 1) >>> 16) / _cardinality).get();
Review comment:
From my understanding, this method preallocate the arrays based on the size of the prefix keys. Since we are going to reuse this writer for all the bitmaps, we will always end up using all the prefix keys from `(0, _nextDocId)`, and we should use `.expectedRange(0, _nextDocId)` to preallocate all of them. Maybe I missed something here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] richardstartin commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r536916030
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OnHeapBitmapInvertedIndexCreator.java
##########
@@ -19,68 +19,85 @@
package org.apache.pinot.core.segment.creator.impl.inv;
import com.google.common.base.Preconditions;
-import java.io.BufferedOutputStream;
-import java.io.DataOutputStream;
+
+
import java.io.File;
-import java.io.FileOutputStream;
import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+
import org.apache.commons.io.FileUtils;
import org.apache.pinot.core.segment.creator.DictionaryBasedInvertedIndexCreator;
import org.apache.pinot.core.segment.creator.impl.V1Constants;
+import org.apache.pinot.core.util.CleanerUtil;
+import org.roaringbitmap.RoaringBitmapWriter;
import org.roaringbitmap.buffer.MutableRoaringBitmap;
+import static java.nio.ByteOrder.LITTLE_ENDIAN;
+
/**
* Implementation of {@link DictionaryBasedInvertedIndexCreator} that uses on-heap memory.
*/
public final class OnHeapBitmapInvertedIndexCreator implements DictionaryBasedInvertedIndexCreator {
private final File _invertedIndexFile;
- private final MutableRoaringBitmap[] _bitmaps;
+ private final RoaringBitmapWriter<MutableRoaringBitmap>[] _bitmapWriters;
private int _nextDocId;
+ @SuppressWarnings("unchecked")
public OnHeapBitmapInvertedIndexCreator(File indexDir, String columnName, int cardinality) {
_invertedIndexFile = new File(indexDir, columnName + V1Constants.Indexes.BITMAP_INVERTED_INDEX_FILE_EXTENSION);
- _bitmaps = new MutableRoaringBitmap[cardinality];
+ _bitmapWriters = new RoaringBitmapWriter[cardinality];
for (int i = 0; i < cardinality; i++) {
- _bitmaps[i] = new MutableRoaringBitmap();
+ _bitmapWriters[i] = RoaringBitmapWriter.bufferWriter().get();
}
}
@Override
public void add(int dictId) {
- _bitmaps[dictId].add(_nextDocId++);
+ _bitmapWriters[dictId].add(_nextDocId++);
}
@Override
public void add(int[] dictIds, int length) {
for (int i = 0; i < length; i++) {
- _bitmaps[dictIds[i]].add(_nextDocId);
+ _bitmapWriters[dictIds[i]].add(_nextDocId);
}
_nextDocId++;
}
@Override
public void seal()
throws IOException {
- try (DataOutputStream out = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)))) {
+ // calculate file size
+ int size = (_bitmapWriters.length + 1) * Integer.BYTES;
+ for (RoaringBitmapWriter<MutableRoaringBitmap> writer : _bitmapWriters) {
+ size += writer.get().serializedSizeInBytes();
+ // Check for int overflow
+ Preconditions.checkState(size > 0, "Inverted index file: %s exceeds 2GB limit", _invertedIndexFile);
+ }
+ ByteBuffer buffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ buffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, size).order(LITTLE_ENDIAN);
Review comment:
I was writing big endian bytes _to a part of the buffer_ whereas the bulk of the file is serialised by the roaring bitmap library, which is little endian by specification. So I was doing a little extra work on the offsets, to favour the bitmaps. Does that make sense?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-io edited a comment on pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#issuecomment-739096511
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=h1) Report
> Merging [#6320](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=desc) (0a7af36) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) (1beaab5) will **increase** coverage by `7.67%`.
> The diff coverage is `68.40%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6320 +/- ##
==========================================
+ Coverage 66.44% 74.12% +7.67%
==========================================
Files 1075 1281 +206
Lines 54773 61813 +7040
Branches 8168 8986 +818
==========================================
+ Hits 36396 45818 +9422
+ Misses 15700 13033 -2667
- Partials 2677 2962 +285
```
| Flag | Coverage Δ | |
|---|---|---|
| integration | `45.65% <52.01%> (?)` | |
| unittests | `65.34% <45.12%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `100.00% <ø> (ø)` | |
| [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `52.83% <0.00%> (-13.84%)` | :arrow_down: |
| [...ker/routing/instanceselector/InstanceSelector.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9pbnN0YW5jZXNlbGVjdG9yL0luc3RhbmNlU2VsZWN0b3IuamF2YQ==) | `100.00% <ø> (ø)` | |
| [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `44.44% <0.00%> (-4.40%)` | :arrow_down: |
| [...not/common/assignment/InstancePartitionsUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnNVdGlscy5qYXZh) | `78.57% <ø> (+5.40%)` | :arrow_up: |
| [.../apache/pinot/common/exception/QueryException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZXhjZXB0aW9uL1F1ZXJ5RXhjZXB0aW9uLmphdmE=) | `90.27% <ø> (+5.55%)` | :arrow_up: |
| [...pinot/common/function/AggregationFunctionType.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vQWdncmVnYXRpb25GdW5jdGlvblR5cGUuamF2YQ==) | `98.27% <ø> (-1.73%)` | :arrow_down: |
| [.../pinot/common/function/DateTimePatternHandler.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRGF0ZVRpbWVQYXR0ZXJuSGFuZGxlci5qYXZh) | `83.33% <ø> (ø)` | |
| [...ot/common/function/FunctionDefinitionRegistry.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRnVuY3Rpb25EZWZpbml0aW9uUmVnaXN0cnkuamF2YQ==) | `88.88% <ø> (+44.44%)` | :arrow_up: |
| [...org/apache/pinot/common/function/FunctionInfo.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRnVuY3Rpb25JbmZvLmphdmE=) | `100.00% <ø> (ø)` | |
| ... and [1066 more](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=footer). Last update [41a7722...0a7af36](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] richardstartin commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r536912358
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
##########
@@ -181,33 +188,49 @@ public void seal()
}
// Create bitmaps from inverted index buffers and serialize them to file
- try (DataOutputStream offsetDataStream = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)));
- FileOutputStream bitmapFileStream = new FileOutputStream(_invertedIndexFile);
- DataOutputStream bitmapDataStream = new DataOutputStream(new BufferedOutputStream(bitmapFileStream))) {
- int bitmapOffset = (_cardinality + 1) * Integer.BYTES;
- offsetDataStream.writeInt(bitmapOffset);
- bitmapFileStream.getChannel().position(bitmapOffset);
-
+ ByteBuffer offsetBuffer = null;
+ ByteBuffer bitmapBuffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
Review comment:
I had a hard time believing it when this came up, but did enough profiling to figure out what was going on. The difference is _up to_ 20x, and it depends on the composition of the bitmap: `DataOutput` penalises dense bitmaps more than others. `DataOutput` has to do things like manually extract bytes from `int`, `long` etc. and this adds up to a lot compared to a simple move instruction. Because of endianness, you get away without reversing the bytes, and with `ByteBuffer` various bounds checks get eliminated. You can see the difference even with a pre-sized `ByteArrayOutputStream`, so it's not a question of unbuffered IO.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#issuecomment-739398210
Find this blog from @richardstartin explaining the optimization very well: https://richardstartin.github.io/posts/roaringbitmaps-construction-performance
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang merged pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang merged pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] richardstartin commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r536910850
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
##########
@@ -181,33 +188,51 @@ public void seal()
}
// Create bitmaps from inverted index buffers and serialize them to file
- try (DataOutputStream offsetDataStream = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)));
- FileOutputStream bitmapFileStream = new FileOutputStream(_invertedIndexFile);
- DataOutputStream bitmapDataStream = new DataOutputStream(new BufferedOutputStream(bitmapFileStream))) {
- int bitmapOffset = (_cardinality + 1) * Integer.BYTES;
- offsetDataStream.writeInt(bitmapOffset);
- bitmapFileStream.getChannel().position(bitmapOffset);
-
+ ByteBuffer offsetBuffer = null;
+ ByteBuffer bitmapBuffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ // map the offsets buffer
+ final int startOfBitmaps = (_cardinality + 1) * Integer.BYTES;
+ int bitmapOffset = startOfBitmaps;
+ offsetBuffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, bitmapOffset).order(LITTLE_ENDIAN);
+ offsetBuffer.putInt(reverseBytes(bitmapOffset));
+ RoaringBitmap[] bitmaps = new RoaringBitmap[_cardinality];
Review comment:
> Do RoaringBitmap and MutableRoaringBitmap serialize to the same bytes?
Yes, they do.
> For off-heap creator, we don't want to keep all bitmaps on heap. We should try to create and serialize the bitmaps one by one. Not sure about the cost if we map a buffer per bitmap
I can see the incentive not to allocate so much on heap, but the alternative is mapping many times (and it's a relatively costly operation) or mapping too many bytes and truncating the channel. I have done the latter in the past to get round this, but there may be downsides I'm unaware of. I've updated to use this strategy and allocation pressure is indeed reduced.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] richardstartin commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r537874091
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
##########
@@ -181,33 +184,45 @@ public void seal()
}
// Create bitmaps from inverted index buffers and serialize them to file
- try (DataOutputStream offsetDataStream = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)));
- FileOutputStream bitmapFileStream = new FileOutputStream(_invertedIndexFile);
- DataOutputStream bitmapDataStream = new DataOutputStream(new BufferedOutputStream(bitmapFileStream))) {
- int bitmapOffset = (_cardinality + 1) * Integer.BYTES;
- offsetDataStream.writeInt(bitmapOffset);
- bitmapFileStream.getChannel().position(bitmapOffset);
-
+ ByteBuffer offsetBuffer = null;
+ ByteBuffer bitmapBuffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ // map the offsets buffer
+ int startOfBitmaps = (_cardinality + 1) * Integer.BYTES;
+ offsetBuffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, startOfBitmaps)
+ .order(ByteOrder.BIG_ENDIAN);
+ offsetBuffer.putInt(startOfBitmaps);
+ bitmapBuffer = channel.map(FileChannel.MapMode.READ_WRITE, startOfBitmaps, Integer.MAX_VALUE - startOfBitmaps)
+ .order(ByteOrder.LITTLE_ENDIAN);
+ RoaringBitmapWriter<RoaringBitmap> writer = RoaringBitmapWriter.writer().get();
Review comment:
No run length encoding was being done before
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] richardstartin commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r537132814
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OnHeapBitmapInvertedIndexCreator.java
##########
@@ -19,68 +19,96 @@
package org.apache.pinot.core.segment.creator.impl.inv;
import com.google.common.base.Preconditions;
-import java.io.BufferedOutputStream;
-import java.io.DataOutputStream;
+
+
import java.io.File;
-import java.io.FileOutputStream;
import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.channels.FileChannel;
+
import org.apache.commons.io.FileUtils;
import org.apache.pinot.core.segment.creator.DictionaryBasedInvertedIndexCreator;
import org.apache.pinot.core.segment.creator.impl.V1Constants;
-import org.roaringbitmap.buffer.MutableRoaringBitmap;
+import org.apache.pinot.core.util.CleanerUtil;
+import org.roaringbitmap.RoaringBitmap;
+import org.roaringbitmap.RoaringBitmapWriter;
/**
* Implementation of {@link DictionaryBasedInvertedIndexCreator} that uses on-heap memory.
*/
public final class OnHeapBitmapInvertedIndexCreator implements DictionaryBasedInvertedIndexCreator {
private final File _invertedIndexFile;
- private final MutableRoaringBitmap[] _bitmaps;
+ private final RoaringBitmapWriter<RoaringBitmap>[] _bitmapWriters;
private int _nextDocId;
+ @SuppressWarnings("unchecked")
public OnHeapBitmapInvertedIndexCreator(File indexDir, String columnName, int cardinality) {
_invertedIndexFile = new File(indexDir, columnName + V1Constants.Indexes.BITMAP_INVERTED_INDEX_FILE_EXTENSION);
- _bitmaps = new MutableRoaringBitmap[cardinality];
+ _bitmapWriters = new RoaringBitmapWriter[cardinality];
for (int i = 0; i < cardinality; i++) {
- _bitmaps[i] = new MutableRoaringBitmap();
+ _bitmapWriters[i] = RoaringBitmapWriter.writer().get();
}
}
@Override
public void add(int dictId) {
- _bitmaps[dictId].add(_nextDocId++);
+ _bitmapWriters[dictId].add(_nextDocId++);
}
@Override
public void add(int[] dictIds, int length) {
for (int i = 0; i < length; i++) {
- _bitmaps[dictIds[i]].add(_nextDocId);
+ _bitmapWriters[dictIds[i]].add(_nextDocId);
}
_nextDocId++;
}
@Override
public void seal()
throws IOException {
- try (DataOutputStream out = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)))) {
+ // calculate file size
+ int startOfBitmaps = (_bitmapWriters.length + 1) * Integer.BYTES;
+ int size = 0;
+ for (RoaringBitmapWriter<RoaringBitmap> writer : _bitmapWriters) {
+ size += writer.get().serializedSizeInBytes();
Review comment:
Yes, good suggestion, I applied this change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-io edited a comment on pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#issuecomment-739096511
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=h1) Report
> Merging [#6320](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=desc) (72e9d8d) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) (1beaab5) will **decrease** coverage by `20.81%`.
> The diff coverage is `52.01%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6320 +/- ##
===========================================
- Coverage 66.44% 45.63% -20.82%
===========================================
Files 1075 1283 +208
Lines 54773 61937 +7164
Branches 8168 8996 +828
===========================================
- Hits 36396 28265 -8131
- Misses 15700 31364 +15664
+ Partials 2677 2308 -369
```
| Flag | Coverage Δ | |
|---|---|---|
| integration | `45.63% <52.01%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `100.00% <ø> (ø)` | |
| [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `52.83% <0.00%> (-13.84%)` | :arrow_down: |
| [...org/apache/pinot/broker/queryquota/HitCounter.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9IaXRDb3VudGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...che/pinot/broker/queryquota/MaxHitRateTracker.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9NYXhIaXRSYXRlVHJhY2tlci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/pinot/broker/queryquota/QueryQuotaEntity.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9RdWVyeVF1b3RhRW50aXR5LmphdmE=) | `0.00% <0.00%> (-50.00%)` | :arrow_down: |
| [...ker/routing/instanceselector/InstanceSelector.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9pbnN0YW5jZXNlbGVjdG9yL0luc3RhbmNlU2VsZWN0b3IuamF2YQ==) | `100.00% <ø> (ø)` | |
| [...ceselector/StrictReplicaGroupInstanceSelector.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9pbnN0YW5jZXNlbGVjdG9yL1N0cmljdFJlcGxpY2FHcm91cEluc3RhbmNlU2VsZWN0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `22.22% <0.00%> (-26.62%)` | :arrow_down: |
| [...not/common/assignment/InstancePartitionsUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnNVdGlscy5qYXZh) | `64.28% <ø> (-8.89%)` | :arrow_down: |
| [.../apache/pinot/common/exception/QueryException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZXhjZXB0aW9uL1F1ZXJ5RXhjZXB0aW9uLmphdmE=) | `90.27% <ø> (+5.55%)` | :arrow_up: |
| ... and [1278 more](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=footer). Last update [41a7722...0d90376](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-io edited a comment on pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#issuecomment-739096511
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=h1) Report
> Merging [#6320](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=desc) (92d9953) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) (1beaab5) will **decrease** coverage by `1.09%`.
> The diff coverage is `45.12%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6320 +/- ##
==========================================
- Coverage 66.44% 65.35% -1.10%
==========================================
Files 1075 1281 +206
Lines 54773 61815 +7042
Branches 8168 8983 +815
==========================================
+ Hits 36396 40397 +4001
- Misses 15700 18527 +2827
- Partials 2677 2891 +214
```
| Flag | Coverage Δ | |
|---|---|---|
| unittests | `65.35% <45.12%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...e/pinot/broker/api/resources/PinotBrokerDebug.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYXBpL3Jlc291cmNlcy9QaW5vdEJyb2tlckRlYnVnLmphdmE=) | `0.00% <0.00%> (-79.32%)` | :arrow_down: |
| [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `71.42% <ø> (-28.58%)` | :arrow_down: |
| [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `33.96% <0.00%> (-32.71%)` | :arrow_down: |
| [...ker/routing/instanceselector/InstanceSelector.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9pbnN0YW5jZXNlbGVjdG9yL0luc3RhbmNlU2VsZWN0b3IuamF2YQ==) | `100.00% <ø> (ø)` | |
| [...ava/org/apache/pinot/client/AbstractResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Fic3RyYWN0UmVzdWx0U2V0LmphdmE=) | `66.66% <0.00%> (+9.52%)` | :arrow_up: |
| [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `35.55% <0.00%> (-13.29%)` | :arrow_down: |
| [...inot/client/JsonAsyncHttpPinotClientTransport.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0pzb25Bc3luY0h0dHBQaW5vdENsaWVudFRyYW5zcG9ydC5qYXZh) | `10.90% <0.00%> (-51.10%)` | :arrow_down: |
| [...not/common/assignment/InstancePartitionsUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnNVdGlscy5qYXZh) | `73.80% <ø> (+0.63%)` | :arrow_up: |
| [.../apache/pinot/common/exception/QueryException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZXhjZXB0aW9uL1F1ZXJ5RXhjZXB0aW9uLmphdmE=) | `90.27% <ø> (+5.55%)` | :arrow_up: |
| [...pinot/common/function/AggregationFunctionType.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vQWdncmVnYXRpb25GdW5jdGlvblR5cGUuamF2YQ==) | `98.27% <ø> (-1.73%)` | :arrow_down: |
| ... and [1128 more](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=footer). Last update [41a7722...92d9953](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] richardstartin commented on pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
richardstartin commented on pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#issuecomment-739076353
Regarding the test failure, this is because the bitmap is being run length encoded incrementally. It becomes deterministic when setting
```java
_bitmapWriters[i] = RoaringBitmapWriter.bufferWriter().runCompress(false).get();
```
To make the test deterministic, RLE could be disabled, but this is bad for file size. Or the off-heap index creator could run-optimise its bitmaps, but that's bad for speed compared to doing it incrementally. Alternatively the test could be modified to check that the bitmaps are equal after memory mapping.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] richardstartin commented on pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
richardstartin commented on pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#issuecomment-739154073
I added a benchmark which times the single shot time to seal an off heap index creator, with a range of cardinalities, numbers of docs, and different assignment strategies of document ids to dictionary ids. One is the best case for roaring (sorted), the other is the worst case (round robin). The benchmark takes quite a long time to run so I only ran one fork and reduced the number of iterations, but this PR demonstrates a significant improvement:
before
```
Benchmark (assignment) (cardinality) (numDocs) Mode Cnt Score Error Units
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 1000000 ss 5 0.030 ± 0.003 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 10000000 ss 5 0.374 ± 0.265 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 100000000 ss 5 4.077 ± 0.191 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 1000000 ss 5 0.090 ± 0.012 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 10000000 ss 5 1.070 ± 0.304 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 100000000 ss 5 11.465 ± 0.308 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 1000000 ss 5 0.152 ± 0.040 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 10000000 ss 5 1.639 ± 0.067 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 100000000 ss 5 17.002 ± 0.217 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 1000000 ss 5 0.021 ± 0.002 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 10000000 ss 5 0.212 ± 0.004 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 100000000 ss 5 2.283 ± 0.109 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 1000000 ss 5 0.075 ± 0.002 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 10000000 ss 5 0.296 ± 0.189 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 100000000 ss 5 2.094 ± 0.284 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 1000000 ss 5 0.093 ± 0.014 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 10000000 ss 5 0.891 ± 0.327 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 100000000 ss 5 3.736 ± 0.067 s/op
```
after
```
Benchmark (assignment) (cardinality) (numDocs) Mode Cnt Score Error Units
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 1000000 ss 5 0.021 ± 0.011 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 10000000 ss 5 0.195 ± 0.026 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10 100000000 ss 5 1.966 ± 0.133 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 1000000 ss 5 0.032 ± 0.009 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 10000000 ss 5 0.287 ± 0.108 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 1000 100000000 ss 5 2.943 ± 0.176 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 1000000 ss 5 0.057 ± 0.036 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 10000000 ss 5 0.491 ± 0.131 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal ROUND_ROBIN 10000 100000000 ss 5 5.502 ± 0.225 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 1000000 ss 5 0.018 ± 0.005 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 10000000 ss 5 0.159 ± 0.012 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10 100000000 ss 5 1.554 ± 0.028 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 1000000 ss 5 0.021 ± 0.006 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 10000000 ss 5 0.181 ± 0.028 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 1000 100000000 ss 5 1.580 ± 0.028 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 1000000 ss 5 0.031 ± 0.003 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 10000000 ss 5 0.193 ± 0.020 s/op
BenchmarkOffheapBitmapInvertedIndexCreator.seal SORTED_UNIFORM 10000 100000000 ss 5 1.752 ± 0.089 s/op
```
Please review the benchmark and let me know if you think the improvements look interesting - I am not going to be able to run cluster scale performance tests on this, but have prior experience of these APIs reducing resource cost in the system I wrote them for.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r536887337
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
##########
@@ -19,18 +19,25 @@
package org.apache.pinot.core.segment.creator.impl.inv;
import com.google.common.base.Preconditions;
-import java.io.BufferedOutputStream;
+
import java.io.Closeable;
-import java.io.DataOutputStream;
import java.io.File;
-import java.io.FileOutputStream;
import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+
import org.apache.commons.io.FileUtils;
import org.apache.pinot.core.segment.creator.DictionaryBasedInvertedIndexCreator;
import org.apache.pinot.core.segment.creator.impl.V1Constants;
import org.apache.pinot.core.segment.memory.PinotDataBuffer;
+import org.apache.pinot.core.util.CleanerUtil;
import org.apache.pinot.spi.data.FieldSpec;
-import org.roaringbitmap.buffer.MutableRoaringBitmap;
+import org.roaringbitmap.RoaringBitmap;
+import org.roaringbitmap.RoaringBitmapWriter;
+
+import static java.lang.Integer.reverseBytes;
Review comment:
Don't use static import in production class
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
##########
@@ -181,33 +188,51 @@ public void seal()
}
// Create bitmaps from inverted index buffers and serialize them to file
- try (DataOutputStream offsetDataStream = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)));
- FileOutputStream bitmapFileStream = new FileOutputStream(_invertedIndexFile);
- DataOutputStream bitmapDataStream = new DataOutputStream(new BufferedOutputStream(bitmapFileStream))) {
- int bitmapOffset = (_cardinality + 1) * Integer.BYTES;
- offsetDataStream.writeInt(bitmapOffset);
- bitmapFileStream.getChannel().position(bitmapOffset);
-
+ ByteBuffer offsetBuffer = null;
+ ByteBuffer bitmapBuffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ // map the offsets buffer
+ final int startOfBitmaps = (_cardinality + 1) * Integer.BYTES;
+ int bitmapOffset = startOfBitmaps;
+ offsetBuffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, bitmapOffset).order(LITTLE_ENDIAN);
+ offsetBuffer.putInt(reverseBytes(bitmapOffset));
+ RoaringBitmap[] bitmaps = new RoaringBitmap[_cardinality];
Review comment:
For off-heap creator, we don't want to keep all bitmaps on heap. We should try to create and serialize the bitmaps one by one. Not sure about the cost if we map a buffer per bitmap
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
##########
@@ -181,33 +188,51 @@ public void seal()
}
// Create bitmaps from inverted index buffers and serialize them to file
- try (DataOutputStream offsetDataStream = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)));
- FileOutputStream bitmapFileStream = new FileOutputStream(_invertedIndexFile);
- DataOutputStream bitmapDataStream = new DataOutputStream(new BufferedOutputStream(bitmapFileStream))) {
- int bitmapOffset = (_cardinality + 1) * Integer.BYTES;
- offsetDataStream.writeInt(bitmapOffset);
- bitmapFileStream.getChannel().position(bitmapOffset);
-
+ ByteBuffer offsetBuffer = null;
+ ByteBuffer bitmapBuffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ // map the offsets buffer
+ final int startOfBitmaps = (_cardinality + 1) * Integer.BYTES;
+ int bitmapOffset = startOfBitmaps;
+ offsetBuffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, bitmapOffset).order(LITTLE_ENDIAN);
Review comment:
The offset buffer does not have to be LE, as all the values are written as BE
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
##########
@@ -181,33 +188,51 @@ public void seal()
}
// Create bitmaps from inverted index buffers and serialize them to file
- try (DataOutputStream offsetDataStream = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)));
- FileOutputStream bitmapFileStream = new FileOutputStream(_invertedIndexFile);
- DataOutputStream bitmapDataStream = new DataOutputStream(new BufferedOutputStream(bitmapFileStream))) {
- int bitmapOffset = (_cardinality + 1) * Integer.BYTES;
- offsetDataStream.writeInt(bitmapOffset);
- bitmapFileStream.getChannel().position(bitmapOffset);
-
+ ByteBuffer offsetBuffer = null;
+ ByteBuffer bitmapBuffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ // map the offsets buffer
+ final int startOfBitmaps = (_cardinality + 1) * Integer.BYTES;
Review comment:
(nit) we don't usually use final for local variable
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
##########
@@ -181,33 +188,51 @@ public void seal()
}
// Create bitmaps from inverted index buffers and serialize them to file
- try (DataOutputStream offsetDataStream = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)));
- FileOutputStream bitmapFileStream = new FileOutputStream(_invertedIndexFile);
- DataOutputStream bitmapDataStream = new DataOutputStream(new BufferedOutputStream(bitmapFileStream))) {
- int bitmapOffset = (_cardinality + 1) * Integer.BYTES;
- offsetDataStream.writeInt(bitmapOffset);
- bitmapFileStream.getChannel().position(bitmapOffset);
-
+ ByteBuffer offsetBuffer = null;
+ ByteBuffer bitmapBuffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ // map the offsets buffer
+ final int startOfBitmaps = (_cardinality + 1) * Integer.BYTES;
+ int bitmapOffset = startOfBitmaps;
+ offsetBuffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, bitmapOffset).order(LITTLE_ENDIAN);
+ offsetBuffer.putInt(reverseBytes(bitmapOffset));
+ RoaringBitmap[] bitmaps = new RoaringBitmap[_cardinality];
Review comment:
Do `RoaringBitmap` and `MutableRoaringBitmap` serialize to the same bytes?
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
##########
@@ -181,33 +188,51 @@ public void seal()
}
// Create bitmaps from inverted index buffers and serialize them to file
- try (DataOutputStream offsetDataStream = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)));
- FileOutputStream bitmapFileStream = new FileOutputStream(_invertedIndexFile);
- DataOutputStream bitmapDataStream = new DataOutputStream(new BufferedOutputStream(bitmapFileStream))) {
- int bitmapOffset = (_cardinality + 1) * Integer.BYTES;
- offsetDataStream.writeInt(bitmapOffset);
- bitmapFileStream.getChannel().position(bitmapOffset);
-
+ ByteBuffer offsetBuffer = null;
+ ByteBuffer bitmapBuffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ // map the offsets buffer
+ final int startOfBitmaps = (_cardinality + 1) * Integer.BYTES;
+ int bitmapOffset = startOfBitmaps;
+ offsetBuffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, bitmapOffset).order(LITTLE_ENDIAN);
+ offsetBuffer.putInt(reverseBytes(bitmapOffset));
+ RoaringBitmap[] bitmaps = new RoaringBitmap[_cardinality];
+ RoaringBitmapWriter<RoaringBitmap> writer = RoaringBitmapWriter.writer()
+ .initialCapacity(((_nextDocId - 1) >>> 16) / _cardinality).get();
Review comment:
Why do you need to divide it with `_cardinality`? Though this should be `.expectedRange(0, _nextDocId)`?
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
##########
@@ -181,33 +188,51 @@ public void seal()
}
// Create bitmaps from inverted index buffers and serialize them to file
- try (DataOutputStream offsetDataStream = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)));
- FileOutputStream bitmapFileStream = new FileOutputStream(_invertedIndexFile);
- DataOutputStream bitmapDataStream = new DataOutputStream(new BufferedOutputStream(bitmapFileStream))) {
- int bitmapOffset = (_cardinality + 1) * Integer.BYTES;
- offsetDataStream.writeInt(bitmapOffset);
- bitmapFileStream.getChannel().position(bitmapOffset);
-
+ ByteBuffer offsetBuffer = null;
+ ByteBuffer bitmapBuffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ // map the offsets buffer
+ final int startOfBitmaps = (_cardinality + 1) * Integer.BYTES;
+ int bitmapOffset = startOfBitmaps;
+ offsetBuffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, bitmapOffset).order(LITTLE_ENDIAN);
+ offsetBuffer.putInt(reverseBytes(bitmapOffset));
+ RoaringBitmap[] bitmaps = new RoaringBitmap[_cardinality];
+ RoaringBitmapWriter<RoaringBitmap> writer = RoaringBitmapWriter.writer()
+ .initialCapacity(((_nextDocId - 1) >>> 16) / _cardinality).get();
int startIndex = 0;
for (int dictId = 0; dictId < _cardinality; dictId++) {
- MutableRoaringBitmap bitmap = new MutableRoaringBitmap();
int endIndex = getInt(_invertedIndexLengthBuffer, dictId);
for (int i = startIndex; i < endIndex; i++) {
- bitmap.add(getInt(_invertedIndexValueBuffer, i));
+ writer.add(getInt(_invertedIndexValueBuffer, i));
}
- startIndex = endIndex;
-
- // Write offset and bitmap into file
- bitmapOffset += bitmap.serializedSizeInBytes();
+ bitmaps[dictId] = writer.get();
+ writer.reset();
+ int serializedSize = bitmaps[dictId].serializedSizeInBytes();
+ bitmapOffset += serializedSize;
// Check for int overflow
Preconditions.checkState(bitmapOffset > 0, "Inverted index file: %s exceeds 2GB limit", _invertedIndexFile);
- offsetDataStream.writeInt(bitmapOffset);
- bitmap.serialize(bitmapDataStream);
+ // write offset into file
+ offsetBuffer.putInt(reverseBytes(bitmapOffset));
+ startIndex = endIndex;
+ }
+ // we know how long the file should be now, so can map it
+ bitmapBuffer = channel.map(FileChannel.MapMode.READ_WRITE, startOfBitmaps, bitmapOffset - startOfBitmaps);
+ for (RoaringBitmap bitmap : bitmaps) {
+ bitmap.serialize(bitmapBuffer);
}
} catch (Exception e) {
FileUtils.deleteQuietly(_invertedIndexFile);
throw e;
+ } finally {
+ if (CleanerUtil.UNMAP_SUPPORTED) {
+ CleanerUtil.BufferCleaner cleaner = CleanerUtil.getCleaner();
+ if (null != offsetBuffer) {
Review comment:
(nit) same for other places
```suggestion
if (offsetBuffer != null) {
```
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OnHeapBitmapInvertedIndexCreator.java
##########
@@ -19,68 +19,85 @@
package org.apache.pinot.core.segment.creator.impl.inv;
import com.google.common.base.Preconditions;
-import java.io.BufferedOutputStream;
-import java.io.DataOutputStream;
+
+
import java.io.File;
-import java.io.FileOutputStream;
import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+
import org.apache.commons.io.FileUtils;
import org.apache.pinot.core.segment.creator.DictionaryBasedInvertedIndexCreator;
import org.apache.pinot.core.segment.creator.impl.V1Constants;
+import org.apache.pinot.core.util.CleanerUtil;
+import org.roaringbitmap.RoaringBitmapWriter;
import org.roaringbitmap.buffer.MutableRoaringBitmap;
+import static java.nio.ByteOrder.LITTLE_ENDIAN;
+
/**
* Implementation of {@link DictionaryBasedInvertedIndexCreator} that uses on-heap memory.
*/
public final class OnHeapBitmapInvertedIndexCreator implements DictionaryBasedInvertedIndexCreator {
private final File _invertedIndexFile;
- private final MutableRoaringBitmap[] _bitmaps;
+ private final RoaringBitmapWriter<MutableRoaringBitmap>[] _bitmapWriters;
private int _nextDocId;
+ @SuppressWarnings("unchecked")
public OnHeapBitmapInvertedIndexCreator(File indexDir, String columnName, int cardinality) {
_invertedIndexFile = new File(indexDir, columnName + V1Constants.Indexes.BITMAP_INVERTED_INDEX_FILE_EXTENSION);
- _bitmaps = new MutableRoaringBitmap[cardinality];
+ _bitmapWriters = new RoaringBitmapWriter[cardinality];
for (int i = 0; i < cardinality; i++) {
- _bitmaps[i] = new MutableRoaringBitmap();
+ _bitmapWriters[i] = RoaringBitmapWriter.bufferWriter().get();
}
}
@Override
public void add(int dictId) {
- _bitmaps[dictId].add(_nextDocId++);
+ _bitmapWriters[dictId].add(_nextDocId++);
}
@Override
public void add(int[] dictIds, int length) {
for (int i = 0; i < length; i++) {
- _bitmaps[dictIds[i]].add(_nextDocId);
+ _bitmapWriters[dictIds[i]].add(_nextDocId);
}
_nextDocId++;
}
@Override
public void seal()
throws IOException {
- try (DataOutputStream out = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)))) {
+ // calculate file size
+ int size = (_bitmapWriters.length + 1) * Integer.BYTES;
+ for (RoaringBitmapWriter<MutableRoaringBitmap> writer : _bitmapWriters) {
+ size += writer.get().serializedSizeInBytes();
+ // Check for int overflow
+ Preconditions.checkState(size > 0, "Inverted index file: %s exceeds 2GB limit", _invertedIndexFile);
+ }
+ ByteBuffer buffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ buffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, size).order(LITTLE_ENDIAN);
// Write bitmap offsets
- int bitmapOffset = (_bitmaps.length + 1) * Integer.BYTES;
- out.writeInt(bitmapOffset);
- for (MutableRoaringBitmap bitmap : _bitmaps) {
- bitmapOffset += bitmap.serializedSizeInBytes();
- // Check for int overflow
- Preconditions.checkState(bitmapOffset > 0, "Inverted index file: %s exceeds 2GB limit", _invertedIndexFile);
- out.writeInt(bitmapOffset);
+ int bitmapOffset = (_bitmapWriters.length + 1) * Integer.BYTES;
+ buffer.putInt(Integer.reverseBytes(bitmapOffset));
+ for (RoaringBitmapWriter<MutableRoaringBitmap> writer : _bitmapWriters) {
+ bitmapOffset += writer.getUnderlying().serializedSizeInBytes();
Review comment:
Do you need to `flush()` before `getUnderlying()`? Or just use `get()` to retrieve the bitmap?
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OnHeapBitmapInvertedIndexCreator.java
##########
@@ -19,68 +19,85 @@
package org.apache.pinot.core.segment.creator.impl.inv;
import com.google.common.base.Preconditions;
-import java.io.BufferedOutputStream;
-import java.io.DataOutputStream;
+
+
import java.io.File;
-import java.io.FileOutputStream;
import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+
import org.apache.commons.io.FileUtils;
import org.apache.pinot.core.segment.creator.DictionaryBasedInvertedIndexCreator;
import org.apache.pinot.core.segment.creator.impl.V1Constants;
+import org.apache.pinot.core.util.CleanerUtil;
+import org.roaringbitmap.RoaringBitmapWriter;
import org.roaringbitmap.buffer.MutableRoaringBitmap;
+import static java.nio.ByteOrder.LITTLE_ENDIAN;
+
/**
* Implementation of {@link DictionaryBasedInvertedIndexCreator} that uses on-heap memory.
*/
public final class OnHeapBitmapInvertedIndexCreator implements DictionaryBasedInvertedIndexCreator {
private final File _invertedIndexFile;
- private final MutableRoaringBitmap[] _bitmaps;
+ private final RoaringBitmapWriter<MutableRoaringBitmap>[] _bitmapWriters;
private int _nextDocId;
+ @SuppressWarnings("unchecked")
public OnHeapBitmapInvertedIndexCreator(File indexDir, String columnName, int cardinality) {
_invertedIndexFile = new File(indexDir, columnName + V1Constants.Indexes.BITMAP_INVERTED_INDEX_FILE_EXTENSION);
- _bitmaps = new MutableRoaringBitmap[cardinality];
+ _bitmapWriters = new RoaringBitmapWriter[cardinality];
for (int i = 0; i < cardinality; i++) {
- _bitmaps[i] = new MutableRoaringBitmap();
+ _bitmapWriters[i] = RoaringBitmapWriter.bufferWriter().get();
}
}
@Override
public void add(int dictId) {
- _bitmaps[dictId].add(_nextDocId++);
+ _bitmapWriters[dictId].add(_nextDocId++);
}
@Override
public void add(int[] dictIds, int length) {
for (int i = 0; i < length; i++) {
- _bitmaps[dictIds[i]].add(_nextDocId);
+ _bitmapWriters[dictIds[i]].add(_nextDocId);
}
_nextDocId++;
}
@Override
public void seal()
throws IOException {
- try (DataOutputStream out = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)))) {
+ // calculate file size
+ int size = (_bitmapWriters.length + 1) * Integer.BYTES;
+ for (RoaringBitmapWriter<MutableRoaringBitmap> writer : _bitmapWriters) {
+ size += writer.get().serializedSizeInBytes();
+ // Check for int overflow
+ Preconditions.checkState(size > 0, "Inverted index file: %s exceeds 2GB limit", _invertedIndexFile);
+ }
+ ByteBuffer buffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ buffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, size).order(LITTLE_ENDIAN);
Review comment:
Maybe keeping 2 buffers similar to the off-heap one?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] richardstartin commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r537132715
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
##########
@@ -181,33 +188,51 @@ public void seal()
}
// Create bitmaps from inverted index buffers and serialize them to file
- try (DataOutputStream offsetDataStream = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)));
- FileOutputStream bitmapFileStream = new FileOutputStream(_invertedIndexFile);
- DataOutputStream bitmapDataStream = new DataOutputStream(new BufferedOutputStream(bitmapFileStream))) {
- int bitmapOffset = (_cardinality + 1) * Integer.BYTES;
- offsetDataStream.writeInt(bitmapOffset);
- bitmapFileStream.getChannel().position(bitmapOffset);
-
+ ByteBuffer offsetBuffer = null;
+ ByteBuffer bitmapBuffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ // map the offsets buffer
+ final int startOfBitmaps = (_cardinality + 1) * Integer.BYTES;
+ int bitmapOffset = startOfBitmaps;
+ offsetBuffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, bitmapOffset).order(LITTLE_ENDIAN);
+ offsetBuffer.putInt(reverseBytes(bitmapOffset));
+ RoaringBitmap[] bitmaps = new RoaringBitmap[_cardinality];
+ RoaringBitmapWriter<RoaringBitmap> writer = RoaringBitmapWriter.writer()
+ .initialCapacity(((_nextDocId - 1) >>> 16) / _cardinality).get();
Review comment:
I felt this heuristic was distracting so removed it.
In case this comes up in the future, expecting the range to be `[0, _nextDocId)` would allocate far too much memory for datasets sorted by the indexed dimension, but would be the right choice if the association between row id and dictionary id were uniformly random. Some in-memory metadata could probably be associated with each dictionary id (number of documents, min, max, density?) and would mean these methods could likely be used more effectively, and not in a way that overfits to a benchmark.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r536517138
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OnHeapBitmapInvertedIndexCreator.java
##########
@@ -19,68 +19,85 @@
package org.apache.pinot.core.segment.creator.impl.inv;
import com.google.common.base.Preconditions;
-import java.io.BufferedOutputStream;
-import java.io.DataOutputStream;
+
+
import java.io.File;
-import java.io.FileOutputStream;
import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+
import org.apache.commons.io.FileUtils;
import org.apache.pinot.core.segment.creator.DictionaryBasedInvertedIndexCreator;
import org.apache.pinot.core.segment.creator.impl.V1Constants;
+import org.apache.pinot.core.util.CleanerUtil;
+import org.roaringbitmap.RoaringBitmapWriter;
import org.roaringbitmap.buffer.MutableRoaringBitmap;
+import static java.nio.ByteOrder.LITTLE_ENDIAN;
+
/**
* Implementation of {@link DictionaryBasedInvertedIndexCreator} that uses on-heap memory.
*/
public final class OnHeapBitmapInvertedIndexCreator implements DictionaryBasedInvertedIndexCreator {
private final File _invertedIndexFile;
- private final MutableRoaringBitmap[] _bitmaps;
+ private final RoaringBitmapWriter<MutableRoaringBitmap>[] _bitmapWriters;
private int _nextDocId;
+ @SuppressWarnings("unchecked")
public OnHeapBitmapInvertedIndexCreator(File indexDir, String columnName, int cardinality) {
_invertedIndexFile = new File(indexDir, columnName + V1Constants.Indexes.BITMAP_INVERTED_INDEX_FILE_EXTENSION);
- _bitmaps = new MutableRoaringBitmap[cardinality];
+ _bitmapWriters = new RoaringBitmapWriter[cardinality];
for (int i = 0; i < cardinality; i++) {
- _bitmaps[i] = new MutableRoaringBitmap();
+ _bitmapWriters[i] = RoaringBitmapWriter.bufferWriter().get();
}
}
@Override
public void add(int dictId) {
- _bitmaps[dictId].add(_nextDocId++);
+ _bitmapWriters[dictId].add(_nextDocId++);
}
@Override
public void add(int[] dictIds, int length) {
for (int i = 0; i < length; i++) {
- _bitmaps[dictIds[i]].add(_nextDocId);
+ _bitmapWriters[dictIds[i]].add(_nextDocId);
}
_nextDocId++;
}
@Override
public void seal()
throws IOException {
- try (DataOutputStream out = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)))) {
+ // calculate file size
+ int size = (_bitmapWriters.length + 1) * Integer.BYTES;
+ for (RoaringBitmapWriter<MutableRoaringBitmap> writer : _bitmapWriters) {
+ size += writer.get().serializedSizeInBytes();
+ // Check for int overflow
+ Preconditions.checkState(size > 0, "Inverted index file: %s exceeds 2GB limit", _invertedIndexFile);
+ }
+ ByteBuffer buffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ buffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, size).order(LITTLE_ENDIAN);
Review comment:
I am assuming the choice of LITTLE_ENDIAN comes from the fact that index generation (typically happening on x86 machines) will not require byte swap when writing the index?
When we load the index, we specifically use BIG_ENDIAN byte order because throughout the index generation code we use BIG_ENDIAN. This works with overhead of byte swap for both read and write.
I am confused how is this change expected to work today for the reader? The reader will load the file specifying the order as BIG_ENDIAN. This means every read operation will swap the bytes on x86. However, with this change index file is generated in LE format which doesn't require any swapping. So swapping bytes during inverted index read will result in incorrect data imo. I am surprised that tests are passing.
See the code in index buffer loader. This code doesn't know index file was written in LE format.
```
// Backward-compatible: index file is always big-endian
PinotDataBuffer buffer;
if (readMode == ReadMode.heap) {
buffer = PinotDataBuffer.loadFile(indexFile, fromFilePos, size, ByteOrder.BIG_ENDIAN, context);
} else {
buffer = PinotDataBuffer.mapFile(indexFile, true, fromFilePos, size, ByteOrder.BIG_ENDIAN, context);
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg commented on pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
kishoreg commented on pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#issuecomment-740314450
> I'm working on a feature which requires extracting the common part of the on-heap and off-heap inverted index creator. Can I add a commit to this PR?
Let's not mix with this PR. We can do that after we push this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-io edited a comment on pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#issuecomment-739096511
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=h1) Report
> Merging [#6320](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=desc) (725e218) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) (1beaab5) will **decrease** coverage by `1.05%`.
> The diff coverage is `45.12%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6320 +/- ##
==========================================
- Coverage 66.44% 65.39% -1.06%
==========================================
Files 1075 1283 +208
Lines 54773 61978 +7205
Branches 8168 8998 +830
==========================================
+ Hits 36396 40528 +4132
- Misses 15700 18563 +2863
- Partials 2677 2887 +210
```
| Flag | Coverage Δ | |
|---|---|---|
| unittests | `65.39% <45.12%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...e/pinot/broker/api/resources/PinotBrokerDebug.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYXBpL3Jlc291cmNlcy9QaW5vdEJyb2tlckRlYnVnLmphdmE=) | `0.00% <0.00%> (-79.32%)` | :arrow_down: |
| [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `71.42% <ø> (-28.58%)` | :arrow_down: |
| [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `33.96% <0.00%> (-32.71%)` | :arrow_down: |
| [...ker/routing/instanceselector/InstanceSelector.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9pbnN0YW5jZXNlbGVjdG9yL0luc3RhbmNlU2VsZWN0b3IuamF2YQ==) | `100.00% <ø> (ø)` | |
| [...ava/org/apache/pinot/client/AbstractResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Fic3RyYWN0UmVzdWx0U2V0LmphdmE=) | `66.66% <0.00%> (+9.52%)` | :arrow_up: |
| [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `35.55% <0.00%> (-13.29%)` | :arrow_down: |
| [...inot/client/JsonAsyncHttpPinotClientTransport.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0pzb25Bc3luY0h0dHBQaW5vdENsaWVudFRyYW5zcG9ydC5qYXZh) | `10.90% <0.00%> (-51.10%)` | :arrow_down: |
| [...not/common/assignment/InstancePartitionsUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnNVdGlscy5qYXZh) | `73.80% <ø> (+0.63%)` | :arrow_up: |
| [.../apache/pinot/common/exception/QueryException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZXhjZXB0aW9uL1F1ZXJ5RXhjZXB0aW9uLmphdmE=) | `90.27% <ø> (+5.55%)` | :arrow_up: |
| [...pinot/common/function/AggregationFunctionType.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vQWdncmVnYXRpb25GdW5jdGlvblR5cGUuamF2YQ==) | `100.00% <ø> (ø)` | |
| ... and [1131 more](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=footer). Last update [41a7722...725e218](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-io edited a comment on pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#issuecomment-739096511
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=h1) Report
> Merging [#6320](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=desc) (725e218) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) (1beaab5) will **increase** coverage by `7.67%`.
> The diff coverage is `68.40%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6320 +/- ##
==========================================
+ Coverage 66.44% 74.11% +7.67%
==========================================
Files 1075 1283 +208
Lines 54773 61978 +7205
Branches 8168 8998 +830
==========================================
+ Hits 36396 45938 +9542
+ Misses 15700 13075 -2625
- Partials 2677 2965 +288
```
| Flag | Coverage Δ | |
|---|---|---|
| integration | `45.57% <52.01%> (?)` | |
| unittests | `65.39% <45.12%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `100.00% <ø> (ø)` | |
| [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `52.83% <0.00%> (-13.84%)` | :arrow_down: |
| [...ker/routing/instanceselector/InstanceSelector.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9pbnN0YW5jZXNlbGVjdG9yL0luc3RhbmNlU2VsZWN0b3IuamF2YQ==) | `100.00% <ø> (ø)` | |
| [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `44.44% <0.00%> (-4.40%)` | :arrow_down: |
| [...not/common/assignment/InstancePartitionsUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnNVdGlscy5qYXZh) | `78.57% <ø> (+5.40%)` | :arrow_up: |
| [.../apache/pinot/common/exception/QueryException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZXhjZXB0aW9uL1F1ZXJ5RXhjZXB0aW9uLmphdmE=) | `90.27% <ø> (+5.55%)` | :arrow_up: |
| [...pinot/common/function/AggregationFunctionType.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vQWdncmVnYXRpb25GdW5jdGlvblR5cGUuamF2YQ==) | `100.00% <ø> (ø)` | |
| [.../pinot/common/function/DateTimePatternHandler.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRGF0ZVRpbWVQYXR0ZXJuSGFuZGxlci5qYXZh) | `83.33% <ø> (ø)` | |
| [...ot/common/function/FunctionDefinitionRegistry.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRnVuY3Rpb25EZWZpbml0aW9uUmVnaXN0cnkuamF2YQ==) | `88.88% <ø> (+44.44%)` | :arrow_up: |
| [...org/apache/pinot/common/function/FunctionInfo.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRnVuY3Rpb25JbmZvLmphdmE=) | `100.00% <ø> (ø)` | |
| ... and [1069 more](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=footer). Last update [41a7722...725e218](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] richardstartin commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r536911397
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OnHeapBitmapInvertedIndexCreator.java
##########
@@ -19,68 +19,85 @@
package org.apache.pinot.core.segment.creator.impl.inv;
import com.google.common.base.Preconditions;
-import java.io.BufferedOutputStream;
-import java.io.DataOutputStream;
+
+
import java.io.File;
-import java.io.FileOutputStream;
import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+
import org.apache.commons.io.FileUtils;
import org.apache.pinot.core.segment.creator.DictionaryBasedInvertedIndexCreator;
import org.apache.pinot.core.segment.creator.impl.V1Constants;
+import org.apache.pinot.core.util.CleanerUtil;
+import org.roaringbitmap.RoaringBitmapWriter;
import org.roaringbitmap.buffer.MutableRoaringBitmap;
+import static java.nio.ByteOrder.LITTLE_ENDIAN;
+
/**
* Implementation of {@link DictionaryBasedInvertedIndexCreator} that uses on-heap memory.
*/
public final class OnHeapBitmapInvertedIndexCreator implements DictionaryBasedInvertedIndexCreator {
private final File _invertedIndexFile;
- private final MutableRoaringBitmap[] _bitmaps;
+ private final RoaringBitmapWriter<MutableRoaringBitmap>[] _bitmapWriters;
private int _nextDocId;
+ @SuppressWarnings("unchecked")
public OnHeapBitmapInvertedIndexCreator(File indexDir, String columnName, int cardinality) {
_invertedIndexFile = new File(indexDir, columnName + V1Constants.Indexes.BITMAP_INVERTED_INDEX_FILE_EXTENSION);
- _bitmaps = new MutableRoaringBitmap[cardinality];
+ _bitmapWriters = new RoaringBitmapWriter[cardinality];
for (int i = 0; i < cardinality; i++) {
- _bitmaps[i] = new MutableRoaringBitmap();
+ _bitmapWriters[i] = RoaringBitmapWriter.bufferWriter().get();
}
}
@Override
public void add(int dictId) {
- _bitmaps[dictId].add(_nextDocId++);
+ _bitmapWriters[dictId].add(_nextDocId++);
}
@Override
public void add(int[] dictIds, int length) {
for (int i = 0; i < length; i++) {
- _bitmaps[dictIds[i]].add(_nextDocId);
+ _bitmapWriters[dictIds[i]].add(_nextDocId);
}
_nextDocId++;
}
@Override
public void seal()
throws IOException {
- try (DataOutputStream out = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)))) {
+ // calculate file size
+ int size = (_bitmapWriters.length + 1) * Integer.BYTES;
+ for (RoaringBitmapWriter<MutableRoaringBitmap> writer : _bitmapWriters) {
+ size += writer.get().serializedSizeInBytes();
+ // Check for int overflow
+ Preconditions.checkState(size > 0, "Inverted index file: %s exceeds 2GB limit", _invertedIndexFile);
+ }
+ ByteBuffer buffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ buffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, size).order(LITTLE_ENDIAN);
// Write bitmap offsets
- int bitmapOffset = (_bitmaps.length + 1) * Integer.BYTES;
- out.writeInt(bitmapOffset);
- for (MutableRoaringBitmap bitmap : _bitmaps) {
- bitmapOffset += bitmap.serializedSizeInBytes();
- // Check for int overflow
- Preconditions.checkState(bitmapOffset > 0, "Inverted index file: %s exceeds 2GB limit", _invertedIndexFile);
- out.writeInt(bitmapOffset);
+ int bitmapOffset = (_bitmapWriters.length + 1) * Integer.BYTES;
+ buffer.putInt(Integer.reverseBytes(bitmapOffset));
+ for (RoaringBitmapWriter<MutableRoaringBitmap> writer : _bitmapWriters) {
+ bitmapOffset += writer.getUnderlying().serializedSizeInBytes();
Review comment:
This API is a little bit overcomplicated. `getUnderlying()` gets the bitmap, without flushing. `flush()` flushes. `get()` flushes and gets the bitmap. I shouldn't have used `getUnderlying()` here because it's confusing, perhaps it should be deprecated
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OnHeapBitmapInvertedIndexCreator.java
##########
@@ -19,68 +19,85 @@
package org.apache.pinot.core.segment.creator.impl.inv;
import com.google.common.base.Preconditions;
-import java.io.BufferedOutputStream;
-import java.io.DataOutputStream;
+
+
import java.io.File;
-import java.io.FileOutputStream;
import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+
import org.apache.commons.io.FileUtils;
import org.apache.pinot.core.segment.creator.DictionaryBasedInvertedIndexCreator;
import org.apache.pinot.core.segment.creator.impl.V1Constants;
+import org.apache.pinot.core.util.CleanerUtil;
+import org.roaringbitmap.RoaringBitmapWriter;
import org.roaringbitmap.buffer.MutableRoaringBitmap;
+import static java.nio.ByteOrder.LITTLE_ENDIAN;
+
/**
* Implementation of {@link DictionaryBasedInvertedIndexCreator} that uses on-heap memory.
*/
public final class OnHeapBitmapInvertedIndexCreator implements DictionaryBasedInvertedIndexCreator {
private final File _invertedIndexFile;
- private final MutableRoaringBitmap[] _bitmaps;
+ private final RoaringBitmapWriter<MutableRoaringBitmap>[] _bitmapWriters;
private int _nextDocId;
+ @SuppressWarnings("unchecked")
public OnHeapBitmapInvertedIndexCreator(File indexDir, String columnName, int cardinality) {
_invertedIndexFile = new File(indexDir, columnName + V1Constants.Indexes.BITMAP_INVERTED_INDEX_FILE_EXTENSION);
- _bitmaps = new MutableRoaringBitmap[cardinality];
+ _bitmapWriters = new RoaringBitmapWriter[cardinality];
for (int i = 0; i < cardinality; i++) {
- _bitmaps[i] = new MutableRoaringBitmap();
+ _bitmapWriters[i] = RoaringBitmapWriter.bufferWriter().get();
}
}
@Override
public void add(int dictId) {
- _bitmaps[dictId].add(_nextDocId++);
+ _bitmapWriters[dictId].add(_nextDocId++);
}
@Override
public void add(int[] dictIds, int length) {
for (int i = 0; i < length; i++) {
- _bitmaps[dictIds[i]].add(_nextDocId);
+ _bitmapWriters[dictIds[i]].add(_nextDocId);
}
_nextDocId++;
}
@Override
public void seal()
throws IOException {
- try (DataOutputStream out = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)))) {
+ // calculate file size
+ int size = (_bitmapWriters.length + 1) * Integer.BYTES;
+ for (RoaringBitmapWriter<MutableRoaringBitmap> writer : _bitmapWriters) {
+ size += writer.get().serializedSizeInBytes();
+ // Check for int overflow
+ Preconditions.checkState(size > 0, "Inverted index file: %s exceeds 2GB limit", _invertedIndexFile);
+ }
+ ByteBuffer buffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ buffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, size).order(LITTLE_ENDIAN);
Review comment:
Yes that's a good idea.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-io edited a comment on pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#issuecomment-739096511
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=h1) Report
> Merging [#6320](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=desc) (0d90376) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) (1beaab5) will **decrease** coverage by `1.12%`.
> The diff coverage is `45.12%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6320 +/- ##
==========================================
- Coverage 66.44% 65.31% -1.13%
==========================================
Files 1075 1283 +208
Lines 54773 61937 +7164
Branches 8168 8996 +828
==========================================
+ Hits 36396 40457 +4061
- Misses 15700 18589 +2889
- Partials 2677 2891 +214
```
| Flag | Coverage Δ | |
|---|---|---|
| unittests | `65.31% <45.12%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...e/pinot/broker/api/resources/PinotBrokerDebug.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYXBpL3Jlc291cmNlcy9QaW5vdEJyb2tlckRlYnVnLmphdmE=) | `0.00% <0.00%> (-79.32%)` | :arrow_down: |
| [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `71.42% <ø> (-28.58%)` | :arrow_down: |
| [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `33.96% <0.00%> (-32.71%)` | :arrow_down: |
| [...ker/routing/instanceselector/InstanceSelector.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9pbnN0YW5jZXNlbGVjdG9yL0luc3RhbmNlU2VsZWN0b3IuamF2YQ==) | `100.00% <ø> (ø)` | |
| [...ava/org/apache/pinot/client/AbstractResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Fic3RyYWN0UmVzdWx0U2V0LmphdmE=) | `66.66% <0.00%> (+9.52%)` | :arrow_up: |
| [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `35.55% <0.00%> (-13.29%)` | :arrow_down: |
| [...inot/client/JsonAsyncHttpPinotClientTransport.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0pzb25Bc3luY0h0dHBQaW5vdENsaWVudFRyYW5zcG9ydC5qYXZh) | `10.90% <0.00%> (-51.10%)` | :arrow_down: |
| [...not/common/assignment/InstancePartitionsUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnNVdGlscy5qYXZh) | `73.80% <ø> (+0.63%)` | :arrow_up: |
| [.../apache/pinot/common/exception/QueryException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZXhjZXB0aW9uL1F1ZXJ5RXhjZXB0aW9uLmphdmE=) | `90.27% <ø> (+5.55%)` | :arrow_up: |
| [...pinot/common/function/AggregationFunctionType.java](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vQWdncmVnYXRpb25GdW5jdGlvblR5cGUuamF2YQ==) | `98.27% <ø> (-1.73%)` | :arrow_down: |
| ... and [1130 more](https://codecov.io/gh/apache/incubator-pinot/pull/6320/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=footer). Last update [41a7722...0d90376](https://codecov.io/gh/apache/incubator-pinot/pull/6320?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg commented on pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
kishoreg commented on pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#issuecomment-740315985
Here is the excel sheet with an embedded chart to visualize your perf changes.
https://docs.google.com/spreadsheets/d/1EhK69Y5UZB-NFrgWBKCT-XXZeJ3oA8WoTDN_B2t37lA/edit?usp=sharing
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] richardstartin commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r536559080
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OnHeapBitmapInvertedIndexCreator.java
##########
@@ -19,68 +19,85 @@
package org.apache.pinot.core.segment.creator.impl.inv;
import com.google.common.base.Preconditions;
-import java.io.BufferedOutputStream;
-import java.io.DataOutputStream;
+
+
import java.io.File;
-import java.io.FileOutputStream;
import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+
import org.apache.commons.io.FileUtils;
import org.apache.pinot.core.segment.creator.DictionaryBasedInvertedIndexCreator;
import org.apache.pinot.core.segment.creator.impl.V1Constants;
+import org.apache.pinot.core.util.CleanerUtil;
+import org.roaringbitmap.RoaringBitmapWriter;
import org.roaringbitmap.buffer.MutableRoaringBitmap;
+import static java.nio.ByteOrder.LITTLE_ENDIAN;
+
/**
* Implementation of {@link DictionaryBasedInvertedIndexCreator} that uses on-heap memory.
*/
public final class OnHeapBitmapInvertedIndexCreator implements DictionaryBasedInvertedIndexCreator {
private final File _invertedIndexFile;
- private final MutableRoaringBitmap[] _bitmaps;
+ private final RoaringBitmapWriter<MutableRoaringBitmap>[] _bitmapWriters;
private int _nextDocId;
+ @SuppressWarnings("unchecked")
public OnHeapBitmapInvertedIndexCreator(File indexDir, String columnName, int cardinality) {
_invertedIndexFile = new File(indexDir, columnName + V1Constants.Indexes.BITMAP_INVERTED_INDEX_FILE_EXTENSION);
- _bitmaps = new MutableRoaringBitmap[cardinality];
+ _bitmapWriters = new RoaringBitmapWriter[cardinality];
for (int i = 0; i < cardinality; i++) {
- _bitmaps[i] = new MutableRoaringBitmap();
+ _bitmapWriters[i] = RoaringBitmapWriter.bufferWriter().get();
}
}
@Override
public void add(int dictId) {
- _bitmaps[dictId].add(_nextDocId++);
+ _bitmapWriters[dictId].add(_nextDocId++);
}
@Override
public void add(int[] dictIds, int length) {
for (int i = 0; i < length; i++) {
- _bitmaps[dictIds[i]].add(_nextDocId);
+ _bitmapWriters[dictIds[i]].add(_nextDocId);
}
_nextDocId++;
}
@Override
public void seal()
throws IOException {
- try (DataOutputStream out = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)))) {
+ // calculate file size
+ int size = (_bitmapWriters.length + 1) * Integer.BYTES;
+ for (RoaringBitmapWriter<MutableRoaringBitmap> writer : _bitmapWriters) {
+ size += writer.get().serializedSizeInBytes();
+ // Check for int overflow
+ Preconditions.checkState(size > 0, "Inverted index file: %s exceeds 2GB limit", _invertedIndexFile);
+ }
+ ByteBuffer buffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
+ buffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, size).order(LITTLE_ENDIAN);
Review comment:
I chose little endian because roaring bitmaps are always serialised in little endian byte order anyway, and they make up the bulk of the file. If you serialise to a `DataOutput` it will reverse every word, and if you serialise to a big endian `ByteBuffer`, it will just slice the buffer and change the order.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] richardstartin commented on pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
richardstartin commented on pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#issuecomment-739086582
The solution I came up with to make sure the same bytes are produced by both on-heap and off-heap creators was to apply the same changes to `OffHeapBitmapInvertedIndexCreator`, which will now also be run-optimised incrementally.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] richardstartin commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r537828747
##########
File path: pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkOffheapBitmapInvertedIndexCreator.java
##########
@@ -0,0 +1,105 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pinot.perf;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.core.segment.creator.impl.inv.OffHeapBitmapInvertedIndexCreator;
+import org.apache.pinot.spi.data.DimensionFieldSpec;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.openjdk.jmh.annotations.*;
+import org.openjdk.jmh.runner.Runner;
+import org.openjdk.jmh.runner.options.ChainedOptionsBuilder;
+import org.openjdk.jmh.runner.options.OptionsBuilder;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+
+@State(Scope.Benchmark)
+public class BenchmarkOffheapBitmapInvertedIndexCreator {
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #6320: More efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #6320:
URL: https://github.com/apache/incubator-pinot/pull/6320#discussion_r536521274
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
##########
@@ -181,33 +188,49 @@ public void seal()
}
// Create bitmaps from inverted index buffers and serialize them to file
- try (DataOutputStream offsetDataStream = new DataOutputStream(
- new BufferedOutputStream(new FileOutputStream(_invertedIndexFile)));
- FileOutputStream bitmapFileStream = new FileOutputStream(_invertedIndexFile);
- DataOutputStream bitmapDataStream = new DataOutputStream(new BufferedOutputStream(bitmapFileStream))) {
- int bitmapOffset = (_cardinality + 1) * Integer.BYTES;
- offsetDataStream.writeInt(bitmapOffset);
- bitmapFileStream.getChannel().position(bitmapOffset);
-
+ ByteBuffer offsetBuffer = null;
+ ByteBuffer bitmapBuffer = null;
+ try (FileChannel channel = new RandomAccessFile(_invertedIndexFile, "rw").getChannel()) {
Review comment:
Where exactly is the potential benefit coming from?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org