You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "jitendrakr88 (via GitHub)" <gi...@apache.org> on 2025/05/19 18:11:20 UTC
[I] DISTINCTCOUNTBITMAP Accuracy Improvement [pinot]
jitendrakr88 opened a new issue, #15840:
URL: https://github.com/apache/pinot/issues/15840
The existing `DISTINCTCOUNTBITMAP` function stores the `hashcode()` of values in a 32-bit RoaringBitmap. This works in general, but in our use case, it's becoming a limitation — accuracy is lost due to the high chance of collisions, since Java hashcodes are only 32 bits.
We want to support a case where the client stores 64-bit identifiers in a bitmap, serialises it as a byte array into a Pinot table, and later computes the cardinality from that 64-bit bitmap during query time.
To support this use case, one option is to introduce a new aggregation function that avoids 32-bit hash codes and instead uses a 64-bit bitmap container to store a 64-bit hash of them.
We came across two 64-bit bitmap implementations:
- `org.roaringbitmap.longlong.Roaring64Bitmap`
- `org.roaringbitmap.longlong.Roaring64NavigableMap`
In a small poc, Roaring64NavigableMap showed slightly better performance and marginally lower memory usage when ingesting 1 billion keys:
```
--- Benchmark: Roaring64Bitmap ---
Time taken in ingesting 1Billion keys: 39953 ms
Memory used: 123.54 MB
--- Benchmark: Roaring64NavigableMap ---
Time taken in ingesting 1Billion keys: 6861 ms
Memory used: 120.13 MB
```
Potential impact:
- Will increase the memory usage on servers
- Latency increases during serving (maybe?)
Would appreciate any feedback or input on this direction, and whether something similar has already been explored?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] DISTINCTCOUNTBITMAP Accuracy Improvement [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15840:
URL: https://github.com/apache/pinot/issues/15840#issuecomment-4331478043
This issue has been closed because it has been inactive for 180 days. If you believe this issue is still relevant, please feel free to reopen it or create a new issue. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] DISTINCTCOUNTBITMAP Accuracy Improvement [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15840:
URL: https://github.com/apache/pinot/issues/15840#issuecomment-4149103161
This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] DISTINCTCOUNTBITMAP Accuracy Improvement [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #15840:
URL: https://github.com/apache/pinot/issues/15840#issuecomment-2891991642
You might want to try out `DISTINCT_COUNT_OFF_HEAP` introduced in #15469, which allows different hashing bits. Storing 64 bit hashes in a bitmap might not be efficient because it is always very sparse.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] DISTINCTCOUNTBITMAP Accuracy Improvement [pinot]
Posted by "rohityadav1993 (via GitHub)" <gi...@apache.org>.
rohityadav1993 commented on issue #15840:
URL: https://github.com/apache/pinot/issues/15840#issuecomment-2894294381
@jitendrakr88, do you also need `aggregateGroupByMV` for precomputed bitmaps which is currently not supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] DISTINCTCOUNTBITMAP Accuracy Improvement [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #15840: DISTINCTCOUNTBITMAP Accuracy Improvement
URL: https://github.com/apache/pinot/issues/15840
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org