You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/03/20 20:01:49 UTC
[GitHub] [druid] clintropolis opened a new pull request #9548: roaring
bitmaps by default
clintropolis opened a new pull request #9548: roaring bitmaps by default
URL: https://github.com/apache/druid/pull/9548
### Description
I think it is finally time to switch to using Roaring bitmaps instead of using CONCISE by default. Using Druid with Roaring is rather well tested by now, and in most cases I think it is going to provide a better out of the box experience, where the speed is generally worth the potential for larger segment sizes that come with high cardinalities. There will still exist cases of datasets with ultra high cardinality columns where CONCISE might produce smaller segments due to the overhead of the Roaring format, but it makes sense to me for the operator to opt into the decision of wanting the smallest possible segments at the potential cost of speed, rather than that being the default.
Related: http://db.ucsd.edu/wp-content/uploads/2017/03/sidm338-wangA.pdf
<hr>
This PR has:
- [x] been self-reviewed.
- [x] added documentation for new or modified features or behaviors.
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths.
- [ ] added integration tests.
- [x] been tested in a test Druid cluster.
<!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist above are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
<hr>
##### Key changed/added classes in this PR
* `DefaultBitmapSerdeFactory`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] himanshug commented on issue #9548: roaring bitmaps by
default
Posted by GitBox <gi...@apache.org>.
himanshug commented on issue #9548: roaring bitmaps by default
URL: https://github.com/apache/druid/pull/9548#issuecomment-602948616
> I added a 'compression' section to the segment .....
LGTM, thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on issue #9548: roaring bitmaps by
default
Posted by GitBox <gi...@apache.org>.
clintropolis commented on issue #9548: roaring bitmaps by default
URL: https://github.com/apache/druid/pull/9548#issuecomment-602899977
> sounds good, but can you document a version of PR description in the doc file where configuration is (or maybe create a separate bitmaps.md file and link to that), so that users can make an informed decision. I know the paper is linked but it would be useful if summary of pros/cons is listed in Druid docs itself.
I added a 'compression' section to the segment documentation page, that attempts to dissuade people from changing the defaults unless they verify that the settings are in fact better for their use case (which is why I left this off at first, because people probably shouldn't be changing these unless they know what they are doing imo).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson merged pull request #9548: roaring bitmaps by
default
Posted by GitBox <gi...@apache.org>.
jihoonson merged pull request #9548: roaring bitmaps by default
URL: https://github.com/apache/druid/pull/9548
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] himanshug commented on issue #9548: roaring bitmaps by
default
Posted by GitBox <gi...@apache.org>.
himanshug commented on issue #9548: roaring bitmaps by default
URL: https://github.com/apache/druid/pull/9548#issuecomment-601959660
sounds good, but can you document a version of PR description in the doc file where configuration is (or maybe create a separate bitmaps.md file and link to that), so that users can make an informed decision. I know the paper is linked but it would be useful if summary of pros/cons is listed in Druid docs itself.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org