[GitHub] b-slim commented on issue #6584: fix druid-bloom-filter thread-safety

[GitBox <gi...@apache.org>]

b-slim commented on issue #6584: fix druid-bloom-filter thread-safety
URL: https://github.com/apache/incubator-druid/pull/6584#issuecomment-436787992
 
 
   The original intent of PR #6222 , is to build the BloomK sketch in Hive and handed it over to Druid (via the wire as a filter) to filter down the scan, therefore, for this to work we need to make sure that both Hive and Druid are **using the same algorithm for building sketch and probing it**.
   This implies that the best way to fix this is to push the fix to Hive Storage API  and drive a storage API release (FYI storage API release is independent from Hive release life cycle hence goes faster). 
   In the meantime to not block 0.13  I think we can go with this option where the code is copied over to Druid it self waiting on the release with the bug fix but for the longer term it is better to have it in one place to avoid issues.
   @clintropolis if your goal is to build your own bloom filter sketch from scratch for performance reasons or any other goal, then it is better to introduce a new kind of filter.  The other way to do it is to introduce a version blob within the sketch it self to pick the correct serde/probing implementation but this will need way more work...

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org