[jira] Commented: (HIVE-1721) use bloom filters to improve the performance of joins

["Siying Dong (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927537#action_12927537 ] 

Siying Dong commented on HIVE-1721:
-----------------------------------

Shouldn't bloom filter even more expensive than normal hash table?
Instead of calculating one hash, we'll have to calculate k of them. And calculating the hash, is the most expensive part for us.
It probably works if most of the key is not in small table and keys and values of small tables are saved in disks. I'm not sure it is our use scenario.



> use bloom filters to improve the performance of joins
> -----------------------------------------------------
>
>                 Key: HIVE-1721
>                 URL: https://issues.apache.org/jira/browse/HIVE-1721
>             Project: Hive
>          Issue Type: New Feature
>          Components: Query Processor
>            Reporter: Namit Jain
>            Assignee: Siying Dong
>
> In case of map-joins, it is likely that the big table will not find many matching rows from the small table.
> Currently, we perform a hash-map lookup for every row in the big table, which can be pretty expensive.
> It might be useful to try out a bloom-filter containing all the elements in the small table.
> Each element from the big table is first searched in the bloom filter, and only in case of a positive match,
> the small table hash table is explored.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.