[jira] [Commented] (FLINK-2240) Use BloomFilter to minimize probe side records which are spilled to disk in Hybrid-Hash-Join

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14660322#comment-14660322 ] 

ASF GitHub Bot commented on FLINK-2240:
---------------------------------------

Github user StephanEwen commented on the pull request:

    https://github.com/apache/flink/pull/888#issuecomment-128440090
  
    Manually merged in 61dcae391cb3b45ba3aff47d4d9163889d2958a4
    
    I added a commit on top to pass the flag to enable/disable bloom filters through the runtime configuration. That is the basis for later allowing it to enable it on a per-job basis. Also, we want to get rid of the `GlobalConfiguration` singleton pattern.


> Use BloomFilter to minimize probe side records which are spilled to disk in Hybrid-Hash-Join
> --------------------------------------------------------------------------------------------
>
>                 Key: FLINK-2240
>                 URL: https://issues.apache.org/jira/browse/FLINK-2240
>             Project: Flink
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Chengxiang Li
>            Assignee: Chengxiang Li
>            Priority: Minor
>
> In Hybrid-Hash-Join, while small table does not fit into memory, part of the small table data would be spilled to disk, and the counterpart partition of big table data would be spilled to disk in probe phase as well. If we build a BloomFilter while spill small table to disk during build phase, and use it to filter the big table records which tend to be spilled to disk, this may greatly  reduce the spilled big table file size, and saved the disk IO cost for writing and further reading.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)