You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "Fabian Hueske (JIRA)" <ji...@apache.org> on 2015/09/15 14:55:47 UTC

[jira] [Closed] (FLINK-2655) Minimize intermediate merging of spilled buffers

     [ https://issues.apache.org/jira/browse/FLINK-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Fabian Hueske closed FLINK-2655.
--------------------------------
       Resolution: Fixed
    Fix Version/s: 0.10

Fixed with a75dd6270fd47720ae1c2ce9464ff6b8f7b43d39

> Minimize intermediate merging of spilled buffers
> ------------------------------------------------
>
>                 Key: FLINK-2655
>                 URL: https://issues.apache.org/jira/browse/FLINK-2655
>             Project: Flink
>          Issue Type: Improvement
>          Components: Distributed Runtime
>    Affects Versions: master
>            Reporter: Greg Hogan
>             Fix For: 0.10
>
>
> If the number of spilled buffers exceeds taskmanager.runtime.max-fan then the number of files must reduced with an intermediate merge by reading, merging, and spilling into a single, larger file.
> The current implementation performs an intermediate merge on all files. An optimal implementation minimizes the amount of merged data by performing partial merges first.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)