[jira] [Commented] (FLINK-1085) Unnecessary failing of GroupReduceCombineDriver

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14594853#comment-14594853 ] 

ASF GitHub Bot commented on FLINK-1085:
---------------------------------------

GitHub user dabaitu opened a pull request:

    https://github.com/apache/flink/pull/854

    FLINK-1085: Unnecessary failing of GroupReduceCombineDriver

    I have a unit test failure and it seems it may have been there before my changes:
    
    Tests in error: 
      UtilsTest.testUberjarLocator:39 NullPointer
    
    It's looking for some uber jar and not finding it. Did I need to compile something else first?

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/dabaitu/flink master

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/854.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #854
    
----
commit e693ba2398c170b6d65a2365a4cc0fe091f3319b
Author: dabaitu <to...@gmail.com>
Date:   2015-06-20T22:35:48Z

    FLINK-1085: change exception to warning log and track oversized Record count

----


> Unnecessary failing of GroupReduceCombineDriver
> -----------------------------------------------
>
>                 Key: FLINK-1085
>                 URL: https://issues.apache.org/jira/browse/FLINK-1085
>             Project: Flink
>          Issue Type: Bug
>          Components: Local Runtime
>    Affects Versions: 0.6.1-incubating, 0.7.0-incubating
>            Reporter: Fabian Hueske
>              Labels: starter
>
> With a recent update (commit cbbcf7820885a8a9734ffeba637b0182a6637939) the GroupReduceCombineDriver was changed to not use an asynchronous partial sorter. Instead, the driver fills a sort buffer with records, sorts it, combines them, clears the buffer, and continues to fill it again.
> The GroupReduceCombineDriver fails if a record cannot be serialized into an empty sort buffer, i.e., if the record is too large for the buffer.
> Alternatively, we should emit a WARN message for the first record that is too large and just forward all records which do not fit into the empty sort buffer (maybe continue to count how many records were simply forwarded and give a second WARN message with this statistic).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)