[jira] [Closed] (BEAM-9660) StreamingDataflowWorker has confusing exception on commits over 2GB

["Sam Whittle (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-9660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sam Whittle closed BEAM-9660.
-----------------------------
    Fix Version/s: 2.21.0
       Resolution: Fixed

> StreamingDataflowWorker has confusing exception on commits over 2GB
> -------------------------------------------------------------------
>
>                 Key: BEAM-9660
>                 URL: https://issues.apache.org/jira/browse/BEAM-9660
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-dataflow
>    Affects Versions: 2.18.0, 2.19.0
>            Reporter: Sam Whittle
>            Assignee: Sam Whittle
>            Priority: Minor
>             Fix For: 2.21.0
>
>          Time Spent: 1h
>  Remaining Estimate: 0h
>
> Commits over 2GB have a negative serialized commit size.
> When not using streaming engine the max commit limit is 2GB.
> https://github.com/apache/beam/blob/v2.19.0/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java#L450
> There appears to be a logging regression introduced by
> https://github.com/apache/beam/pull/10013
> With the new code, if the serialization overflows the estimated bytes is set to Integer.MAX which equals the commit limit for appliance.
> Then the comparison here:
> https://github.com/apache/beam/blob/v2.19.0/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java#L1371
> which uses > does not trigger and the large commit is just passed on to the commit queue, triggering the exception seen in #3 [2] when the weigher uses the negative serialized size for the semaphore acquire call. 
> So previously where we would have thrown a KeyCommitTooLargeException we are throwing the IllegalArgumentException.
> From that exception description: https://github.com/apache/beam/blob/v2.19.0/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java#L236
>           ". This may be caused by grouping a very "
>               + "large amount of data in a single window without using Combine,"
>               + " or by producing a large amount of data from a single input element."
> The overflow could be remembered explicitly instead of just comparing with max.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)