You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@beam.apache.org by "Davor Bonaci (JIRA)" <ji...@apache.org> on 2017/08/24 21:01:00 UTC

[jira] [Assigned] (BEAM-2302) WriteFiles with runner-determined sharding and large numbers of windows causes OOM errors

     [ https://issues.apache.org/jira/browse/BEAM-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Davor Bonaci reassigned BEAM-2302:
----------------------------------

    Assignee: Reuven Lax  (was: Davor Bonaci)

> WriteFiles with runner-determined sharding and large numbers of windows causes OOM errors
> -----------------------------------------------------------------------------------------
>
>                 Key: BEAM-2302
>                 URL: https://issues.apache.org/jira/browse/BEAM-2302
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-core
>            Reporter: Reuven Lax
>            Assignee: Reuven Lax
>
> This is because the WriteWindowedBundles transform will create many file writers, and the sheer number of file buffers (which defaults to 64mb per writer) uses up all memory. The fix is the same as was done in BigQueryIO - if too many writers are opened, spill into a shuffle, and write the files after the shuffle



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)