You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@beam.apache.org by "Kyle Winkelman (JIRA)" <ji...@apache.org> on 2019/04/04 15:10:00 UTC

[jira] [Commented] (BEAM-6713) FileIO and TextIO unable to alter WriteFiles maxNumWritersPerBundle

    [ https://issues.apache.org/jira/browse/BEAM-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16809952#comment-16809952 ] 

Kyle Winkelman commented on BEAM-6713:
--------------------------------------

At this time we don't want to expose an additional knob. Look at {BEAM-6735|https://issues.apache.org/jira/browse/BEAM-6735] as a possible alternative to remove the limit imposed by DEFAULT_MAX_NUM_WRITERS_PER_BUNDLE.

> FileIO and TextIO unable to alter WriteFiles maxNumWritersPerBundle
> -------------------------------------------------------------------
>
>                 Key: BEAM-6713
>                 URL: https://issues.apache.org/jira/browse/BEAM-6713
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-java-core
>            Reporter: Kyle Winkelman
>            Priority: Major
>          Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> When attempting to run a batch workflow with a FileIO.write() I was getting job failures due to WriteFiles.DEFAULT_MAX_NUM_WRITERS_PER_BUNDLE causing a significant amount of data to be shuffled. My issues would be solved by increasing this and luckily WriteFiles already has withMaxNumWritersPerBundle but unfortunately FileIO and TextIO do not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)