You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 15:44:40 UTC
[GitHub] [beam] damccorm opened a new issue, #20201: Modify behavior of Dynamic Destinations
damccorm opened a new issue, #20201:
URL: https://github.com/apache/beam/issues/20201
The writeDynamic() method, implementing Dynamic Destinations writes files per destination per window per pane.
This leads to an increase in the number of files generated.
The request is as follows:
A way to make it possible for the user to modify the behavior of Dynamic Destinations to control the number of output files being produced.
a.) We can consider adding user-configurable parameters like writers per bundle, increasing number of records processed per bundle
and/or
b.) Introduce a method implementing Dynamic Destinations but more dependent on the data passing through the pipeline, instead of windows/panes.
So instead of splitting every output file into roughly the number of destinations being written to, we let the user configure how output files should be divided across destinations.
Links:
[1] [https://beam.apache.org/releases/javadoc/2.19.0/index.html?org/apache/beam/sdk/io/FileIO.html](https://beam.apache.org/releases/javadoc/2.19.0/index.html?org/apache/beam/sdk/io/FileIO.html)
[2] [https://github.com/apache/beam/blob/da9e17288e8473925674a4691d9e86252e67d7d7/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java](https://github.com/apache/beam/blob/da9e17288e8473925674a4691d9e86252e67d7d7/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java)
Imported from Jira [BEAM-10068](https://issues.apache.org/jira/browse/BEAM-10068). Original Jira may contain additional context.
Reported by: mborkar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org