You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@beam.apache.org by "Daniel Halperin (JIRA)" <ji...@apache.org> on 2016/10/20 18:26:58 UTC

[jira] [Updated] (BEAM-57) FileBasedSink should write intermediate output to a different folder

     [ https://issues.apache.org/jira/browse/BEAM-57?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Halperin updated BEAM-57:
--------------------------------
    Assignee: Eugene Kirpichov

> FileBasedSink should write intermediate output to a different folder
> --------------------------------------------------------------------
>
>                 Key: BEAM-57
>                 URL: https://issues.apache.org/jira/browse/BEAM-57
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-java-core
>            Reporter: Daniel Halperin
>            Assignee: Eugene Kirpichov
>            Priority: Minor
>
> Currently, FileBasedSink writes intermediate files and final output files to the same folder.  Since deletion of intermediate files is prone to eventual consistency issues with GCS,S3,other systems and users may use globs in subsequent pipelines, intermediate files should be written to a different folder.
> This may be a subdirectory or a pipeline-defined temp directory (with probably some additional subdirectories for this source).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)