[jira] [Commented] (BEAM-57) FileBasedSink should write intermediate output to a different folder

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15634012#comment-15634012 ] 

ASF GitHub Bot commented on BEAM-57:
------------------------------------

Github user asfgit closed the pull request at:

    https://github.com/apache/incubator-beam/pull/1050


> FileBasedSink should write intermediate output to a different folder
> --------------------------------------------------------------------
>
>                 Key: BEAM-57
>                 URL: https://issues.apache.org/jira/browse/BEAM-57
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-java-core
>            Reporter: Daniel Halperin
>            Assignee: Eugene Kirpichov
>            Priority: Minor
>
> Currently, FileBasedSink writes intermediate files and final output files to the same folder.  Since deletion of intermediate files is prone to eventual consistency issues with GCS,S3,other systems and users may use globs in subsequent pipelines, intermediate files should be written to a different folder.
> This may be a subdirectory or a pipeline-defined temp directory (with probably some additional subdirectories for this source).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)