You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Eugene Kirpichov (JIRA)" <ji...@apache.org> on 2017/03/28 23:20:42 UTC
[jira] [Commented] (BEAM-60) FileBasedSource/IOChannelFactory:
Custom glob expansion
[ https://issues.apache.org/jira/browse/BEAM-60?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946172#comment-15946172 ]
Eugene Kirpichov commented on BEAM-60:
--------------------------------------
BEAM-1822 probably justifies raising the priority of this.
Also, I unassigned from Pei because AFAIK he no longer works on Beam.
> FileBasedSource/IOChannelFactory: Custom glob expansion
> -------------------------------------------------------
>
> Key: BEAM-60
> URL: https://issues.apache.org/jira/browse/BEAM-60
> Project: Beam
> Issue Type: New Feature
> Components: sdk-java-core
> Reporter: Daniel Halperin
>
> Many cloud and distributed filesystems are eventually consistent, for instance Amazon s3 and Google Cloud Storage.
> To work around this, many systems that produce files such as Beam's FileBasedSinks, or Google BigQuery will provide methods to determine the number and set of files produced. E.g.,
> * Beam FileBasedSink uses -00000-of-NNNNN
> * BigQuery export jobs uses -000000 -000001 -000002 ... until an empty file is produced
> * Another system may produce a .filelist suffix that contains a list of all files.
> Users should be able to supply a glob to FileBasedSource but additionally supply a "glob expander" that can provide a custom implementation for file expansion. That way, e.g., Beam pipelines can be run back-to-back-to-back where each consumes the output of the previous, on an inconsistent filesystem, without data loss.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)