[jira] [Assigned] (BEAM-66) Unbounded source for directory/file scanning

["Daniel Halperin (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-66?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Halperin reassigned BEAM-66:
-----------------------------------

    Assignee: Daniel Halperin  (was: Davor Bonaci)

> Unbounded source for directory/file scanning
> --------------------------------------------
>
>                 Key: BEAM-66
>                 URL: https://issues.apache.org/jira/browse/BEAM-66
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-java-core
>            Reporter: Daniel Halperin
>            Assignee: Daniel Halperin
>            Priority: Minor
>
> (Blocked on SplittableDoFn, https://issues.apache.org/jira/browse/BEAM-65.)
> This is a streaming glob watcher, for instance a directory tailer -- poll a directory read the data from new files as they show up. To get this right in a streaming pipeline requires careful management of timestamps and watermarks -- e.g., a filename or other metadata should reflect a timestamp earlier than any records in it, so that we are able to provide accurate lower bounds on output timestamp.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)