You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@gobblin.apache.org by "Sudarshan Vasudevan (JIRA)" <ji...@apache.org> on 2018/04/12 04:41:00 UTC

[jira] [Created] (GOBBLIN-464) Enhance LoopingDatasetFinderSource to support global watermark and per-dataset watermark

Sudarshan Vasudevan created GOBBLIN-464:
-------------------------------------------

             Summary: Enhance LoopingDatasetFinderSource to support global watermark and per-dataset watermark
                 Key: GOBBLIN-464
                 URL: https://issues.apache.org/jira/browse/GOBBLIN-464
             Project: Apache Gobblin
          Issue Type: Improvement
          Components: gobblin-compliance
    Affects Versions: 0.13.0
            Reporter: Sudarshan Vasudevan
            Assignee: Sudarshan Vasudevan
             Fix For: 0.13.0


We need the ability to support a global watermark that spans all datasets as well as a per-dataset watermark in LoopingDatasetFinderSource. In particular we need to keep track of:
 # The last dataset processed in the previous run of a gobblin job (global watermark) so that the subsequent run can process the "next" dataset (based on some ordering between datasets), and
 # The last time a particular dataset is processed (per-dataset watermark).

The actual use case is to limit the time range of queries to an external database based on the last processed time of a dataset. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)