You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@gobblin.apache.org by "Zhixiong Chen (Jira)" <ji...@apache.org> on 2020/08/13 20:51:01 UTC

[jira] [Closed] (GOBBLIN-1012) Implement CompactionWithWatermarkSuite

     [ https://issues.apache.org/jira/browse/GOBBLIN-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zhixiong Chen closed GOBBLIN-1012.
----------------------------------
    Resolution: Fixed

> Implement CompactionWithWatermarkSuite
> --------------------------------------
>
>                 Key: GOBBLIN-1012
>                 URL: https://issues.apache.org/jira/browse/GOBBLIN-1012
>             Project: Apache Gobblin
>          Issue Type: Task
>            Reporter: Zhixiong Chen
>            Assignee: Zhixiong Chen
>            Priority: Major
>          Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> - A `compactionWatermark` is a timestamp indicating the data we've seen up to that time is compacted. If we explicitly checked that we've seen all the data up to that time, the timestamp would be published also as `completionAndCompactionWatermark`.
>  - `CompactionWithWatermarkSuite` reports compaction watermarks as part of the compaction pipeline and publishes watermarks as hive table properties



--
This message was sent by Atlassian Jira
(v8.3.4#803005)