You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@gobblin.apache.org by "Hanghang Liu (Jira)" <ji...@apache.org> on 2021/04/12 21:17:00 UTC

[jira] [Closed] (GOBBLIN-1413) Emit GMCE as long as watermark moved

     [ https://issues.apache.org/jira/browse/GOBBLIN-1413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hanghang Liu closed GOBBLIN-1413.
---------------------------------

> Emit GMCE as long as watermark moved
> ------------------------------------
>
>                 Key: GOBBLIN-1413
>                 URL: https://issues.apache.org/jira/browse/GOBBLIN-1413
>             Project: Apache Gobblin
>          Issue Type: Bug
>            Reporter: Hanghang Liu
>            Priority: Critical
>          Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Emit GMCE(Gobblin Metadata Change Event) as long as watermark moved on streaming pipeline. 
> Currently the GMCE won't be triggered within streaming pipeline if no new file being generated. This causes problem if watermarks moved, while no file being generated(for example, data been filtered out by quality checker), GMCE will be missed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)