[jira] [Created] (GOBBLIN-1657) Update completion watermark on change_property in IcebergMetadataWriter

["Vikram Bohra (Jira)" <ji...@apache.org>]

Vikram Bohra created GOBBLIN-1657:
-------------------------------------

             Summary: Update completion watermark on change_property in IcebergMetadataWriter 
                 Key: GOBBLIN-1657
                 URL: https://issues.apache.org/jira/browse/GOBBLIN-1657
             Project: Apache Gobblin
          Issue Type: Task
            Reporter: Vikram Bohra


There are quiet topics where data can be generated quiet infrequently causing the completion watermark to lag. Also, a commit operation could miss completion watermark update if there is a lag in kafka audit.

To fix above issues:
 # Added logic to check audit count on change_property operation type for the next window of current watermark 
 # If counts match or an empty map is returned then consider audit count as complete and update the watermark.

Added a config in KafkaAuditVerifier to return complete if found an empty map (with no counts from audit system) 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)