[GitHub] [gobblin] rzhang10 commented on a diff in pull request #3659: Add completion watermark to State

["rzhang10 (via GitHub)" <gi...@apache.org>]

rzhang10 commented on code in PR #3659:
URL: https://github.com/apache/gobblin/pull/3659#discussion_r1136322807


##########
gobblin-api/src/main/java/org/apache/gobblin/configuration/State.java:
##########
@@ -62,16 +63,21 @@ public class State implements WritableShim {
   @Getter
   private Properties specProperties;
 
+  // This in-mem state will be used to share partition completion information across different MetadataWriter impls
+  public Map<String, Set<String>> tableToCompletedPartitions;

Review Comment:
   My intention is to make this map an in-memory transient object. I feel we shouldn't expose this as a config key that users can access and set externally. WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org