[jira] [Created] (GOBBLIN-1778) Add house keeping thread in DagManager to periodically sync in memory state with mysql table

["Zihan Li (Jira)" <ji...@apache.org>]

Zihan Li created GOBBLIN-1778:
---------------------------------

             Summary: Add house keeping thread in DagManager to periodically sync in memory state with mysql table 
                 Key: GOBBLIN-1778
                 URL: https://issues.apache.org/jira/browse/GOBBLIN-1778
             Project: Apache Gobblin
          Issue Type: Improvement
            Reporter: Zihan Li


Now dag managers have the assumption that it is the only process that can update mysql table and the in-memory state is always in sync with mysql. But we do notice that during the leader transforms period, it's possible that two dag manager can run concurrently and update the mysql db at the same time. 

To address that, we need either add a lock to make sure only one dag manager is working at one time, or we need to have a housekeeping thread to periodically sync the in-memory state with the mysql table. After discussion, we choose to go with later approach



--
This message was sent by Atlassian Jira
(v8.20.10#820010)