You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@gobblin.apache.org by "Sudarshan Vasudevan (JIRA)" <ji...@apache.org> on 2019/02/20 04:33:00 UTC

[jira] [Created] (GOBBLIN-687) Pass TopologySpec map to DagManager

Sudarshan Vasudevan created GOBBLIN-687:
-------------------------------------------

             Summary: Pass TopologySpec map to DagManager 
                 Key: GOBBLIN-687
                 URL: https://issues.apache.org/jira/browse/GOBBLIN-687
             Project: Apache Gobblin
          Issue Type: Improvement
          Components: gobblin-service
    Affects Versions: 0.15.0
            Reporter: Sudarshan Vasudevan
            Assignee: Abhishek Tiwari
             Fix For: 0.15.0


DagManager maintains state of all currently executing DAGs, by serializing each DAG on compilation and persisting it to a durable store. The serialized DAG includes Job config as well as the SpecExecutor config for each job in the DAG. This is done to correctly resume execution of DAGs in case of service restarts or leadership change. 

Currently, on service restart/leadership change, the new master de-serializes SpecExecutor config and creates a SpecExecutor instance for each job in the DAG. If the number of DAGs is large, this can result in many connections to the underlying executor instance. The proposed fix allows the DagManager to re-use the SpecExecutor instances created by the TopologySpecFactory when it deserializes a DAG.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)