You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "Sudarshan Vasudevan (JIRA)" <ji...@apache.org> on 2019/02/20 04:35:00 UTC
[jira] [Updated] (GOBBLIN-687) Pass TopologySpec map to DagManager
to allow reuse of SpecExecutors during DAG deserialization
[ https://issues.apache.org/jira/browse/GOBBLIN-687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sudarshan Vasudevan updated GOBBLIN-687:
----------------------------------------
Summary: Pass TopologySpec map to DagManager to allow reuse of SpecExecutors during DAG deserialization (was: Pass TopologySpec map to DagManager )
> Pass TopologySpec map to DagManager to allow reuse of SpecExecutors during DAG deserialization
> ----------------------------------------------------------------------------------------------
>
> Key: GOBBLIN-687
> URL: https://issues.apache.org/jira/browse/GOBBLIN-687
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-service
> Affects Versions: 0.15.0
> Reporter: Sudarshan Vasudevan
> Assignee: Abhishek Tiwari
> Priority: Major
> Fix For: 0.15.0
>
>
> DagManager maintains state of all currently executing DAGs, by serializing each DAG on compilation and persisting it to a durable store. The serialized DAG includes Job config as well as the SpecExecutor config for each job in the DAG. This is done to correctly resume execution of DAGs in case of service restarts or leadership change.
> Currently, on service restart/leadership change, the new master de-serializes SpecExecutor config and creates a SpecExecutor instance for each job in the DAG. If the number of DAGs is large, this can result in many connections to the underlying executor instance. The proposed fix allows the DagManager to re-use the SpecExecutor instances created by the TopologySpecFactory when it deserializes a DAG.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)