[jira] [Comment Edited] (FLINK-4037) Introduce ArchivedExecutionGraph without any user classes

["Stefan Richter (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-4037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331982#comment-15331982 ] 

Stefan Richter edited comment on FLINK-4037 at 6/15/16 4:11 PM:
----------------------------------------------------------------

I think introducing an {{ArchivedExecutionGraph}} to maintain relevant information from the {{ExecutionGraph}} of finished jobs is a little more involved. E.g. the web interface has to deal with objects from {{ArchivedExecutionGraph}} as well as {{ExecutionGraph}} to display finished and in-flight jobs.

To achieve this, we could for example:
* Make {{ExecutionGraph}} a subclass of {{ArchivedExecutionGraph}}. However this probably not the best idea as it violates the substitution principle.
* Introduce a common interface for both, {{ExecutionGraph}} and {{ArchivedExecutionGraph}}. This interface provides means for the web interface to extract information for display.

The method {{prepareForArchiving()}} could convert ExecutionGraphs into ArchivedExecutionGraphs. We should ensure that asynchronous request by the web interface are routed to a valid ExecutionGraph until the conversion is complete and the object that acts as model for the web interface is substituted.

Furthermore, we need to identify all references in ExecutionGraph that could hold objects from user-provided classes (e.g. accumulators, metrics, ...), stringify their information (see {{JobConfigHandler.handleRequest()}}, and release the references so that the user classloader can be garbage-collected. Corresponding parts of the mentioned stringification have to be pushed the ExecutionGraphs. The web interface needs to be changed to extract the information through the interface.    


was (Author: srichter):
I think introducing an {{ArchivedExecutionGraph}} to maintain relevant information from the {{ExecutionGraph}} of finished jobs is a little more involved. E.g. the web interface has to deal with objects from {{ArchivedExecutionGraph}} as well as {{ExecutionGraph}} to display finished and in-flight jobs.

To achieve this, we could for example:
* Make {{ExecutionGraph}} a subclass of {{ArchivedExecutionGraph}}. However this probably not the best idea as it violates the substitution principle.
* Introduce a common interface for both, {ExecutionGraph}} and {{ArchivedExecutionGraph}}. This interface provides means for the web interface to extract information for display.

The method {{prepareForArchiving()}} could convert ExecutionGraphs into ArchivedExecutionGraphs. We should ensure that asynchronous request by the web interface are routed to a valid ExecutionGraph until the conversion is complete and the object that acts as model for the web interface is substituted.

Furthermore, we need to identify all references in ExecutionGraph that could hold objects from user-provided classes (e.g. accumulators, metrics, ...), stringify their information (see {{JobConfigHandler.handleRequest()}}, and release the references so that the user classloader can be garbage-collected. Corresponding parts of the mentioned stringification have to be pushed the ExecutionGraphs. The web interface needs to be changed to extract the information through the interface.    

> Introduce ArchivedExecutionGraph without any user classes
> ---------------------------------------------------------
>
>                 Key: FLINK-4037
>                 URL: https://issues.apache.org/jira/browse/FLINK-4037
>             Project: Flink
>          Issue Type: Improvement
>          Components: Webfrontend
>            Reporter: Robert Metzger
>            Assignee: Stefan Richter
>
> As a follow up to FLINK-4011: In order to allow the JobManager to unload all classes from a finished job, we need to convert the ExecutionGraph (and some attached objects like the ExecutionConfig) into a stringified version, not containing any user classes.
> The web frontend can show strings only anyways.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)