[jira] [Work logged] (BEAM-5708) Support caching of SDKHarness environments in flink

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-5708?focusedWorklogId=154217&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-154217 ]

ASF GitHub Bot logged work on BEAM-5708:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 15/Oct/18 08:55
            Start Date: 15/Oct/18 08:55
    Worklog Time Spent: 10m 
      Work Description: mxm commented on a change in pull request #6638: [BEAM-5708] Cache environment in portable flink runner
URL: https://github.com/apache/beam/pull/6638#discussion_r225085233
 
 

 ##########
 File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/ReferenceCountingFlinkExecutableStageContextFactory.java
 ##########
 @@ -104,9 +109,30 @@ private void scheduleRelease(JobInfo jobInfo) {
     WrappedContext wrapper = getCache().get(jobInfo.jobId());
     Preconditions.checkState(
         wrapper != null, "Releasing context for unknown job: " + jobInfo.jobId());
-    // Do not release this asynchronously, as the releasing could fail due to the classloader not being
-    // available anymore after the tasks have been removed from the execution engine.
-    release(wrapper);
+
+    PipelineOptions pipelineOptions =
+        PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions());
+    int environmentCacheTTLMillis =
+        pipelineOptions.as(PortablePipelineOptions.class).getEnvironmentCacheMillis();
+    if (environmentCacheTTLMillis > 0) {
+      if (this.getClass().getClassLoader() != ExecutionEnvironment.class.getClassLoader()) {
 
 Review comment:
   Are we sure that always works? What if we ship ExecutionEnvironment with the child classloader, e.g. in a fat jar? IMHO safer here would be to check for `FlinkUserCodeClassLoader` and see if it is still open.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 154217)
    Time Spent: 2h 20m  (was: 2h 10m)

> Support caching of SDKHarness environments in flink
> ---------------------------------------------------
>
>                 Key: BEAM-5708
>                 URL: https://issues.apache.org/jira/browse/BEAM-5708
>             Project: Beam
>          Issue Type: Improvement
>          Components: runner-flink
>            Reporter: Ankur Goenka
>            Assignee: Ankur Goenka
>            Priority: Major
>          Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Cache and reuse environment to improve performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)