[jira] [Commented] (BEAM-6460) Jackson Cache may hold on to Classloader after pipeline restart

["Maximilian Michels (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-6460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16748279#comment-16748279 ] 

Maximilian Michels commented on BEAM-6460:
------------------------------------------

Update on this: Not a Beam issue. Flink leaks classloaders when libraries are loaded through the Flink root classloader which have static caches. For example, when Flink's "yarn-cluster" mode is used, the user jar is part of the Flink root classloader.

The subtlety of this issue warrants that we clear the Jackson cache to avoid other users running into this. Ultimately, this should be fixed upstream by Flink.

> Jackson Cache may hold on to Classloader after pipeline restart
> ---------------------------------------------------------------
>
>                 Key: BEAM-6460
>                 URL: https://issues.apache.org/jira/browse/BEAM-6460
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-flink
>    Affects Versions: 2.7.0
>            Reporter: Maximilian Michels
>            Assignee: Maximilian Michels
>            Priority: Blocker
>             Fix For: 2.10.0
>
>          Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> It looks like Jackson has an internal cache which may continue to hold the Flink application classloader through its {{TypeFactory}} class. When the pipeline is restarted due to a failure, a new classloader is created which can result in too many classes being loaded.
> Reported on the user mailing list: https://lists.apache.org/thread.html/e201891684ef3dcffce48d20d1f9be0e19fc2294334362cc7092c0ff@%3Cuser.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)