Netty Client Thread - Classloader leak

[Sudharsan R <su...@gmail.com>]

Hello,
I'm running a flink 1.11.1 cluster. When I submit a job, it spawns a thread
named "Flink Netty Client (0) Thread 0 Thread". It seems to be executing
"org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap"
This thread associates as its context classloader a ChildFirstClassLoader
(the one that is also the classloader for my app jar).
Eventually, I cancel the job and this particular ChildFirstClassLoader
remains with the GC root for this being the netty thread.

If I submit another job (same app jar as before) and cancel it, the
ChildFirstClassLoader of this submission eventually gets GC'ed. However,
the original one remains and seems to be leaked.

I don't see others complaining about this. And I don't do anything with
this netty client directly! Any thoughts on what I can do?

Thanks
Sudharsan

Re: Netty Client Thread - Classloader leak

[yu'an huang <h....@gmail.com>]

I suddenly realised that I replied directly to Sudharsan. FYI if anyone wants to check this email too:



> On 2 Mar 2022, at 11:36 PM, yu'an huang <h....@gmail.com> wrote:
> 
> Hi Sudharsan,
> 
> I think you are right. I just tried your scenario. I set a standalone session cluster with 2 task managers (with one slot only each) in Flink 1.11. Then I submitted a job (parallelism 2), cancelled this job, submitted this job second times, and then canceled the job again. After that, I used the VisualVM to analyse the dumped memory of the task managers. 
> 
> What I observed is that there are two instances of ChildFirstClassLoader (user class loader). The first one (belongs to the first job) do have a GC root which is “Flink Netty Client (0) Thread 0” so it can not be GCed. The second one (belongs to the second job) don’t have GC root so it will be GCed in the future.
> 
> So it is interesting why the user class loader becomes the ContextClassLoader of Flink Netty Client (0) Thread 0. After checking the code, in my understanding:
> 1. When creating a thread, the context ClassLoader is provided by the creator of the thread. If not set, the default is the ClassLoader context of the parent Thread. 
> 2. A task is running in the “executingThread". Before starting the running loop, the executingThread will set the userClassLoader to its ContextClassLoader. Then the task call the invokable (Contains UDF etc).
> 3. Before the invokable enters the MailboxLoop, it needs to create the input channel to receive data from upstream. If upstream is not local, it needs to set up connection to remote thus a Flink Netty Client Thread may be created which will have the executingThread’s ContextClassLoader as its ContextClassLoader.
> 4. Even though this job is canceled, the  Flink Netty Client Thread still exists because future job will need this thread. That is why the user class loader can not be GCed until Flink Netty Client destroyed.
> 
> I feel that it might be very difficult to ensure no leak happened. If this doesn't influence you job, it is probably okay to skip now. This issue may be fixed or fix already (haven’t confirmed) in future Flink.
> 
> 
> Hopes this answer is not too late for you.
> 
> Best,
> Yuan
> 
> 
> 
> 
> 
> 
> 
>> On 1 Mar 2022, at 11:52 PM, Sudharsan R <sud.rang@gmail.com <ma...@gmail.com>> wrote:
>> 
>> Hello,
>> I'm running a flink 1.11.1 cluster. When I submit a job, it spawns a thread named "Flink Netty Client (0) Thread 0 Thread". It seems to be executing "org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap"
>> This thread associates as its context classloader a ChildFirstClassLoader (the one that is also the classloader for my app jar). 
>> Eventually, I cancel the job and this particular ChildFirstClassLoader remains with the GC root for this being the netty thread.
>> 
>> If I submit another job (same app jar as before) and cancel it, the ChildFirstClassLoader of this submission eventually gets GC'ed. However, the original one remains and seems to be leaked.
>> 
>> I don't see others complaining about this. And I don't do anything with this netty client directly! Any thoughts on what I can do?
>> 
>> Thanks
>> Sudharsan
>> 
>> 
>