You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Flavio Pompermaier <po...@okkam.it> on 2020/11/16 13:27:00 UTC
Random Task executor shutdown (java.lang.OutOfMemoryError: Metaspace)
Hello everybody,
I was writing this email when a similar thread on this mailing list
appeared..
The difference is that the other problem seems to be related with Flink
1.10 on YARN and does not output anything helpful in debugging the cause of
the problem.
Indeed, in my use case I use Flink 1.11.0 and Flink on a standalone session
cluster (the job is submitted to the cluster using the CLI client).
The problem arises when I submit the same job for about 20 times (this
number unfortunately is not deterministic and can change a little bit). The
error reported by the Task Executor is related to the ever growing
Metaspace..the error seems to be pretty detailed [1].
I found the same issue in some previous threads on this mailing list and
I've tried to figure it out the cause of the problem. The issue is that
looking at the objects allocated I don't really get an idea of the source
of the problem because the type of objects that are consuming the memory
are of general purpose (i.e. Bytes, Integers and Strings)...these are my
"top" memory consumers if looking at the output of jmap -histo <PID>:
At run 0:
num #instances #bytes class name (module)
-------------------------------------------------------
1: 46238 13224056 [B (java.base@11.0.9.1)
2: 3736 6536672 [I (java.base@11.0.9.1)
3: 38081 913944 java.lang.String (java.base@11.0.9.1)
4: 26 852384 [Lakka.dispatch.forkjoin.ForkJoinTask;
5: 7146 844984 java.lang.Class (java.base@11.0.9.1)
At run 1:
1: 77.608 25.317.496 [B (java.base@11.0.9.1)
2: 7.004 9.088.360 [I (java.base@11.0.9.1)
3: 15.814 1.887.256 java.lang.Class (java.base@11.0.9.1)
4: 67.381 1.617.144 java.lang.String (java.base@11.0.9.1)
5: 3.906 1.422.960 [Ljava.util.HashMap$Node; (
java.base@11.0.9.1)
At run 6:
1: 81.408 25.375.400 [B (java.base@11.0.9.1)
2: 12.479 7.249.392 [I (java.base@11.0.9.1)
3: 29.090 3.496.168 java.lang.Class (java.base@11.0.9.1)
4: 4.347 2.813.416 [Ljava.util.HashMap$Node; (
java.base@11.0.9.1)
5: 71.584 1.718.016 java.lang.String (java.base@11.0.9.1)
At run 8:
1: 985.979 127.193.256 [B (java.base@11.0.9.1)
2: 35.400 13.702.112 [I (java.base@11.0.9.1)
3: 260.387 6.249.288 java.lang.String (java.base@11.0.9.1)
4: 148.836 5.953.440 java.util.HashMap$KeyIterator (
java.base@11.0.9.1)
5: 17.641 5.222.344 [Ljava.util.HashMap$Node; (
java.base@11.0.9.1)
Thanks in advance for any help,
Flavio
[1]
--------------------------------------------------------------------------------------------------
java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error
has occurred. This can mean two things: either the job requires a larger
size of JVM metaspace to load classes or there is a class loading leak. In
the first case 'taskmanager.memory.jvm-metaspace.size' configuration option
should be increased. If the error persists (usually in cluster after
several job (re-)submissions) then there is probably a class loading leak
in user code or some of its dependencies which has to be investigated and
fixed. The task executor has to be shutdown...
at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
at java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
~[?:?]
at java.net.URLClassLoader.defineClass(URLClassLoader.java:550)
~[?:?]
at java.net.URLClassLoader$1.run(URLClassLoader.java:458) ~[?:?]
at java.net.URLClassLoader$1.run(URLClassLoader.java:452) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at java.net.URLClassLoader.findClass(URLClassLoader.java:451) ~[?:?]
at
org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71)
~[flink-dist_2.12-1.11.0.jar:1.11.0]
at
org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
[flink-dist_2.12-1.11.0.jar:1.11.0]
at java.lang.ClassLoader.loadClass(ClassLoader.java:522) [?:?]
Re: [External Sender] Re: Random Task executor shutdown
(java.lang.OutOfMemoryError: Metaspace)
Posted by Flavio Pompermaier <po...@okkam.it>.
Could this problem be related to
https://issues.apache.org/jira/browse/FLINK-20333, where the instances of
java.io.ObjectStreamClass$Caches need to be cleaned up as a fix of
https://bugs.openjdk.java.net/browse/JDK-8199589?
On Wed, Nov 18, 2020 at 3:04 PM Flavio Pompermaier <po...@okkam.it>
wrote:
> Hi Kye,
> thank you a lot for your help.
>
> There are a couple of things that are not clear to me: in an ideal world,
> ChildFirstClassLoader instances should all be garbage collected right?
> In my case they are not because they have some pointers to some parent
> class loader for some class..am I right?
> Debugging a little bit the internals of the
> ChildFirstClass.loadClassWithoutExceptionHandling I saw that there are a
> couple of classes that are demanded to the parent class loader (i.e. the
> super.loadClassWithoutExceptionHandling() is called because a class is not
> found in the user classloader in the snippet at [1]):
>
> - javax.security.*
> - sun.security.krb5.Config
> - jdk.internal.reflect.*
> - javax.net.*
> - sun.misc.unsafe
> - com.microsoft.sqlserver.jdbc.*
>
> While the latter could be moved into the fat jar because it is the
> dependency of the JDBC connector, the other classes are correctly loaded
> from the system class loader..
> Is this stopping the user classloader to be correctly garbage collected?
> And a final question about the possible workaround to put the user jar
> into the Flink lib: in this case I don't have a jar to pass as argument to
> the CLI java client..is this a problem of the CLI client or not..?
> Should the run action be callable without specifying a user JAR but just
> the main class (in the case that it is in the lib dir)?
>
> [1]
> try {
> // check the URLs
> c = findClass(name);
> } catch (ClassNotFoundException e) {
> // let URLClassLoader do it, which will eventually call the parent
> c = super.loadClassWithoutExceptionHandling(name, resolve);
> }
>
>
> On Tue, Nov 17, 2020 at 8:51 PM Kye Bae <ky...@capitalone.com> wrote:
>
>> I see:
>> ( Metadata: )
>> ( reserved=122880KB, committed=121168KB)
>> ( used=117529KB)
>> ( free=3639KB)
>> ( waste=0KB =0.00%)
>>
>> Internal (reserved=855KB, committed=855KB)
>> (malloc=823KB #2442)
>> (mmap: reserved=32KB, committed=32KB)
>>
>> and
>>
>> 0x00000008405a5288 0x000000080036f648 0x00007f1d14005160 2547
>> 15675392 15603040 org.apache.flink.util.ChildFirstClassLoader
>>
>> 0x00000008405a5288 0x000000080036f648 0x00007f1ccc059790 2648
>> 16330752 16287960 org.apache.flink.util.ChildFirstClassLoader
>>
>> 0x00000008405a5288 0x000000080036f648 0x00007f1c90004d90 2626
>> 16461824 16391336 org.apache.flink.util.ChildFirstClassLoader
>>
>> 0x00000008405a5288 0x000000080036f648 0x00007f1c38685650 2648
>> 16330752 16286608 org.apache.flink.util.ChildFirstClassLoader
>>
>> 0x00000008405a5288 0x000000080036f648 0x00007f1c3801e590 2651
>> 16408576 16320192 org.apache.flink.util.ChildFirstClassLoader
>>
>>
>> The "Metadata" looks like there is a little over 3.5MB left. Not sure if
>> this is enough. Also, "Internal" memory reserved is less than 1 MB, but
>> this is where the Java direct memory buffers get allocated (Java NIO).
>> Don't know the details of your Flink job, but is this enough?
>>
>>
>> Flink 1.10 memory architecture requires care, and there should be enough
>> for both Flink and JVM internal operations. It shouldn't hurt to review
>> your memory requirements and allocate extra to account for unexpected
>> spikes.
>>
>>
>> Please keep in mind Yarn will kill a container immediately if the
>> physical memory limit is breached.
>>
>>
>> Also, it looks like ChildFirstClassLoader (user-code loader - dynamic)
>> was used 5 times to load classes onto this taskmanager. Were there 5 job
>> submissions to your standalone cluster or 4 job restarts for the same job?
>> Additional (dynamic) loadings will likely result in exceeding the max
>> metaspace limit.
>>
>>
>> If your Flink cluster is meant to host a single job, I would avoid using
>> the dynamic classloading altogether and then recapture the stats to see if
>> there are still multiple loading of the user (your application) classes. If
>> it doesn't, you might want to make that change to not use the dynamic
>> classloading.
>>
>>
>> -K
>>
>>
>>
>>
>> On Tue, Nov 17, 2020 at 2:00 PM Flavio Pompermaier <po...@okkam.it>
>> wrote:
>>
>>> Hi Kye, thanks a lot for the support! See my answers inline below
>>>
>>> On Tue, Nov 17, 2020 at 7:22 PM Kye Bae <ky...@capitalone.com> wrote:
>>>
>>>> It is possible, but I am not entirely sure about the load order
>>>> affecting the metaspace usage.
>>>>
>>>> To find out why your taskmanager container is exceeding the metaspace,
>>>> we would need to know what value the max metaspace size is set to and then
>>>> find out how much of the metaspace is actually being used and what the
>>>> classloading stats are (before Yarn kills the container).
>>>>
>>>> If possible, please share the current metaspace setting and the native
>>>> memory stats from one of your taskmanager instances (Java process as a Yarn
>>>> container). I think those run as the yarn user as opposed to the hadoop
>>>> user (it depends on your environment).
>>>>
>>>>
>>> Actually I don't use YARN, I use standalone cluster and MaxMetspaceSize
>>> is automatically computed and it is set to 256 MB
>>> (-XX:MaxMetaspaceSize=268435456)
>>>
>>> # enable the native memory tracking referenced earlier - this may need
>>>> to be passed in as the taskmanager JVM option parameter not the jobmanager.
>>>> # log onto one of the taskmanager HW instances
>>>> sudo su - yarn # or log on as the user that runs Yarn
>>>> jps -v # grab the PID for one of the processes named YarnTaskExecutorRunner
>>>> - this would be a taskmanager
>>>> jcmd TM_PID VM.native_memory summary # size information for "Class" -
>>>> metaspace
>>>>
>>>
>>> jcmd 6976 VM.native_memory summary
>>> 6976:
>>>
>>> Native Memory Tracking:
>>>
>>> Total: reserved=1473311KB, committed=954707KB
>>> - Java Heap (reserved=524288KB, committed=524288KB)
>>> (mmap: reserved=524288KB,
>>> committed=524288KB)
>>>
>>> - Class (reserved=379640KB, committed=142024KB)
>>> (classes #23605)
>>> ( instance classes #22058, array classes
>>> #1547)
>>> (malloc=2808KB #48655)
>>> (mmap: reserved=376832KB,
>>> committed=139216KB)
>>> ( Metadata: )
>>> ( reserved=122880KB, committed=121168KB)
>>> ( used=117529KB)
>>> ( free=3639KB)
>>> ( waste=0KB =0.00%)
>>> ( Class space:)
>>> ( reserved=253952KB, committed=18048KB)
>>> ( used=16516KB)
>>> ( free=1532KB)
>>> ( waste=0KB =0.00%)
>>>
>>> - Thread (reserved=76359KB, committed=6867KB)
>>> (thread #74)
>>> (stack: reserved=76012KB, committed=6520KB)
>>> (malloc=262KB #446)
>>> (arena=85KB #146)
>>>
>>> - Code (reserved=250407KB, committed=38911KB)
>>> (malloc=2719KB #11989)
>>> (mmap: reserved=247688KB, committed=36192KB)
>>>
>>> - GC (reserved=71481KB, committed=71481KB)
>>> (malloc=19121KB #17323)
>>> (mmap: reserved=52360KB, committed=52360KB)
>>>
>>> - Compiler (reserved=561KB, committed=561KB)
>>> (malloc=429KB #846)
>>> (arena=133KB #5)
>>>
>>> - Internal (reserved=855KB, committed=855KB)
>>> (malloc=823KB #2442)
>>> (mmap: reserved=32KB, committed=32KB)
>>>
>>> - Other (reserved=131857KB, committed=131857KB)
>>> (malloc=131857KB #4128)
>>>
>>> - Symbol (reserved=15702KB, committed=15702KB)
>>> (malloc=14288KB #165850)
>>> (arena=1415KB #1)
>>>
>>> - Native Memory Tracking (reserved=4072KB, committed=4072KB)
>>> (malloc=27KB #353)
>>> (tracking overhead=4045KB)
>>>
>>> - Shared class space (reserved=17044KB, committed=17044KB)
>>> (mmap: reserved=17044KB, committed=17044KB)
>>>
>>> - Arena Chunk (reserved=199KB, committed=199KB)
>>> (malloc=199KB)
>>>
>>> - Logging (reserved=4KB, committed=4KB)
>>> (malloc=4KB #191)
>>>
>>> - Arguments (reserved=20KB, committed=20KB)
>>> (malloc=20KB #501)
>>>
>>> - Module (reserved=568KB, committed=568KB)
>>> (malloc=568KB #3919)
>>>
>>> - Synchronizer (reserved=244KB, committed=244KB)
>>> (malloc=244KB #2054)
>>>
>>> - Safepoint (reserved=8KB, committed=8KB)
>>> (mmap: reserved=8KB, committed=8KB)
>>>
>>>
>>>
>>>> jcmd TM_PID VM.classloader_stats # how many classes were loaded by
>>>> which classloader, sizes, etc.
>>>>
>>>
>>> jcmd 6976 VM.classloader_stats
>>> 6976:
>>> ClassLoader Parent CLD* Classes
>>> ChunkSz BlockSz Type
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d8140 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c38193d50 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00a9f10 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ab290 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c38011150 1
>>> 6144 4488 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b5d90 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9004dec0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380ae230 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c6402a2f0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c5803b430 1
>>> 4096 3232 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1d1c072ce0 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c800b7ee0 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc336610 1
>>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8084aa0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cc0011ba0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c2f70 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90021090 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc8085cf0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d3830 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00becf0 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812a7c0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc000c880 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc018030 1
>>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8139a90 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90020cd0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9806f6c0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c9804c2d0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80bcf80 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc347ec0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc01aca0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38576590 1
>>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380f71f0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc40110d0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813c030 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c2440 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c8003c3a0 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9816bb00 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c900cee70 1
>>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c98053d30 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80b5a00 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e29b0 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90259870 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8053250 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c60057820 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c3867f350 1
>>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1d58d765c0 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cc4014c00 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e2f90 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ae450 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc014ae0 1
>>> 10240 7392 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813e880 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036ef80 0x00007f1c380851e0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb811f7f0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb811f080 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc0629c0 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc8064b40 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc14e8a0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813ab20 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c64026060 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc00078a0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc805dc50 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80b45f0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c3867eeb0 1
>>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc0008590 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c98050230 1
>>> 6144 4488 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00cbfa0 1
>>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8135800 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38060c30 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c0e20 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b0800 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c640399e0 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00cd350 1
>>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b3dc0 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c38089ee0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c98053b00 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c301da250 1
>>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc80f1050 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc014bf0 1
>>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c902a1a60 1
>>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c100bc340 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c304d0370 1
>>> 6144 4464 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812bf60 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c64051870 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38094a60 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38068320 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80b84d0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ca4037920 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8046890 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c30000dc0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00cbdd0 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380c0e40 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00dfae0 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90258ee0 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c9804f6f0 1
>>> 6144 4488 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90045b60 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c3808b700 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8065550 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8060390 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b5e40 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80e5cf0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc807f2c0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc805d8b0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c800adb40 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8081720 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc400d760 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c30326b80 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc804a0c0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9029f4a0 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90259b80 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc00048c0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00af830 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c8550 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb811e7a0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3808e490 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8129480 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x00000008405a5288 0x000000080036f648 0x00007f1d14005160 2547
>>> 15675392 15603040 org.apache.flink.util.ChildFirstClassLoader
>>> 61
>>> 172032 106152 + unsafe anonymous classes
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c380ea520 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c3857a720 1
>>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c7930 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cac004710 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00df0f0 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c3861a300 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80c6120 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c980d50c0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc00020d0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c64026190 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc009490 1
>>> 6144 4488 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d5510 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x0000559cae5496b0 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38081400 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d7110 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc8052210 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c64052a10 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc008aa0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c1930 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8134380 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812d770 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc337240 1
>>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c58016fd0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8140140 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc017460 1
>>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c98059040 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c38674d70 1
>>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38013c50 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c30072cb0 1
>>> 6144 4032 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc0005930 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80b4ae0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812fb30 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38032ba0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc0166a0 1
>>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000b9f0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80b8ca0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b48d0 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cf00647f0 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8053f00 1
>>> 6144 4488 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1d58d73980 1
>>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080036f648 0x000000080036ef80 0x00007f1d5822a1b0 4979
>>> 34353152 34252632 jdk.internal.loader.ClassLoaders$AppClassLoader
>>> 564
>>> 2703360 1908736 + unsafe anonymous classes
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc804fc70 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00bf800 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c6402f6c0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3808efa0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00cc960 1
>>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380b0820 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c6003d940 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c3863da40 1
>>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8135b60 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38078d80 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c902a0060 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x00000008405a5288 0x000000080036f648 0x00007f1ccc059790 2648
>>> 16330752 16287960 org.apache.flink.util.ChildFirstClassLoader
>>> 61
>>> 175104 109552 + unsafe anonymous classes
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc4013060 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00cb480 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c7b60 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c3054d770 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x00000008405a8e68 0x000000080036f648 0x00007f1c60007a50 1
>>> 6144 3024 sun.reflect.misc.MethodUtil
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00de270 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ca0028160 1
>>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c381d6b10 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d4c80 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38077ab0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c5660 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b8010 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1d58f59c10 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cac010bb0 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00bd6c0 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b53e0 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc400db60 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1d1404e940 1
>>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc0147d0 1
>>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b3430 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c98093270 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380bc7c0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cb8116dc0 1
>>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1d58d8aa50 1
>>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e49d0 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000f340 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cb8113720 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c980d21d0 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c88072820 1
>>> 6144 4024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3808f610 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d67e0 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc80883a0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc0134f0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3868db90 1
>>> 6144 4520 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c303249f0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3807aff0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc80e08d0 1
>>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1d1c045720 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c902475f0 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000fbb0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e2cd0 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb81297a0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00abd60 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8134860 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc000a6c0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c303246d0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38099a70 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812f150 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000c610 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c900072d0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cd4052fc0 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380140b0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90257e30 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cd402bd50 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38015370 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1d5857f7a0 1
>>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813b5d0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8136870 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb811a8c0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cf402a5a0 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c9804ebf0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c980aecb0 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80c4330 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc80b7850 1
>>> 6144 4488 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8084d80 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc017d20 1
>>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc804c7a0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c68003140 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ca4033900 1
>>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1d14042ba0 1
>>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8046210 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90006f20 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1d5857f230 1
>>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d7750 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ac830 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38088e70 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d0350 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb81378e0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c30325540 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1d5a281630 1
>>> 6144 4768 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813cbd0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c8860 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c3808a920 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c380899f0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc803f350 1
>>> 6144 4488 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c6d90 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380db5b0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d2130 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00adbc0 1
>>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb81312a0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc000ad00 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ca2c0 1
>>> 6144 4464 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00be210 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00bb5b0 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c386785e0 1
>>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cf0064e50 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38035d90 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e2e60 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d36c0 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc0002210 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c385c6f00 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e53c0 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c3a80 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x0000000000000000 0x0000000000000000 0x00007f1d58223a40 3735
>>> 16637952 16601232 <boot class loader>
>>> 364
>>> 1014784 658032 + unsafe anonymous classes
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90033cd0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b6a40 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80a1520 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1d58f38e10 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d0a00 1
>>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8120170 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1d1c04fb30 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c100c9b10 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb81205e0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8122f30 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000d170 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c60045bc0 1
>>> 6144 3992 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc04a8f0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1d58d8ce60 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e3930 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb81346e0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38079f60 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ad8c0 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380bc900 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc011de0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c9804e7f0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c7580 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3009d730 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a8e68 0x00007f1cc0011e00 1
>>> 3072 2176 jdk.internal.reflect.DelegatingClassLoader
>>> 0x00000008405a5288 0x000000080036f648 0x00007f1c90004d90 2626
>>> 16461824 16391336 org.apache.flink.util.ChildFirstClassLoader
>>> 61
>>> 177152 111624 + unsafe anonymous classes
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c0310 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc05d1f0 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c380e8400 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8130f80 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c9820e410 1
>>> 6144 3848 jdk.internal.reflect.DelegatingClassLoader
>>> 0x00000008405a5288 0x000000080036f648 0x00007f1c38685650 2648
>>> 16330752 16286608 org.apache.flink.util.ChildFirstClassLoader
>>> 61
>>> 175104 109552 + unsafe anonymous classes
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8042c50 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb81406e0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90034980 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000bd60 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80839f0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c38083d50 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90257460 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3803d6f0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c98074170 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380bb950 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38096730 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d9bd0 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c780016c0 1
>>> 4096 3160 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc80e27c0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38087f00 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38086150 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e3fe0 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1d1c047930 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c303d77a0 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cd402bf60 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38063580 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9816bdf0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9004e8c0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c5980 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c4570 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812b740 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc00074b0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380c1560 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8134b70 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ad5b0 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00bcbd0 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc4016dc0 1
>>> 3072 2192 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d99c0 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b7500 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380ad230 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38051d00 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8137a20 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80ba3d0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c980700f0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c386121b0 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ae770 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9029fe30 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x00000008405a5288 0x000000080036f648 0x00007f1c3801e590 2651
>>> 16408576 16320192 org.apache.flink.util.ChildFirstClassLoader
>>> 61
>>> 175104 109552 + unsafe anonymous classes
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ac470 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc0009c00 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c88061f30 1
>>> 6144 4024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c580154c0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3807be20 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8139df0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c5bd0 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8142d80 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8137710 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c60056030 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380c1990 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c902a0420 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1d58d74a40 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38061f80 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc80e05a0 1
>>> 4096 3160 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00bc0c0 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c981c06d0 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc046f80 1
>>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c30328250 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc804a340 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c60057560 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c304fdc50 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812d450 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c900222f0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cc001e850 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc00b6b0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc335880 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c38138510 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c3857d030 1
>>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38086290 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e04d0 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380870c0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e0830 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c980d5490 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1d58d75010 1
>>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc33bb70 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c900333a0 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3807cbb0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc019db0 1
>>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc000c510 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc000ba60 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8140870 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813a170 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8122d40 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c600092b0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c381d7610 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80aa1c0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc00062f0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8131f90 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c880739f0 1
>>> 6144 4024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d7420 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b1310 1
>>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3c20f0b0 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc01b6b0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc02fd70 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb811ac10 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc062900 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80aa4c0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc80bb720 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d9610 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8121e80 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c30327500 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80e1ad0 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080036ef80 0x0000000000000000 0x00007f1d582879b0 112
>>> 843776 783312 jdk.internal.loader.ClassLoaders$PlatformClassLoader
>>> 3
>>> 9216 5432 + unsafe anonymous classes
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8080100 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc80873d0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc00b300 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc2e7b90 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813c350 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90021200 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3c210130 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc8044950 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ca4031cc0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1d58d74ec0 1
>>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8081b00 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80ba9e0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc011840 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00da8d0 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000d5a0 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e57b0 1
>>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812e180 1
>>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc8047410 1
>>> 6144 4008 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3807a130 1
>>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8146e70 1
>>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>>> Total = 431 23605
>>> 140124160 137261400
>>> ChunkSz: Total size of all allocated metaspace chunks
>>> BlockSz: Total size of all allocated metaspace blocks (each chunk has
>>> several blocks)
>>>
>>>
>>>>
>>>> There is a gentleman who posts blogs on Flink, and he goes into a bit
>>>> more detail there. This was against Flink 1.9, but the foundational
>>>> concepts still apply to 1.10.
>>>> http://cloudsqale.com/2020/04/29/flink-1-9-off-heap-memory-on-yarn-troubleshooting-container-is-running-beyond-physical-memory-limits-errors/
>>>> <https://urldefense.com/v3/__http://cloudsqale.com/2020/04/29/flink-1-9-off-heap-memory-on-yarn-troubleshooting-container-is-running-beyond-physical-memory-limits-errors/__;!!EFVe01R3CjU!NKfuq7CJ3Ok0zh_7rBJ8PMx-Wo_Yh2_f993EG1lS-hJ4lbtgJtRRUH_EoOhvmEMAkQ$>
>>>>
>>>> -K
>>>>
>>>> On Tue, Nov 17, 2020 at 12:47 PM Flavio Pompermaier <
>>>> pompermaier@okkam.it> wrote:
>>>>
>>>>> Another big potential candidate is the fact that JDBC libs I use in my
>>>>> job are put into the Flink lib folder instead of putting them into the fat
>>>>> jar..tomorrow I'll try to see if the metaspace is getting cleared correctly
>>>>> after that change.
>>>>> Unfortunately our jobs were written before the child-first /
>>>>> parent-first classloading refactoring and at that time that was the way to
>>>>> go..but now it can cause this kind of problems if using child-first policy.
>>>>>
>>>>> On Mon, Nov 16, 2020 at 8:44 PM Flavio Pompermaier <
>>>>> pompermaier@okkam.it> wrote:
>>>>>
>>>>>> Thank you Kye for your insights...in my mind, if the job runs without
>>>>>> problems one or more times the heap size, and thus the medatadata-size, is
>>>>>> big enough and I should not increase it (on the same data of course).
>>>>>> So I'll try to understand who is leaking what..the advice to avoid
>>>>>> the dynamic class loading is just a workaround to me..there's something
>>>>>> wrong going on and tomorrow I'll try to understand the root cause of the
>>>>>> problem using -XX:NativeMemoryTracking=summary as you suggested.
>>>>>>
>>>>>> I'll keep you up to date with my findings..
>>>>>>
>>>>>> Best,
>>>>>> Flavio
>>>>>>
>>>>>> On Mon, Nov 16, 2020 at 8:22 PM Kye Bae <ky...@capitalone.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello!
>>>>>>>
>>>>>>> The JVM metaspace is where all the classes (not class instances or
>>>>>>> objects) get loaded. jmap -histo is going to show you the heap space usage
>>>>>>> info not the metaspace.
>>>>>>>
>>>>>>> You could inspect what is happening in the metaspace by using jcmd (e.g.,
>>>>>>> jcmd JPID VM.native_memory summary) after restarting the cluster
>>>>>>> with "*-XX:NativeMemoryTracking=summary"*
>>>>>>>
>>>>>>> *As the error message suggests, you may need to increase *taskmanager.memory.jvm-metaspace.size,
>>>>>>> but you need to be slightly careful when specifying the memory parameters
>>>>>>> in flink-conf.yaml in Flink 1.10 (they have an issue with a confusing error
>>>>>>> message).
>>>>>>>
>>>>>>> Anothing thing to keep in mind is that you may want to avoid using
>>>>>>> dynamic classloading (
>>>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/debugging_classloading.html#avoiding-dynamic-classloading-for-user-code
>>>>>>> <https://urldefense.com/v3/__https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/debugging_classloading.html*avoiding-dynamic-classloading-for-user-code__;Iw!!EFVe01R3CjU!L48mrpYG-_Tcldpp26edL4MKDU3RIHFmX89E-BzmJk6_RiLUqQUFvy4RHE5kTFVDig$>):
>>>>>>> when the job continuously fails for some temporary issues, it will load the
>>>>>>> same class files into the metaspace multiple times and could exceed
>>>>>>> whatever the limit you set it.
>>>>>>>
>>>>>>> -K
>>>>>>>
>>>>>>> On Mon, Nov 16, 2020 at 12:39 PM Jan Lukavský <je...@seznam.cz>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> The exclusions should not have any impact on that, because what
>>>>>>>> defines which classloader will load which class is not the presence or
>>>>>>>> particular class in a specific jar, but the configuration of
>>>>>>>> parent-first-patterns [1].
>>>>>>>>
>>>>>>>> If you don't use any flink internal imports, than it still might be
>>>>>>>> the case, that a class in any of the packages defined by the
>>>>>>>> parent-first-pattern to hold reference to your user-code classes, which
>>>>>>>> would cause the leak. I'd recommend to inspect the heap dump after several
>>>>>>>> restarts of the application and look for reference to Class objects from
>>>>>>>> the root set.
>>>>>>>>
>>>>>>>> Jan
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#class-loading
>>>>>>>> <https://urldefense.com/v3/__https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html*class-loading__;Iw!!EFVe01R3CjU!NF2uHM8m-1kZSP7E3-7ZhdKcQa2U3wTqWKDA8zSI4727gH8ASTFc4h9qBaX4_W2wJA$>
>>>>>>>> On 11/16/20 5:34 PM, Flavio Pompermaier wrote:
>>>>>>>>
>>>>>>>> I've tried to remove all possible imports of classes not contained
>>>>>>>> in the fat jar but I still face the same problem.
>>>>>>>> I've also tried to reduce as much as possible the exclude in the
>>>>>>>> shade section of the maven plugin (I took the one at [1]) so now I exclude
>>>>>>>> only few dependencies..could it be that I should include org.slf4j:* if I
>>>>>>>> use static import of it?
>>>>>>>>
>>>>>>>> <artifactSet>
>>>>>>>> <excludes>
>>>>>>>> <exclude>com.google.code.findbugs:jsr305</exclude>
>>>>>>>> <exclude>org.slf4j:*</exclude>
>>>>>>>> <exclude>log4j:*</exclude>
>>>>>>>> </excludes>
>>>>>>>> </artifactSet>
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html#appendix-template-for-building-a-jar-with-dependencies
>>>>>>>> <https://urldefense.com/v3/__https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html*appendix-template-for-building-a-jar-with-dependencies__;Iw!!EFVe01R3CjU!NF2uHM8m-1kZSP7E3-7ZhdKcQa2U3wTqWKDA8zSI4727gH8ASTFc4h9qBaWGhZYoqQ$>
>>>>>>>>
>>>>>>>> On Mon, Nov 16, 2020 at 3:29 PM Jan Lukavský <je...@seznam.cz>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Yes, that could definitely cause this. You should probably avoid
>>>>>>>>> using these flink-internal shaded classes and ship your own versions (not
>>>>>>>>> shaded).
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>>
>>>>>>>>> Jan
>>>>>>>>> On 11/16/20 3:22 PM, Flavio Pompermaier wrote:
>>>>>>>>>
>>>>>>>>> Thank you Jan for your valuable feedback.
>>>>>>>>> Could it be that I should not use import shaded-jackson classes in
>>>>>>>>> my user code?
>>>>>>>>> For example import
>>>>>>>>> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper?
>>>>>>>>>
>>>>>>>>> Bets,
>>>>>>>>> Flavio
>>>>>>>>>
>>>>>>>>> On Mon, Nov 16, 2020 at 3:15 PM Jan Lukavský <je...@seznam.cz>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Flavio,
>>>>>>>>>>
>>>>>>>>>> when I encountered quite similar problem that you describe, it
>>>>>>>>>> was related to a static storage located in class that was loaded
>>>>>>>>>> "parent-first". In my case it was it was in java.lang.ClassValue, but it
>>>>>>>>>> might (and probably will be) different in your case. The problem is that if
>>>>>>>>>> user-code registers something in some (static) storage located in class
>>>>>>>>>> loaded with parent (TaskTracker) classloader, then its associated classes
>>>>>>>>>> will never be GC'd and Metaspace will grow. A good starting point would be
>>>>>>>>>> not to focus on biggest consumers of heap (in general), but to look at
>>>>>>>>>> where the 15k objects of type Class are referenced from. That might help
>>>>>>>>>> you figure this out. I'm not sure if there is something that can be done in
>>>>>>>>>> general to prevent this type of leaks. That would be probably question on
>>>>>>>>>> dev@ mailing list.
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>>
>>>>>>>>>> Jan
>>>>>>>>>> On 11/16/20 2:27 PM, Flavio Pompermaier wrote:
>>>>>>>>>>
>>>>>>>>>> Hello everybody,
>>>>>>>>>> I was writing this email when a similar thread on this mailing
>>>>>>>>>> list appeared..
>>>>>>>>>> The difference is that the other problem seems to be related
>>>>>>>>>> with Flink 1.10 on YARN and does not output anything helpful in debugging
>>>>>>>>>> the cause of the problem.
>>>>>>>>>>
>>>>>>>>>> Indeed, in my use case I use Flink 1.11.0 and Flink on a
>>>>>>>>>> standalone session cluster (the job is submitted to the cluster using the
>>>>>>>>>> CLI client).
>>>>>>>>>> The problem arises when I submit the same job for about 20 times
>>>>>>>>>> (this number unfortunately is not deterministic and can change a little
>>>>>>>>>> bit). The error reported by the Task Executor is related to the ever
>>>>>>>>>> growing Metaspace..the error seems to be pretty detailed [1].
>>>>>>>>>>
>>>>>>>>>> I found the same issue in some previous threads on this mailing
>>>>>>>>>> list and I've tried to figure it out the cause of the problem. The issue is
>>>>>>>>>> that looking at the objects allocated I don't really get an idea of the
>>>>>>>>>> source of the problem because the type of objects that are consuming the
>>>>>>>>>> memory are of general purpose (i.e. Bytes, Integers and Strings)...these
>>>>>>>>>> are my "top" memory consumers if looking at the output of jmap -histo
>>>>>>>>>> <PID>:
>>>>>>>>>>
>>>>>>>>>> At run 0:
>>>>>>>>>>
>>>>>>>>>> num #instances #bytes class name (module)
>>>>>>>>>> -------------------------------------------------------
>>>>>>>>>> 1: 46238 13224056 [B (java.base@11.0.9.1)
>>>>>>>>>> 2: 3736 6536672 [I (java.base@11.0.9.1)
>>>>>>>>>> 3: 38081 913944 java.lang.String (
>>>>>>>>>> java.base@11.0.9.1)
>>>>>>>>>> 4: 26 852384
>>>>>>>>>> [Lakka.dispatch.forkjoin.ForkJoinTask;
>>>>>>>>>> 5: 7146 844984 java.lang.Class (
>>>>>>>>>> java.base@11.0.9.1)
>>>>>>>>>>
>>>>>>>>>> At run 1:
>>>>>>>>>>
>>>>>>>>>> 1: 77.608 25.317.496 [B (java.base@11.0.9.1)
>>>>>>>>>> 2: 7.004 9.088.360 [I (java.base@11.0.9.1)
>>>>>>>>>> 3: 15.814 1.887.256 java.lang.Class (
>>>>>>>>>> java.base@11.0.9.1)
>>>>>>>>>> 4: 67.381 1.617.144 java.lang.String (
>>>>>>>>>> java.base@11.0.9.1)
>>>>>>>>>> 5: 3.906 1.422.960 [Ljava.util.HashMap$Node; (
>>>>>>>>>> java.base@11.0.9.1)
>>>>>>>>>>
>>>>>>>>>> At run 6:
>>>>>>>>>>
>>>>>>>>>> 1: 81.408 25.375.400 [B (java.base@11.0.9.1)
>>>>>>>>>> 2: 12.479 7.249.392 [I (java.base@11.0.9.1)
>>>>>>>>>> 3: 29.090 3.496.168 java.lang.Class (
>>>>>>>>>> java.base@11.0.9.1)
>>>>>>>>>> 4: 4.347 2.813.416 [Ljava.util.HashMap$Node; (
>>>>>>>>>> java.base@11.0.9.1)
>>>>>>>>>> 5: 71.584 1.718.016 java.lang.String (
>>>>>>>>>> java.base@11.0.9.1)
>>>>>>>>>>
>>>>>>>>>> At run 8:
>>>>>>>>>>
>>>>>>>>>> 1: 985.979 127.193.256 [B (java.base@11.0.9.1)
>>>>>>>>>> 2: 35.400 13.702.112 [I (java.base@11.0.9.1)
>>>>>>>>>> 3: 260.387 6.249.288 java.lang.String (
>>>>>>>>>> java.base@11.0.9.1)
>>>>>>>>>> 4: 148.836 5.953.440
>>>>>>>>>> java.util.HashMap$KeyIterator (java.base@11.0.9.1)
>>>>>>>>>> 5: 17.641 5.222.344 [Ljava.util.HashMap$Node; (
>>>>>>>>>> java.base@11.0.9.1)
>>>>>>>>>>
>>>>>>>>>> Thanks in advance for any help,
>>>>>>>>>> Flavio
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> --------------------------------------------------------------------------------------------------
>>>>>>>>>> java.lang.OutOfMemoryError: Metaspace. The metaspace
>>>>>>>>>> out-of-memory error has occurred. This can mean two things: either the job
>>>>>>>>>> requires a larger size of JVM metaspace to load classes or there is a class
>>>>>>>>>> loading leak. In the first case 'taskmanager.memory.jvm-metaspace.size'
>>>>>>>>>> configuration option should be increased. If the error persists (usually in
>>>>>>>>>> cluster after several job (re-)submissions) then there is probably a class
>>>>>>>>>> loading leak in user code or some of its dependencies which has to be
>>>>>>>>>> investigated and fixed. The task executor has to be shutdown...
>>>>>>>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>>>>>>> ~[?:?]
>>>>>>>>>> at
>>>>>>>>>> java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
>>>>>>>>>> at
>>>>>>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
>>>>>>>>>> ~[?:?]
>>>>>>>>>> at
>>>>>>>>>> java.net.URLClassLoader.defineClass(URLClassLoader.java:550) ~[?:?]
>>>>>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:458)
>>>>>>>>>> ~[?:?]
>>>>>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:452)
>>>>>>>>>> ~[?:?]
>>>>>>>>>> at java.security.AccessController.doPrivileged(Native
>>>>>>>>>> Method) ~[?:?]
>>>>>>>>>> at
>>>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:451) ~[?:?]
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71)
>>>>>>>>>> ~[flink-dist_2.12-1.11.0.jar:1.11.0]
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
>>>>>>>>>> [flink-dist_2.12-1.11.0.jar:1.11.0]
>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:522)
>>>>>>>>>> [?:?]
>>>>>>>>>>
>>>>>>>>>> ------------------------------
>>>>>>>
>>>>>>> The information contained in this e-mail is confidential and/or
>>>>>>> proprietary to Capital One and/or its affiliates and may only be used
>>>>>>> solely in performance of work or services for Capital One. The information
>>>>>>> transmitted herewith is intended only for use by the individual or entity
>>>>>>> to which it is addressed. If the reader of this message is not the intended
>>>>>>> recipient, you are hereby notified that any review, retransmission,
>>>>>>> dissemination, distribution, copying or other use of, or taking of any
>>>>>>> action in reliance upon this information is strictly prohibited. If you
>>>>>>> have received this communication in error, please contact the sender and
>>>>>>> delete the material from your computer.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>> ------------------------------
>>>>
>>>> The information contained in this e-mail is confidential and/or
>>>> proprietary to Capital One and/or its affiliates and may only be used
>>>> solely in performance of work or services for Capital One. The information
>>>> transmitted herewith is intended only for use by the individual or entity
>>>> to which it is addressed. If the reader of this message is not the intended
>>>> recipient, you are hereby notified that any review, retransmission,
>>>> dissemination, distribution, copying or other use of, or taking of any
>>>> action in reliance upon this information is strictly prohibited. If you
>>>> have received this communication in error, please contact the sender and
>>>> delete the material from your computer.
>>>>
>>>>
>>>>
>>>> ------------------------------
>>
>> The information contained in this e-mail is confidential and/or
>> proprietary to Capital One and/or its affiliates and may only be used
>> solely in performance of work or services for Capital One. The information
>> transmitted herewith is intended only for use by the individual or entity
>> to which it is addressed. If the reader of this message is not the intended
>> recipient, you are hereby notified that any review, retransmission,
>> dissemination, distribution, copying or other use of, or taking of any
>> action in reliance upon this information is strictly prohibited. If you
>> have received this communication in error, please contact the sender and
>> delete the material from your computer.
>>
>>
>>
>>
Re: [External Sender] Re: Random Task executor shutdown
(java.lang.OutOfMemoryError: Metaspace)
Posted by Flavio Pompermaier <po...@okkam.it>.
Hi Kye,
thank you a lot for your help.
There are a couple of things that are not clear to me: in an ideal world,
ChildFirstClassLoader instances should all be garbage collected right?
In my case they are not because they have some pointers to some parent
class loader for some class..am I right?
Debugging a little bit the internals of the
ChildFirstClass.loadClassWithoutExceptionHandling I saw that there are a
couple of classes that are demanded to the parent class loader (i.e. the
super.loadClassWithoutExceptionHandling() is called because a class is not
found in the user classloader in the snippet at [1]):
- javax.security.*
- sun.security.krb5.Config
- jdk.internal.reflect.*
- javax.net.*
- sun.misc.unsafe
- com.microsoft.sqlserver.jdbc.*
While the latter could be moved into the fat jar because it is the
dependency of the JDBC connector, the other classes are correctly loaded
from the system class loader..
Is this stopping the user classloader to be correctly garbage collected?
And a final question about the possible workaround to put the user jar into
the Flink lib: in this case I don't have a jar to pass as argument to the
CLI java client..is this a problem of the CLI client or not..?
Should the run action be callable without specifying a user JAR but just
the main class (in the case that it is in the lib dir)?
[1]
try {
// check the URLs
c = findClass(name);
} catch (ClassNotFoundException e) {
// let URLClassLoader do it, which will eventually call the parent
c = super.loadClassWithoutExceptionHandling(name, resolve);
}
On Tue, Nov 17, 2020 at 8:51 PM Kye Bae <ky...@capitalone.com> wrote:
> I see:
> ( Metadata: )
> ( reserved=122880KB, committed=121168KB)
> ( used=117529KB)
> ( free=3639KB)
> ( waste=0KB =0.00%)
>
> Internal (reserved=855KB, committed=855KB)
> (malloc=823KB #2442)
> (mmap: reserved=32KB, committed=32KB)
>
> and
>
> 0x00000008405a5288 0x000000080036f648 0x00007f1d14005160 2547
> 15675392 15603040 org.apache.flink.util.ChildFirstClassLoader
>
> 0x00000008405a5288 0x000000080036f648 0x00007f1ccc059790 2648
> 16330752 16287960 org.apache.flink.util.ChildFirstClassLoader
>
> 0x00000008405a5288 0x000000080036f648 0x00007f1c90004d90 2626
> 16461824 16391336 org.apache.flink.util.ChildFirstClassLoader
>
> 0x00000008405a5288 0x000000080036f648 0x00007f1c38685650 2648
> 16330752 16286608 org.apache.flink.util.ChildFirstClassLoader
>
> 0x00000008405a5288 0x000000080036f648 0x00007f1c3801e590 2651
> 16408576 16320192 org.apache.flink.util.ChildFirstClassLoader
>
>
> The "Metadata" looks like there is a little over 3.5MB left. Not sure if
> this is enough. Also, "Internal" memory reserved is less than 1 MB, but
> this is where the Java direct memory buffers get allocated (Java NIO).
> Don't know the details of your Flink job, but is this enough?
>
>
> Flink 1.10 memory architecture requires care, and there should be enough
> for both Flink and JVM internal operations. It shouldn't hurt to review
> your memory requirements and allocate extra to account for unexpected
> spikes.
>
>
> Please keep in mind Yarn will kill a container immediately if the physical
> memory limit is breached.
>
>
> Also, it looks like ChildFirstClassLoader (user-code loader - dynamic) was
> used 5 times to load classes onto this taskmanager. Were there 5 job
> submissions to your standalone cluster or 4 job restarts for the same job?
> Additional (dynamic) loadings will likely result in exceeding the max
> metaspace limit.
>
>
> If your Flink cluster is meant to host a single job, I would avoid using
> the dynamic classloading altogether and then recapture the stats to see if
> there are still multiple loading of the user (your application) classes. If
> it doesn't, you might want to make that change to not use the dynamic
> classloading.
>
>
> -K
>
>
>
>
> On Tue, Nov 17, 2020 at 2:00 PM Flavio Pompermaier <po...@okkam.it>
> wrote:
>
>> Hi Kye, thanks a lot for the support! See my answers inline below
>>
>> On Tue, Nov 17, 2020 at 7:22 PM Kye Bae <ky...@capitalone.com> wrote:
>>
>>> It is possible, but I am not entirely sure about the load order
>>> affecting the metaspace usage.
>>>
>>> To find out why your taskmanager container is exceeding the metaspace,
>>> we would need to know what value the max metaspace size is set to and then
>>> find out how much of the metaspace is actually being used and what the
>>> classloading stats are (before Yarn kills the container).
>>>
>>> If possible, please share the current metaspace setting and the native
>>> memory stats from one of your taskmanager instances (Java process as a Yarn
>>> container). I think those run as the yarn user as opposed to the hadoop
>>> user (it depends on your environment).
>>>
>>>
>> Actually I don't use YARN, I use standalone cluster and MaxMetspaceSize
>> is automatically computed and it is set to 256 MB
>> (-XX:MaxMetaspaceSize=268435456)
>>
>> # enable the native memory tracking referenced earlier - this may need to
>>> be passed in as the taskmanager JVM option parameter not the jobmanager.
>>> # log onto one of the taskmanager HW instances
>>> sudo su - yarn # or log on as the user that runs Yarn
>>> jps -v # grab the PID for one of the processes named YarnTaskExecutorRunner
>>> - this would be a taskmanager
>>> jcmd TM_PID VM.native_memory summary # size information for "Class" -
>>> metaspace
>>>
>>
>> jcmd 6976 VM.native_memory summary
>> 6976:
>>
>> Native Memory Tracking:
>>
>> Total: reserved=1473311KB, committed=954707KB
>> - Java Heap (reserved=524288KB, committed=524288KB)
>> (mmap: reserved=524288KB, committed=524288KB)
>>
>> - Class (reserved=379640KB, committed=142024KB)
>> (classes #23605)
>> ( instance classes #22058, array classes
>> #1547)
>> (malloc=2808KB #48655)
>> (mmap: reserved=376832KB, committed=139216KB)
>> ( Metadata: )
>> ( reserved=122880KB, committed=121168KB)
>> ( used=117529KB)
>> ( free=3639KB)
>> ( waste=0KB =0.00%)
>> ( Class space:)
>> ( reserved=253952KB, committed=18048KB)
>> ( used=16516KB)
>> ( free=1532KB)
>> ( waste=0KB =0.00%)
>>
>> - Thread (reserved=76359KB, committed=6867KB)
>> (thread #74)
>> (stack: reserved=76012KB, committed=6520KB)
>> (malloc=262KB #446)
>> (arena=85KB #146)
>>
>> - Code (reserved=250407KB, committed=38911KB)
>> (malloc=2719KB #11989)
>> (mmap: reserved=247688KB, committed=36192KB)
>>
>> - GC (reserved=71481KB, committed=71481KB)
>> (malloc=19121KB #17323)
>> (mmap: reserved=52360KB, committed=52360KB)
>>
>> - Compiler (reserved=561KB, committed=561KB)
>> (malloc=429KB #846)
>> (arena=133KB #5)
>>
>> - Internal (reserved=855KB, committed=855KB)
>> (malloc=823KB #2442)
>> (mmap: reserved=32KB, committed=32KB)
>>
>> - Other (reserved=131857KB, committed=131857KB)
>> (malloc=131857KB #4128)
>>
>> - Symbol (reserved=15702KB, committed=15702KB)
>> (malloc=14288KB #165850)
>> (arena=1415KB #1)
>>
>> - Native Memory Tracking (reserved=4072KB, committed=4072KB)
>> (malloc=27KB #353)
>> (tracking overhead=4045KB)
>>
>> - Shared class space (reserved=17044KB, committed=17044KB)
>> (mmap: reserved=17044KB, committed=17044KB)
>>
>> - Arena Chunk (reserved=199KB, committed=199KB)
>> (malloc=199KB)
>>
>> - Logging (reserved=4KB, committed=4KB)
>> (malloc=4KB #191)
>>
>> - Arguments (reserved=20KB, committed=20KB)
>> (malloc=20KB #501)
>>
>> - Module (reserved=568KB, committed=568KB)
>> (malloc=568KB #3919)
>>
>> - Synchronizer (reserved=244KB, committed=244KB)
>> (malloc=244KB #2054)
>>
>> - Safepoint (reserved=8KB, committed=8KB)
>> (mmap: reserved=8KB, committed=8KB)
>>
>>
>>
>>> jcmd TM_PID VM.classloader_stats # how many classes were loaded by
>>> which classloader, sizes, etc.
>>>
>>
>> jcmd 6976 VM.classloader_stats
>> 6976:
>> ClassLoader Parent CLD* Classes
>> ChunkSz BlockSz Type
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d8140 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c38193d50 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00a9f10 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ab290 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c38011150 1
>> 6144 4488 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b5d90 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9004dec0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380ae230 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c6402a2f0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c5803b430 1
>> 4096 3232 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1d1c072ce0 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c800b7ee0 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc336610 1
>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8084aa0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cc0011ba0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c2f70 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90021090 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc8085cf0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d3830 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00becf0 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812a7c0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc000c880 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc018030 1
>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8139a90 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90020cd0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9806f6c0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c9804c2d0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80bcf80 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc347ec0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc01aca0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38576590 1
>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380f71f0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc40110d0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813c030 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c2440 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c8003c3a0 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9816bb00 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c900cee70 1
>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c98053d30 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80b5a00 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e29b0 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90259870 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8053250 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c60057820 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c3867f350 1
>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1d58d765c0 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cc4014c00 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e2f90 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ae450 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc014ae0 1
>> 10240 7392 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813e880 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036ef80 0x00007f1c380851e0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb811f7f0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb811f080 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc0629c0 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc8064b40 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc14e8a0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813ab20 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c64026060 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc00078a0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc805dc50 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80b45f0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c3867eeb0 1
>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc0008590 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c98050230 1
>> 6144 4488 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00cbfa0 1
>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8135800 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38060c30 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c0e20 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b0800 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c640399e0 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00cd350 1
>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b3dc0 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c38089ee0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c98053b00 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c301da250 1
>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc80f1050 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc014bf0 1
>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c902a1a60 1
>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c100bc340 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c304d0370 1
>> 6144 4464 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812bf60 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c64051870 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38094a60 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38068320 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80b84d0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ca4037920 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8046890 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c30000dc0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00cbdd0 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380c0e40 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00dfae0 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90258ee0 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c9804f6f0 1
>> 6144 4488 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90045b60 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c3808b700 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8065550 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8060390 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b5e40 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80e5cf0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc807f2c0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc805d8b0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c800adb40 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8081720 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc400d760 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c30326b80 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc804a0c0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9029f4a0 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90259b80 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc00048c0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00af830 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c8550 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb811e7a0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3808e490 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8129480 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x00000008405a5288 0x000000080036f648 0x00007f1d14005160 2547
>> 15675392 15603040 org.apache.flink.util.ChildFirstClassLoader
>> 61
>> 172032 106152 + unsafe anonymous classes
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c380ea520 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c3857a720 1
>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c7930 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cac004710 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00df0f0 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c3861a300 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80c6120 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c980d50c0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc00020d0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c64026190 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc009490 1
>> 6144 4488 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d5510 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x0000559cae5496b0 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38081400 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d7110 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc8052210 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c64052a10 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc008aa0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c1930 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8134380 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812d770 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc337240 1
>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c58016fd0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8140140 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc017460 1
>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c98059040 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c38674d70 1
>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38013c50 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c30072cb0 1
>> 6144 4032 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc0005930 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80b4ae0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812fb30 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38032ba0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc0166a0 1
>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000b9f0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80b8ca0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b48d0 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cf00647f0 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8053f00 1
>> 6144 4488 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1d58d73980 1
>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080036f648 0x000000080036ef80 0x00007f1d5822a1b0 4979
>> 34353152 34252632 jdk.internal.loader.ClassLoaders$AppClassLoader
>> 564
>> 2703360 1908736 + unsafe anonymous classes
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc804fc70 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00bf800 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c6402f6c0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3808efa0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00cc960 1
>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380b0820 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c6003d940 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c3863da40 1
>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8135b60 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38078d80 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c902a0060 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x00000008405a5288 0x000000080036f648 0x00007f1ccc059790 2648
>> 16330752 16287960 org.apache.flink.util.ChildFirstClassLoader
>> 61
>> 175104 109552 + unsafe anonymous classes
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc4013060 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00cb480 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c7b60 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c3054d770 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x00000008405a8e68 0x000000080036f648 0x00007f1c60007a50 1
>> 6144 3024 sun.reflect.misc.MethodUtil
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00de270 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ca0028160 1
>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c381d6b10 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d4c80 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38077ab0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c5660 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b8010 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1d58f59c10 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cac010bb0 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00bd6c0 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b53e0 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc400db60 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1d1404e940 1
>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc0147d0 1
>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b3430 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c98093270 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380bc7c0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cb8116dc0 1
>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1d58d8aa50 1
>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e49d0 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000f340 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cb8113720 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c980d21d0 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c88072820 1
>> 6144 4024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3808f610 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d67e0 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc80883a0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc0134f0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3868db90 1
>> 6144 4520 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c303249f0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3807aff0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc80e08d0 1
>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1d1c045720 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c902475f0 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000fbb0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e2cd0 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb81297a0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00abd60 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8134860 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc000a6c0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c303246d0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38099a70 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812f150 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000c610 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c900072d0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cd4052fc0 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380140b0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90257e30 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cd402bd50 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38015370 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1d5857f7a0 1
>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813b5d0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8136870 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb811a8c0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cf402a5a0 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c9804ebf0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c980aecb0 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80c4330 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc80b7850 1
>> 6144 4488 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8084d80 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc017d20 1
>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc804c7a0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c68003140 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ca4033900 1
>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1d14042ba0 1
>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8046210 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90006f20 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1d5857f230 1
>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d7750 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ac830 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38088e70 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d0350 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb81378e0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c30325540 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1d5a281630 1
>> 6144 4768 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813cbd0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c8860 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c3808a920 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c380899f0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc803f350 1
>> 6144 4488 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c6d90 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380db5b0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d2130 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00adbc0 1
>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb81312a0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc000ad00 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ca2c0 1
>> 6144 4464 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00be210 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00bb5b0 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c386785e0 1
>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cf0064e50 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38035d90 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e2e60 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d36c0 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc0002210 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c385c6f00 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e53c0 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c3a80 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x0000000000000000 0x0000000000000000 0x00007f1d58223a40 3735
>> 16637952 16601232 <boot class loader>
>> 364
>> 1014784 658032 + unsafe anonymous classes
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90033cd0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b6a40 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80a1520 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1d58f38e10 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d0a00 1
>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8120170 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1d1c04fb30 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c100c9b10 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb81205e0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8122f30 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000d170 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c60045bc0 1
>> 6144 3992 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc04a8f0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1d58d8ce60 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e3930 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb81346e0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38079f60 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ad8c0 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380bc900 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc011de0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c9804e7f0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c7580 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3009d730 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a8e68 0x00007f1cc0011e00 1
>> 3072 2176 jdk.internal.reflect.DelegatingClassLoader
>> 0x00000008405a5288 0x000000080036f648 0x00007f1c90004d90 2626
>> 16461824 16391336 org.apache.flink.util.ChildFirstClassLoader
>> 61
>> 177152 111624 + unsafe anonymous classes
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c0310 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc05d1f0 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c380e8400 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8130f80 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c9820e410 1
>> 6144 3848 jdk.internal.reflect.DelegatingClassLoader
>> 0x00000008405a5288 0x000000080036f648 0x00007f1c38685650 2648
>> 16330752 16286608 org.apache.flink.util.ChildFirstClassLoader
>> 61
>> 175104 109552 + unsafe anonymous classes
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8042c50 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb81406e0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90034980 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000bd60 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80839f0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c38083d50 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90257460 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3803d6f0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c98074170 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380bb950 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38096730 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d9bd0 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c780016c0 1
>> 4096 3160 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc80e27c0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38087f00 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38086150 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e3fe0 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1d1c047930 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c303d77a0 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cd402bf60 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38063580 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9816bdf0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9004e8c0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c5980 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c4570 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812b740 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc00074b0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380c1560 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8134b70 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ad5b0 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00bcbd0 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc4016dc0 1
>> 3072 2192 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d99c0 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b7500 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380ad230 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38051d00 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8137a20 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80ba3d0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c980700f0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c386121b0 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ae770 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9029fe30 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x00000008405a5288 0x000000080036f648 0x00007f1c3801e590 2651
>> 16408576 16320192 org.apache.flink.util.ChildFirstClassLoader
>> 61
>> 175104 109552 + unsafe anonymous classes
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ac470 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc0009c00 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c88061f30 1
>> 6144 4024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c580154c0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3807be20 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8139df0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c5bd0 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8142d80 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8137710 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c60056030 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380c1990 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c902a0420 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1d58d74a40 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38061f80 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc80e05a0 1
>> 4096 3160 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00bc0c0 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c981c06d0 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc046f80 1
>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c30328250 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc804a340 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c60057560 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c304fdc50 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812d450 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c900222f0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cc001e850 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc00b6b0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc335880 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c38138510 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c3857d030 1
>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38086290 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e04d0 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380870c0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e0830 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c980d5490 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1d58d75010 1
>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc33bb70 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c900333a0 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3807cbb0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc019db0 1
>> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc000c510 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc000ba60 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8140870 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813a170 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8122d40 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c600092b0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c381d7610 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80aa1c0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc00062f0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8131f90 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1c880739f0 1
>> 6144 4024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d7420 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b1310 1
>> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3c20f0b0 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc01b6b0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc02fd70 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb811ac10 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc062900 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80aa4c0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc80bb720 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d9610 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8121e80 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c30327500 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80e1ad0 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080036ef80 0x0000000000000000 0x00007f1d582879b0 112
>> 843776 783312 jdk.internal.loader.ClassLoaders$PlatformClassLoader
>> 3
>> 9216 5432 + unsafe anonymous classes
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8080100 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc80873d0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc00b300 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc2e7b90 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813c350 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90021200 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3c210130 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc8044950 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ca4031cc0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1d58d74ec0 1
>> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8081b00 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80ba9e0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc011840 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00da8d0 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000d5a0 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e57b0 1
>> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812e180 1
>> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc8047410 1
>> 6144 4008 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3807a130 1
>> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
>> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8146e70 1
>> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
>> Total = 431 23605
>> 140124160 137261400
>> ChunkSz: Total size of all allocated metaspace chunks
>> BlockSz: Total size of all allocated metaspace blocks (each chunk has
>> several blocks)
>>
>>
>>>
>>> There is a gentleman who posts blogs on Flink, and he goes into a bit
>>> more detail there. This was against Flink 1.9, but the foundational
>>> concepts still apply to 1.10.
>>> http://cloudsqale.com/2020/04/29/flink-1-9-off-heap-memory-on-yarn-troubleshooting-container-is-running-beyond-physical-memory-limits-errors/
>>> <https://urldefense.com/v3/__http://cloudsqale.com/2020/04/29/flink-1-9-off-heap-memory-on-yarn-troubleshooting-container-is-running-beyond-physical-memory-limits-errors/__;!!EFVe01R3CjU!NKfuq7CJ3Ok0zh_7rBJ8PMx-Wo_Yh2_f993EG1lS-hJ4lbtgJtRRUH_EoOhvmEMAkQ$>
>>>
>>> -K
>>>
>>> On Tue, Nov 17, 2020 at 12:47 PM Flavio Pompermaier <
>>> pompermaier@okkam.it> wrote:
>>>
>>>> Another big potential candidate is the fact that JDBC libs I use in my
>>>> job are put into the Flink lib folder instead of putting them into the fat
>>>> jar..tomorrow I'll try to see if the metaspace is getting cleared correctly
>>>> after that change.
>>>> Unfortunately our jobs were written before the child-first /
>>>> parent-first classloading refactoring and at that time that was the way to
>>>> go..but now it can cause this kind of problems if using child-first policy.
>>>>
>>>> On Mon, Nov 16, 2020 at 8:44 PM Flavio Pompermaier <
>>>> pompermaier@okkam.it> wrote:
>>>>
>>>>> Thank you Kye for your insights...in my mind, if the job runs without
>>>>> problems one or more times the heap size, and thus the medatadata-size, is
>>>>> big enough and I should not increase it (on the same data of course).
>>>>> So I'll try to understand who is leaking what..the advice to avoid the
>>>>> dynamic class loading is just a workaround to me..there's something wrong
>>>>> going on and tomorrow I'll try to understand the root cause of the
>>>>> problem using -XX:NativeMemoryTracking=summary as you suggested.
>>>>>
>>>>> I'll keep you up to date with my findings..
>>>>>
>>>>> Best,
>>>>> Flavio
>>>>>
>>>>> On Mon, Nov 16, 2020 at 8:22 PM Kye Bae <ky...@capitalone.com>
>>>>> wrote:
>>>>>
>>>>>> Hello!
>>>>>>
>>>>>> The JVM metaspace is where all the classes (not class instances or
>>>>>> objects) get loaded. jmap -histo is going to show you the heap space usage
>>>>>> info not the metaspace.
>>>>>>
>>>>>> You could inspect what is happening in the metaspace by using jcmd (e.g.,
>>>>>> jcmd JPID VM.native_memory summary) after restarting the cluster
>>>>>> with "*-XX:NativeMemoryTracking=summary"*
>>>>>>
>>>>>> *As the error message suggests, you may need to increase *taskmanager.memory.jvm-metaspace.size,
>>>>>> but you need to be slightly careful when specifying the memory parameters
>>>>>> in flink-conf.yaml in Flink 1.10 (they have an issue with a confusing error
>>>>>> message).
>>>>>>
>>>>>> Anothing thing to keep in mind is that you may want to avoid using
>>>>>> dynamic classloading (
>>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/debugging_classloading.html#avoiding-dynamic-classloading-for-user-code
>>>>>> <https://urldefense.com/v3/__https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/debugging_classloading.html*avoiding-dynamic-classloading-for-user-code__;Iw!!EFVe01R3CjU!L48mrpYG-_Tcldpp26edL4MKDU3RIHFmX89E-BzmJk6_RiLUqQUFvy4RHE5kTFVDig$>):
>>>>>> when the job continuously fails for some temporary issues, it will load the
>>>>>> same class files into the metaspace multiple times and could exceed
>>>>>> whatever the limit you set it.
>>>>>>
>>>>>> -K
>>>>>>
>>>>>> On Mon, Nov 16, 2020 at 12:39 PM Jan Lukavský <je...@seznam.cz>
>>>>>> wrote:
>>>>>>
>>>>>>> The exclusions should not have any impact on that, because what
>>>>>>> defines which classloader will load which class is not the presence or
>>>>>>> particular class in a specific jar, but the configuration of
>>>>>>> parent-first-patterns [1].
>>>>>>>
>>>>>>> If you don't use any flink internal imports, than it still might be
>>>>>>> the case, that a class in any of the packages defined by the
>>>>>>> parent-first-pattern to hold reference to your user-code classes, which
>>>>>>> would cause the leak. I'd recommend to inspect the heap dump after several
>>>>>>> restarts of the application and look for reference to Class objects from
>>>>>>> the root set.
>>>>>>>
>>>>>>> Jan
>>>>>>>
>>>>>>> [1]
>>>>>>> https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#class-loading
>>>>>>> <https://urldefense.com/v3/__https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html*class-loading__;Iw!!EFVe01R3CjU!NF2uHM8m-1kZSP7E3-7ZhdKcQa2U3wTqWKDA8zSI4727gH8ASTFc4h9qBaX4_W2wJA$>
>>>>>>> On 11/16/20 5:34 PM, Flavio Pompermaier wrote:
>>>>>>>
>>>>>>> I've tried to remove all possible imports of classes not contained
>>>>>>> in the fat jar but I still face the same problem.
>>>>>>> I've also tried to reduce as much as possible the exclude in the
>>>>>>> shade section of the maven plugin (I took the one at [1]) so now I exclude
>>>>>>> only few dependencies..could it be that I should include org.slf4j:* if I
>>>>>>> use static import of it?
>>>>>>>
>>>>>>> <artifactSet>
>>>>>>> <excludes>
>>>>>>> <exclude>com.google.code.findbugs:jsr305</exclude>
>>>>>>> <exclude>org.slf4j:*</exclude>
>>>>>>> <exclude>log4j:*</exclude>
>>>>>>> </excludes>
>>>>>>> </artifactSet>
>>>>>>>
>>>>>>> [1]
>>>>>>> https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html#appendix-template-for-building-a-jar-with-dependencies
>>>>>>> <https://urldefense.com/v3/__https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html*appendix-template-for-building-a-jar-with-dependencies__;Iw!!EFVe01R3CjU!NF2uHM8m-1kZSP7E3-7ZhdKcQa2U3wTqWKDA8zSI4727gH8ASTFc4h9qBaWGhZYoqQ$>
>>>>>>>
>>>>>>> On Mon, Nov 16, 2020 at 3:29 PM Jan Lukavský <je...@seznam.cz>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Yes, that could definitely cause this. You should probably avoid
>>>>>>>> using these flink-internal shaded classes and ship your own versions (not
>>>>>>>> shaded).
>>>>>>>>
>>>>>>>> Best,
>>>>>>>>
>>>>>>>> Jan
>>>>>>>> On 11/16/20 3:22 PM, Flavio Pompermaier wrote:
>>>>>>>>
>>>>>>>> Thank you Jan for your valuable feedback.
>>>>>>>> Could it be that I should not use import shaded-jackson classes in
>>>>>>>> my user code?
>>>>>>>> For example import
>>>>>>>> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper?
>>>>>>>>
>>>>>>>> Bets,
>>>>>>>> Flavio
>>>>>>>>
>>>>>>>> On Mon, Nov 16, 2020 at 3:15 PM Jan Lukavský <je...@seznam.cz>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Flavio,
>>>>>>>>>
>>>>>>>>> when I encountered quite similar problem that you describe, it was
>>>>>>>>> related to a static storage located in class that was loaded
>>>>>>>>> "parent-first". In my case it was it was in java.lang.ClassValue, but it
>>>>>>>>> might (and probably will be) different in your case. The problem is that if
>>>>>>>>> user-code registers something in some (static) storage located in class
>>>>>>>>> loaded with parent (TaskTracker) classloader, then its associated classes
>>>>>>>>> will never be GC'd and Metaspace will grow. A good starting point would be
>>>>>>>>> not to focus on biggest consumers of heap (in general), but to look at
>>>>>>>>> where the 15k objects of type Class are referenced from. That might help
>>>>>>>>> you figure this out. I'm not sure if there is something that can be done in
>>>>>>>>> general to prevent this type of leaks. That would be probably question on
>>>>>>>>> dev@ mailing list.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>>
>>>>>>>>> Jan
>>>>>>>>> On 11/16/20 2:27 PM, Flavio Pompermaier wrote:
>>>>>>>>>
>>>>>>>>> Hello everybody,
>>>>>>>>> I was writing this email when a similar thread on this mailing
>>>>>>>>> list appeared..
>>>>>>>>> The difference is that the other problem seems to be related
>>>>>>>>> with Flink 1.10 on YARN and does not output anything helpful in debugging
>>>>>>>>> the cause of the problem.
>>>>>>>>>
>>>>>>>>> Indeed, in my use case I use Flink 1.11.0 and Flink on a
>>>>>>>>> standalone session cluster (the job is submitted to the cluster using the
>>>>>>>>> CLI client).
>>>>>>>>> The problem arises when I submit the same job for about 20 times
>>>>>>>>> (this number unfortunately is not deterministic and can change a little
>>>>>>>>> bit). The error reported by the Task Executor is related to the ever
>>>>>>>>> growing Metaspace..the error seems to be pretty detailed [1].
>>>>>>>>>
>>>>>>>>> I found the same issue in some previous threads on this mailing
>>>>>>>>> list and I've tried to figure it out the cause of the problem. The issue is
>>>>>>>>> that looking at the objects allocated I don't really get an idea of the
>>>>>>>>> source of the problem because the type of objects that are consuming the
>>>>>>>>> memory are of general purpose (i.e. Bytes, Integers and Strings)...these
>>>>>>>>> are my "top" memory consumers if looking at the output of jmap -histo
>>>>>>>>> <PID>:
>>>>>>>>>
>>>>>>>>> At run 0:
>>>>>>>>>
>>>>>>>>> num #instances #bytes class name (module)
>>>>>>>>> -------------------------------------------------------
>>>>>>>>> 1: 46238 13224056 [B (java.base@11.0.9.1)
>>>>>>>>> 2: 3736 6536672 [I (java.base@11.0.9.1)
>>>>>>>>> 3: 38081 913944 java.lang.String (
>>>>>>>>> java.base@11.0.9.1)
>>>>>>>>> 4: 26 852384
>>>>>>>>> [Lakka.dispatch.forkjoin.ForkJoinTask;
>>>>>>>>> 5: 7146 844984 java.lang.Class (
>>>>>>>>> java.base@11.0.9.1)
>>>>>>>>>
>>>>>>>>> At run 1:
>>>>>>>>>
>>>>>>>>> 1: 77.608 25.317.496 [B (java.base@11.0.9.1)
>>>>>>>>> 2: 7.004 9.088.360 [I (java.base@11.0.9.1)
>>>>>>>>> 3: 15.814 1.887.256 java.lang.Class (
>>>>>>>>> java.base@11.0.9.1)
>>>>>>>>> 4: 67.381 1.617.144 java.lang.String (
>>>>>>>>> java.base@11.0.9.1)
>>>>>>>>> 5: 3.906 1.422.960 [Ljava.util.HashMap$Node; (
>>>>>>>>> java.base@11.0.9.1)
>>>>>>>>>
>>>>>>>>> At run 6:
>>>>>>>>>
>>>>>>>>> 1: 81.408 25.375.400 [B (java.base@11.0.9.1)
>>>>>>>>> 2: 12.479 7.249.392 [I (java.base@11.0.9.1)
>>>>>>>>> 3: 29.090 3.496.168 java.lang.Class (
>>>>>>>>> java.base@11.0.9.1)
>>>>>>>>> 4: 4.347 2.813.416 [Ljava.util.HashMap$Node; (
>>>>>>>>> java.base@11.0.9.1)
>>>>>>>>> 5: 71.584 1.718.016 java.lang.String (
>>>>>>>>> java.base@11.0.9.1)
>>>>>>>>>
>>>>>>>>> At run 8:
>>>>>>>>>
>>>>>>>>> 1: 985.979 127.193.256 [B (java.base@11.0.9.1)
>>>>>>>>> 2: 35.400 13.702.112 [I (java.base@11.0.9.1)
>>>>>>>>> 3: 260.387 6.249.288 java.lang.String (
>>>>>>>>> java.base@11.0.9.1)
>>>>>>>>> 4: 148.836 5.953.440
>>>>>>>>> java.util.HashMap$KeyIterator (java.base@11.0.9.1)
>>>>>>>>> 5: 17.641 5.222.344 [Ljava.util.HashMap$Node; (
>>>>>>>>> java.base@11.0.9.1)
>>>>>>>>>
>>>>>>>>> Thanks in advance for any help,
>>>>>>>>> Flavio
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> --------------------------------------------------------------------------------------------------
>>>>>>>>> java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory
>>>>>>>>> error has occurred. This can mean two things: either the job requires a
>>>>>>>>> larger size of JVM metaspace to load classes or there is a class loading
>>>>>>>>> leak. In the first case 'taskmanager.memory.jvm-metaspace.size'
>>>>>>>>> configuration option should be increased. If the error persists (usually in
>>>>>>>>> cluster after several job (re-)submissions) then there is probably a class
>>>>>>>>> loading leak in user code or some of its dependencies which has to be
>>>>>>>>> investigated and fixed. The task executor has to be shutdown...
>>>>>>>>> at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
>>>>>>>>> at
>>>>>>>>> java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
>>>>>>>>> at
>>>>>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
>>>>>>>>> ~[?:?]
>>>>>>>>> at
>>>>>>>>> java.net.URLClassLoader.defineClass(URLClassLoader.java:550) ~[?:?]
>>>>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:458)
>>>>>>>>> ~[?:?]
>>>>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:452)
>>>>>>>>> ~[?:?]
>>>>>>>>> at java.security.AccessController.doPrivileged(Native
>>>>>>>>> Method) ~[?:?]
>>>>>>>>> at
>>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:451) ~[?:?]
>>>>>>>>> at
>>>>>>>>> org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71)
>>>>>>>>> ~[flink-dist_2.12-1.11.0.jar:1.11.0]
>>>>>>>>> at
>>>>>>>>> org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
>>>>>>>>> [flink-dist_2.12-1.11.0.jar:1.11.0]
>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:522)
>>>>>>>>> [?:?]
>>>>>>>>>
>>>>>>>>> ------------------------------
>>>>>>
>>>>>> The information contained in this e-mail is confidential and/or
>>>>>> proprietary to Capital One and/or its affiliates and may only be used
>>>>>> solely in performance of work or services for Capital One. The information
>>>>>> transmitted herewith is intended only for use by the individual or entity
>>>>>> to which it is addressed. If the reader of this message is not the intended
>>>>>> recipient, you are hereby notified that any review, retransmission,
>>>>>> dissemination, distribution, copying or other use of, or taking of any
>>>>>> action in reliance upon this information is strictly prohibited. If you
>>>>>> have received this communication in error, please contact the sender and
>>>>>> delete the material from your computer.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> ------------------------------
>>>
>>> The information contained in this e-mail is confidential and/or
>>> proprietary to Capital One and/or its affiliates and may only be used
>>> solely in performance of work or services for Capital One. The information
>>> transmitted herewith is intended only for use by the individual or entity
>>> to which it is addressed. If the reader of this message is not the intended
>>> recipient, you are hereby notified that any review, retransmission,
>>> dissemination, distribution, copying or other use of, or taking of any
>>> action in reliance upon this information is strictly prohibited. If you
>>> have received this communication in error, please contact the sender and
>>> delete the material from your computer.
>>>
>>>
>>>
>>> ------------------------------
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
>
>
>
Re: [External Sender] Re: Random Task executor shutdown
(java.lang.OutOfMemoryError: Metaspace)
Posted by Kye Bae <ky...@capitalone.com>.
I see:
( Metadata: )
( reserved=122880KB, committed=121168KB)
( used=117529KB)
( free=3639KB)
( waste=0KB =0.00%)
Internal (reserved=855KB, committed=855KB)
(malloc=823KB #2442)
(mmap: reserved=32KB, committed=32KB)
and
0x00000008405a5288 0x000000080036f648 0x00007f1d14005160 2547 15675392
15603040 org.apache.flink.util.ChildFirstClassLoader
0x00000008405a5288 0x000000080036f648 0x00007f1ccc059790 2648 16330752
16287960 org.apache.flink.util.ChildFirstClassLoader
0x00000008405a5288 0x000000080036f648 0x00007f1c90004d90 2626 16461824
16391336 org.apache.flink.util.ChildFirstClassLoader
0x00000008405a5288 0x000000080036f648 0x00007f1c38685650 2648 16330752
16286608 org.apache.flink.util.ChildFirstClassLoader
0x00000008405a5288 0x000000080036f648 0x00007f1c3801e590 2651 16408576
16320192 org.apache.flink.util.ChildFirstClassLoader
The "Metadata" looks like there is a little over 3.5MB left. Not sure if
this is enough. Also, "Internal" memory reserved is less than 1 MB, but
this is where the Java direct memory buffers get allocated (Java NIO).
Don't know the details of your Flink job, but is this enough?
Flink 1.10 memory architecture requires care, and there should be enough
for both Flink and JVM internal operations. It shouldn't hurt to review
your memory requirements and allocate extra to account for unexpected
spikes.
Please keep in mind Yarn will kill a container immediately if the physical
memory limit is breached.
Also, it looks like ChildFirstClassLoader (user-code loader - dynamic) was
used 5 times to load classes onto this taskmanager. Were there 5 job
submissions to your standalone cluster or 4 job restarts for the same job?
Additional (dynamic) loadings will likely result in exceeding the max
metaspace limit.
If your Flink cluster is meant to host a single job, I would avoid using
the dynamic classloading altogether and then recapture the stats to see if
there are still multiple loading of the user (your application) classes. If
it doesn't, you might want to make that change to not use the dynamic
classloading.
-K
On Tue, Nov 17, 2020 at 2:00 PM Flavio Pompermaier <po...@okkam.it>
wrote:
> Hi Kye, thanks a lot for the support! See my answers inline below
>
> On Tue, Nov 17, 2020 at 7:22 PM Kye Bae <ky...@capitalone.com> wrote:
>
>> It is possible, but I am not entirely sure about the load order affecting
>> the metaspace usage.
>>
>> To find out why your taskmanager container is exceeding the metaspace, we
>> would need to know what value the max metaspace size is set to and then
>> find out how much of the metaspace is actually being used and what the
>> classloading stats are (before Yarn kills the container).
>>
>> If possible, please share the current metaspace setting and the native
>> memory stats from one of your taskmanager instances (Java process as a Yarn
>> container). I think those run as the yarn user as opposed to the hadoop
>> user (it depends on your environment).
>>
>>
> Actually I don't use YARN, I use standalone cluster and MaxMetspaceSize is
> automatically computed and it is set to 256 MB
> (-XX:MaxMetaspaceSize=268435456)
>
> # enable the native memory tracking referenced earlier - this may need to
>> be passed in as the taskmanager JVM option parameter not the jobmanager.
>> # log onto one of the taskmanager HW instances
>> sudo su - yarn # or log on as the user that runs Yarn
>> jps -v # grab the PID for one of the processes named YarnTaskExecutorRunner
>> - this would be a taskmanager
>> jcmd TM_PID VM.native_memory summary # size information for "Class" -
>> metaspace
>>
>
> jcmd 6976 VM.native_memory summary
> 6976:
>
> Native Memory Tracking:
>
> Total: reserved=1473311KB, committed=954707KB
> - Java Heap (reserved=524288KB, committed=524288KB)
> (mmap: reserved=524288KB, committed=524288KB)
>
> - Class (reserved=379640KB, committed=142024KB)
> (classes #23605)
> ( instance classes #22058, array classes
> #1547)
> (malloc=2808KB #48655)
> (mmap: reserved=376832KB, committed=139216KB)
> ( Metadata: )
> ( reserved=122880KB, committed=121168KB)
> ( used=117529KB)
> ( free=3639KB)
> ( waste=0KB =0.00%)
> ( Class space:)
> ( reserved=253952KB, committed=18048KB)
> ( used=16516KB)
> ( free=1532KB)
> ( waste=0KB =0.00%)
>
> - Thread (reserved=76359KB, committed=6867KB)
> (thread #74)
> (stack: reserved=76012KB, committed=6520KB)
> (malloc=262KB #446)
> (arena=85KB #146)
>
> - Code (reserved=250407KB, committed=38911KB)
> (malloc=2719KB #11989)
> (mmap: reserved=247688KB, committed=36192KB)
>
> - GC (reserved=71481KB, committed=71481KB)
> (malloc=19121KB #17323)
> (mmap: reserved=52360KB, committed=52360KB)
>
> - Compiler (reserved=561KB, committed=561KB)
> (malloc=429KB #846)
> (arena=133KB #5)
>
> - Internal (reserved=855KB, committed=855KB)
> (malloc=823KB #2442)
> (mmap: reserved=32KB, committed=32KB)
>
> - Other (reserved=131857KB, committed=131857KB)
> (malloc=131857KB #4128)
>
> - Symbol (reserved=15702KB, committed=15702KB)
> (malloc=14288KB #165850)
> (arena=1415KB #1)
>
> - Native Memory Tracking (reserved=4072KB, committed=4072KB)
> (malloc=27KB #353)
> (tracking overhead=4045KB)
>
> - Shared class space (reserved=17044KB, committed=17044KB)
> (mmap: reserved=17044KB, committed=17044KB)
>
> - Arena Chunk (reserved=199KB, committed=199KB)
> (malloc=199KB)
>
> - Logging (reserved=4KB, committed=4KB)
> (malloc=4KB #191)
>
> - Arguments (reserved=20KB, committed=20KB)
> (malloc=20KB #501)
>
> - Module (reserved=568KB, committed=568KB)
> (malloc=568KB #3919)
>
> - Synchronizer (reserved=244KB, committed=244KB)
> (malloc=244KB #2054)
>
> - Safepoint (reserved=8KB, committed=8KB)
> (mmap: reserved=8KB, committed=8KB)
>
>
>
>> jcmd TM_PID VM.classloader_stats # how many classes were loaded by which
>> classloader, sizes, etc.
>>
>
> jcmd 6976 VM.classloader_stats
> 6976:
> ClassLoader Parent CLD* Classes
> ChunkSz BlockSz Type
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d8140 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c38193d50 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00a9f10 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ab290 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c38011150 1
> 6144 4488 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b5d90 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9004dec0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380ae230 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c6402a2f0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c5803b430 1
> 4096 3232 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1d1c072ce0 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c800b7ee0 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc336610 1
> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8084aa0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cc0011ba0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c2f70 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90021090 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc8085cf0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d3830 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00becf0 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812a7c0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc000c880 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc018030 1
> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8139a90 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90020cd0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9806f6c0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c9804c2d0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80bcf80 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc347ec0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc01aca0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38576590 1
> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380f71f0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc40110d0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813c030 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c2440 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c8003c3a0 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9816bb00 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c900cee70 1
> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c98053d30 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80b5a00 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e29b0 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90259870 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8053250 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c60057820 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c3867f350 1
> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1d58d765c0 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cc4014c00 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e2f90 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ae450 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc014ae0 1
> 10240 7392 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813e880 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036ef80 0x00007f1c380851e0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb811f7f0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb811f080 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc0629c0 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc8064b40 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc14e8a0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813ab20 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c64026060 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc00078a0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc805dc50 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80b45f0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c3867eeb0 1
> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc0008590 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c98050230 1
> 6144 4488 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00cbfa0 1
> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8135800 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38060c30 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c0e20 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b0800 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c640399e0 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00cd350 1
> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b3dc0 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c38089ee0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c98053b00 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c301da250 1
> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc80f1050 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc014bf0 1
> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c902a1a60 1
> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c100bc340 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c304d0370 1
> 6144 4464 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812bf60 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c64051870 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38094a60 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38068320 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80b84d0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1ca4037920 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8046890 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c30000dc0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00cbdd0 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380c0e40 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00dfae0 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90258ee0 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c9804f6f0 1
> 6144 4488 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90045b60 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c3808b700 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8065550 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8060390 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b5e40 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80e5cf0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc807f2c0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc805d8b0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c800adb40 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8081720 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc400d760 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c30326b80 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc804a0c0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9029f4a0 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90259b80 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc00048c0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00af830 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c8550 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb811e7a0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3808e490 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8129480 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x00000008405a5288 0x000000080036f648 0x00007f1d14005160 2547
> 15675392 15603040 org.apache.flink.util.ChildFirstClassLoader
> 61
> 172032 106152 + unsafe anonymous classes
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c380ea520 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c3857a720 1
> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c7930 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cac004710 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00df0f0 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c3861a300 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80c6120 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c980d50c0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc00020d0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c64026190 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc009490 1
> 6144 4488 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d5510 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x0000559cae5496b0 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38081400 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d7110 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc8052210 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c64052a10 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc008aa0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c1930 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8134380 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812d770 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc337240 1
> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c58016fd0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8140140 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc017460 1
> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c98059040 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c38674d70 1
> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38013c50 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c30072cb0 1
> 6144 4032 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc0005930 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80b4ae0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812fb30 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38032ba0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc0166a0 1
> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000b9f0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80b8ca0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b48d0 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cf00647f0 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8053f00 1
> 6144 4488 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1d58d73980 1
> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080036f648 0x000000080036ef80 0x00007f1d5822a1b0 4979
> 34353152 34252632 jdk.internal.loader.ClassLoaders$AppClassLoader
> 564
> 2703360 1908736 + unsafe anonymous classes
> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc804fc70 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00bf800 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c6402f6c0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3808efa0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00cc960 1
> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380b0820 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c6003d940 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c3863da40 1
> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8135b60 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38078d80 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c902a0060 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x00000008405a5288 0x000000080036f648 0x00007f1ccc059790 2648
> 16330752 16287960 org.apache.flink.util.ChildFirstClassLoader
> 61
> 175104 109552 + unsafe anonymous classes
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc4013060 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00cb480 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c7b60 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c3054d770 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x00000008405a8e68 0x000000080036f648 0x00007f1c60007a50 1
> 6144 3024 sun.reflect.misc.MethodUtil
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00de270 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1ca0028160 1
> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c381d6b10 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d4c80 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38077ab0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c5660 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b8010 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1d58f59c10 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cac010bb0 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00bd6c0 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b53e0 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc400db60 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1d1404e940 1
> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc0147d0 1
> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b3430 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c98093270 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380bc7c0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cb8116dc0 1
> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1d58d8aa50 1
> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e49d0 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000f340 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cb8113720 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c980d21d0 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c88072820 1
> 6144 4024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3808f610 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d67e0 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc80883a0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc0134f0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3868db90 1
> 6144 4520 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c303249f0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3807aff0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc80e08d0 1
> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1d1c045720 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c902475f0 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000fbb0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e2cd0 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb81297a0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00abd60 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8134860 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc000a6c0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c303246d0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38099a70 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812f150 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000c610 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c900072d0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cd4052fc0 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380140b0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90257e30 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cd402bd50 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38015370 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1d5857f7a0 1
> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813b5d0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8136870 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb811a8c0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cf402a5a0 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c9804ebf0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c980aecb0 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80c4330 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc80b7850 1
> 6144 4488 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8084d80 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc017d20 1
> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc804c7a0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c68003140 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1ca4033900 1
> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1d14042ba0 1
> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8046210 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90006f20 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1d5857f230 1
> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d7750 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ac830 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38088e70 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d0350 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb81378e0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c30325540 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1d5a281630 1
> 6144 4768 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813cbd0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c8860 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c3808a920 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c380899f0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc803f350 1
> 6144 4488 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c6d90 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380db5b0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d2130 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00adbc0 1
> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb81312a0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc000ad00 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ca2c0 1
> 6144 4464 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00be210 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00bb5b0 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c386785e0 1
> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cf0064e50 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38035d90 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e2e60 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d36c0 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc0002210 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c385c6f00 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e53c0 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c3a80 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x0000000000000000 0x0000000000000000 0x00007f1d58223a40 3735
> 16637952 16601232 <boot class loader>
> 364
> 1014784 658032 + unsafe anonymous classes
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90033cd0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b6a40 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80a1520 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1d58f38e10 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d0a00 1
> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8120170 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1d1c04fb30 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c100c9b10 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb81205e0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8122f30 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000d170 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c60045bc0 1
> 6144 3992 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc04a8f0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1d58d8ce60 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e3930 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb81346e0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38079f60 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ad8c0 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380bc900 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc011de0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c9804e7f0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c7580 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3009d730 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a8e68 0x00007f1cc0011e00 1
> 3072 2176 jdk.internal.reflect.DelegatingClassLoader
> 0x00000008405a5288 0x000000080036f648 0x00007f1c90004d90 2626
> 16461824 16391336 org.apache.flink.util.ChildFirstClassLoader
> 61
> 177152 111624 + unsafe anonymous classes
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c0310 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc05d1f0 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c380e8400 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8130f80 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c9820e410 1
> 6144 3848 jdk.internal.reflect.DelegatingClassLoader
> 0x00000008405a5288 0x000000080036f648 0x00007f1c38685650 2648
> 16330752 16286608 org.apache.flink.util.ChildFirstClassLoader
> 61
> 175104 109552 + unsafe anonymous classes
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8042c50 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb81406e0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90034980 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000bd60 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80839f0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c38083d50 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90257460 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3803d6f0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c98074170 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380bb950 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38096730 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d9bd0 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c780016c0 1
> 4096 3160 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc80e27c0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38087f00 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38086150 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e3fe0 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1d1c047930 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c303d77a0 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cd402bf60 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38063580 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9816bdf0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9004e8c0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c5980 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c4570 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812b740 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc00074b0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380c1560 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8134b70 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ad5b0 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00bcbd0 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc4016dc0 1
> 3072 2192 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d99c0 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b7500 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380ad230 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38051d00 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8137a20 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80ba3d0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c980700f0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c386121b0 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ae770 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9029fe30 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x00000008405a5288 0x000000080036f648 0x00007f1c3801e590 2651
> 16408576 16320192 org.apache.flink.util.ChildFirstClassLoader
> 61
> 175104 109552 + unsafe anonymous classes
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ac470 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc0009c00 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c88061f30 1
> 6144 4024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c580154c0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3807be20 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8139df0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c5bd0 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8142d80 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8137710 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c60056030 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380c1990 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c902a0420 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1d58d74a40 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38061f80 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc80e05a0 1
> 4096 3160 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00bc0c0 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c981c06d0 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc046f80 1
> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c30328250 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc804a340 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c60057560 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c304fdc50 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812d450 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c900222f0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cc001e850 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc00b6b0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc335880 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c38138510 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c3857d030 1
> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c38086290 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e04d0 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c380870c0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e0830 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c980d5490 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1d58d75010 1
> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc33bb70 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c900333a0 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3807cbb0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc019db0 1
> 6144 4760 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc000c510 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc000ba60 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8140870 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813a170 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8122d40 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c600092b0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c381d7610 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80aa1c0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc00062f0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8131f90 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1c880739f0 1
> 6144 4024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d7420 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b1310 1
> 3072 2024 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3c20f0b0 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc01b6b0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc02fd70 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb811ac10 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1ccc062900 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80aa4c0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc80bb720 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d9610 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8121e80 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c30327500 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80e1ad0 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080036ef80 0x0000000000000000 0x00007f1d582879b0 112
> 843776 783312 jdk.internal.loader.ClassLoaders$PlatformClassLoader
> 3
> 9216 5432 + unsafe anonymous classes
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8080100 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc80873d0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc00b300 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc2e7b90 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813c350 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c90021200 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3c210130 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc8044950 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1ca4031cc0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1d58d74ec0 1
> 6144 3960 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc8081b00 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1cc80ba9e0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1ccc011840 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00da8d0 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000d5a0 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e57b0 1
> 6144 4048 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812e180 1
> 6144 3952 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x0000000000000000 0x00007f1cc8047410 1
> 6144 4008 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x000000080036f648 0x00007f1c3807a130 1
> 6144 3872 jdk.internal.reflect.DelegatingClassLoader
> 0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8146e70 1
> 6144 3928 jdk.internal.reflect.DelegatingClassLoader
> Total = 431 23605
> 140124160 137261400
> ChunkSz: Total size of all allocated metaspace chunks
> BlockSz: Total size of all allocated metaspace blocks (each chunk has
> several blocks)
>
>
>>
>> There is a gentleman who posts blogs on Flink, and he goes into a bit
>> more detail there. This was against Flink 1.9, but the foundational
>> concepts still apply to 1.10.
>> http://cloudsqale.com/2020/04/29/flink-1-9-off-heap-memory-on-yarn-troubleshooting-container-is-running-beyond-physical-memory-limits-errors/
>> <https://urldefense.com/v3/__http://cloudsqale.com/2020/04/29/flink-1-9-off-heap-memory-on-yarn-troubleshooting-container-is-running-beyond-physical-memory-limits-errors/__;!!EFVe01R3CjU!NKfuq7CJ3Ok0zh_7rBJ8PMx-Wo_Yh2_f993EG1lS-hJ4lbtgJtRRUH_EoOhvmEMAkQ$>
>>
>> -K
>>
>> On Tue, Nov 17, 2020 at 12:47 PM Flavio Pompermaier <po...@okkam.it>
>> wrote:
>>
>>> Another big potential candidate is the fact that JDBC libs I use in my
>>> job are put into the Flink lib folder instead of putting them into the fat
>>> jar..tomorrow I'll try to see if the metaspace is getting cleared correctly
>>> after that change.
>>> Unfortunately our jobs were written before the child-first /
>>> parent-first classloading refactoring and at that time that was the way to
>>> go..but now it can cause this kind of problems if using child-first policy.
>>>
>>> On Mon, Nov 16, 2020 at 8:44 PM Flavio Pompermaier <po...@okkam.it>
>>> wrote:
>>>
>>>> Thank you Kye for your insights...in my mind, if the job runs without
>>>> problems one or more times the heap size, and thus the medatadata-size, is
>>>> big enough and I should not increase it (on the same data of course).
>>>> So I'll try to understand who is leaking what..the advice to avoid the
>>>> dynamic class loading is just a workaround to me..there's something wrong
>>>> going on and tomorrow I'll try to understand the root cause of the
>>>> problem using -XX:NativeMemoryTracking=summary as you suggested.
>>>>
>>>> I'll keep you up to date with my findings..
>>>>
>>>> Best,
>>>> Flavio
>>>>
>>>> On Mon, Nov 16, 2020 at 8:22 PM Kye Bae <ky...@capitalone.com> wrote:
>>>>
>>>>> Hello!
>>>>>
>>>>> The JVM metaspace is where all the classes (not class instances or
>>>>> objects) get loaded. jmap -histo is going to show you the heap space usage
>>>>> info not the metaspace.
>>>>>
>>>>> You could inspect what is happening in the metaspace by using jcmd (e.g.,
>>>>> jcmd JPID VM.native_memory summary) after restarting the cluster with
>>>>> "*-XX:NativeMemoryTracking=summary"*
>>>>>
>>>>> *As the error message suggests, you may need to increase *taskmanager.memory.jvm-metaspace.size,
>>>>> but you need to be slightly careful when specifying the memory parameters
>>>>> in flink-conf.yaml in Flink 1.10 (they have an issue with a confusing error
>>>>> message).
>>>>>
>>>>> Anothing thing to keep in mind is that you may want to avoid using
>>>>> dynamic classloading (
>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/debugging_classloading.html#avoiding-dynamic-classloading-for-user-code
>>>>> <https://urldefense.com/v3/__https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/debugging_classloading.html*avoiding-dynamic-classloading-for-user-code__;Iw!!EFVe01R3CjU!L48mrpYG-_Tcldpp26edL4MKDU3RIHFmX89E-BzmJk6_RiLUqQUFvy4RHE5kTFVDig$>):
>>>>> when the job continuously fails for some temporary issues, it will load the
>>>>> same class files into the metaspace multiple times and could exceed
>>>>> whatever the limit you set it.
>>>>>
>>>>> -K
>>>>>
>>>>> On Mon, Nov 16, 2020 at 12:39 PM Jan Lukavský <je...@seznam.cz> wrote:
>>>>>
>>>>>> The exclusions should not have any impact on that, because what
>>>>>> defines which classloader will load which class is not the presence or
>>>>>> particular class in a specific jar, but the configuration of
>>>>>> parent-first-patterns [1].
>>>>>>
>>>>>> If you don't use any flink internal imports, than it still might be
>>>>>> the case, that a class in any of the packages defined by the
>>>>>> parent-first-pattern to hold reference to your user-code classes, which
>>>>>> would cause the leak. I'd recommend to inspect the heap dump after several
>>>>>> restarts of the application and look for reference to Class objects from
>>>>>> the root set.
>>>>>>
>>>>>> Jan
>>>>>>
>>>>>> [1]
>>>>>> https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#class-loading
>>>>>> <https://urldefense.com/v3/__https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html*class-loading__;Iw!!EFVe01R3CjU!NF2uHM8m-1kZSP7E3-7ZhdKcQa2U3wTqWKDA8zSI4727gH8ASTFc4h9qBaX4_W2wJA$>
>>>>>> On 11/16/20 5:34 PM, Flavio Pompermaier wrote:
>>>>>>
>>>>>> I've tried to remove all possible imports of classes not contained in
>>>>>> the fat jar but I still face the same problem.
>>>>>> I've also tried to reduce as much as possible the exclude in the
>>>>>> shade section of the maven plugin (I took the one at [1]) so now I exclude
>>>>>> only few dependencies..could it be that I should include org.slf4j:* if I
>>>>>> use static import of it?
>>>>>>
>>>>>> <artifactSet>
>>>>>> <excludes>
>>>>>> <exclude>com.google.code.findbugs:jsr305</exclude>
>>>>>> <exclude>org.slf4j:*</exclude>
>>>>>> <exclude>log4j:*</exclude>
>>>>>> </excludes>
>>>>>> </artifactSet>
>>>>>>
>>>>>> [1]
>>>>>> https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html#appendix-template-for-building-a-jar-with-dependencies
>>>>>> <https://urldefense.com/v3/__https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html*appendix-template-for-building-a-jar-with-dependencies__;Iw!!EFVe01R3CjU!NF2uHM8m-1kZSP7E3-7ZhdKcQa2U3wTqWKDA8zSI4727gH8ASTFc4h9qBaWGhZYoqQ$>
>>>>>>
>>>>>> On Mon, Nov 16, 2020 at 3:29 PM Jan Lukavský <je...@seznam.cz> wrote:
>>>>>>
>>>>>>> Yes, that could definitely cause this. You should probably avoid
>>>>>>> using these flink-internal shaded classes and ship your own versions (not
>>>>>>> shaded).
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> Jan
>>>>>>> On 11/16/20 3:22 PM, Flavio Pompermaier wrote:
>>>>>>>
>>>>>>> Thank you Jan for your valuable feedback.
>>>>>>> Could it be that I should not use import shaded-jackson classes in
>>>>>>> my user code?
>>>>>>> For example import
>>>>>>> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper?
>>>>>>>
>>>>>>> Bets,
>>>>>>> Flavio
>>>>>>>
>>>>>>> On Mon, Nov 16, 2020 at 3:15 PM Jan Lukavský <je...@seznam.cz>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Flavio,
>>>>>>>>
>>>>>>>> when I encountered quite similar problem that you describe, it was
>>>>>>>> related to a static storage located in class that was loaded
>>>>>>>> "parent-first". In my case it was it was in java.lang.ClassValue, but it
>>>>>>>> might (and probably will be) different in your case. The problem is that if
>>>>>>>> user-code registers something in some (static) storage located in class
>>>>>>>> loaded with parent (TaskTracker) classloader, then its associated classes
>>>>>>>> will never be GC'd and Metaspace will grow. A good starting point would be
>>>>>>>> not to focus on biggest consumers of heap (in general), but to look at
>>>>>>>> where the 15k objects of type Class are referenced from. That might help
>>>>>>>> you figure this out. I'm not sure if there is something that can be done in
>>>>>>>> general to prevent this type of leaks. That would be probably question on
>>>>>>>> dev@ mailing list.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>>
>>>>>>>> Jan
>>>>>>>> On 11/16/20 2:27 PM, Flavio Pompermaier wrote:
>>>>>>>>
>>>>>>>> Hello everybody,
>>>>>>>> I was writing this email when a similar thread on this mailing list
>>>>>>>> appeared..
>>>>>>>> The difference is that the other problem seems to be related
>>>>>>>> with Flink 1.10 on YARN and does not output anything helpful in debugging
>>>>>>>> the cause of the problem.
>>>>>>>>
>>>>>>>> Indeed, in my use case I use Flink 1.11.0 and Flink on a standalone
>>>>>>>> session cluster (the job is submitted to the cluster using the CLI client).
>>>>>>>> The problem arises when I submit the same job for about 20 times
>>>>>>>> (this number unfortunately is not deterministic and can change a little
>>>>>>>> bit). The error reported by the Task Executor is related to the ever
>>>>>>>> growing Metaspace..the error seems to be pretty detailed [1].
>>>>>>>>
>>>>>>>> I found the same issue in some previous threads on this mailing
>>>>>>>> list and I've tried to figure it out the cause of the problem. The issue is
>>>>>>>> that looking at the objects allocated I don't really get an idea of the
>>>>>>>> source of the problem because the type of objects that are consuming the
>>>>>>>> memory are of general purpose (i.e. Bytes, Integers and Strings)...these
>>>>>>>> are my "top" memory consumers if looking at the output of jmap -histo
>>>>>>>> <PID>:
>>>>>>>>
>>>>>>>> At run 0:
>>>>>>>>
>>>>>>>> num #instances #bytes class name (module)
>>>>>>>> -------------------------------------------------------
>>>>>>>> 1: 46238 13224056 [B (java.base@11.0.9.1)
>>>>>>>> 2: 3736 6536672 [I (java.base@11.0.9.1)
>>>>>>>> 3: 38081 913944 java.lang.String (
>>>>>>>> java.base@11.0.9.1)
>>>>>>>> 4: 26 852384
>>>>>>>> [Lakka.dispatch.forkjoin.ForkJoinTask;
>>>>>>>> 5: 7146 844984 java.lang.Class (
>>>>>>>> java.base@11.0.9.1)
>>>>>>>>
>>>>>>>> At run 1:
>>>>>>>>
>>>>>>>> 1: 77.608 25.317.496 [B (java.base@11.0.9.1)
>>>>>>>> 2: 7.004 9.088.360 [I (java.base@11.0.9.1)
>>>>>>>> 3: 15.814 1.887.256 java.lang.Class (
>>>>>>>> java.base@11.0.9.1)
>>>>>>>> 4: 67.381 1.617.144 java.lang.String (
>>>>>>>> java.base@11.0.9.1)
>>>>>>>> 5: 3.906 1.422.960 [Ljava.util.HashMap$Node; (
>>>>>>>> java.base@11.0.9.1)
>>>>>>>>
>>>>>>>> At run 6:
>>>>>>>>
>>>>>>>> 1: 81.408 25.375.400 [B (java.base@11.0.9.1)
>>>>>>>> 2: 12.479 7.249.392 [I (java.base@11.0.9.1)
>>>>>>>> 3: 29.090 3.496.168 java.lang.Class (
>>>>>>>> java.base@11.0.9.1)
>>>>>>>> 4: 4.347 2.813.416 [Ljava.util.HashMap$Node; (
>>>>>>>> java.base@11.0.9.1)
>>>>>>>> 5: 71.584 1.718.016 java.lang.String (
>>>>>>>> java.base@11.0.9.1)
>>>>>>>>
>>>>>>>> At run 8:
>>>>>>>>
>>>>>>>> 1: 985.979 127.193.256 [B (java.base@11.0.9.1)
>>>>>>>> 2: 35.400 13.702.112 [I (java.base@11.0.9.1)
>>>>>>>> 3: 260.387 6.249.288 java.lang.String (
>>>>>>>> java.base@11.0.9.1)
>>>>>>>> 4: 148.836 5.953.440
>>>>>>>> java.util.HashMap$KeyIterator (java.base@11.0.9.1)
>>>>>>>> 5: 17.641 5.222.344 [Ljava.util.HashMap$Node; (
>>>>>>>> java.base@11.0.9.1)
>>>>>>>>
>>>>>>>> Thanks in advance for any help,
>>>>>>>> Flavio
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> --------------------------------------------------------------------------------------------------
>>>>>>>> java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory
>>>>>>>> error has occurred. This can mean two things: either the job requires a
>>>>>>>> larger size of JVM metaspace to load classes or there is a class loading
>>>>>>>> leak. In the first case 'taskmanager.memory.jvm-metaspace.size'
>>>>>>>> configuration option should be increased. If the error persists (usually in
>>>>>>>> cluster after several job (re-)submissions) then there is probably a class
>>>>>>>> loading leak in user code or some of its dependencies which has to be
>>>>>>>> investigated and fixed. The task executor has to be shutdown...
>>>>>>>> at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
>>>>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
>>>>>>>> ~[?:?]
>>>>>>>> at
>>>>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
>>>>>>>> ~[?:?]
>>>>>>>> at
>>>>>>>> java.net.URLClassLoader.defineClass(URLClassLoader.java:550) ~[?:?]
>>>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:458)
>>>>>>>> ~[?:?]
>>>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:452)
>>>>>>>> ~[?:?]
>>>>>>>> at java.security.AccessController.doPrivileged(Native
>>>>>>>> Method) ~[?:?]
>>>>>>>> at
>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:451) ~[?:?]
>>>>>>>> at
>>>>>>>> org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71)
>>>>>>>> ~[flink-dist_2.12-1.11.0.jar:1.11.0]
>>>>>>>> at
>>>>>>>> org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
>>>>>>>> [flink-dist_2.12-1.11.0.jar:1.11.0]
>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:522)
>>>>>>>> [?:?]
>>>>>>>>
>>>>>>>> ------------------------------
>>>>>
>>>>> The information contained in this e-mail is confidential and/or
>>>>> proprietary to Capital One and/or its affiliates and may only be used
>>>>> solely in performance of work or services for Capital One. The information
>>>>> transmitted herewith is intended only for use by the individual or entity
>>>>> to which it is addressed. If the reader of this message is not the intended
>>>>> recipient, you are hereby notified that any review, retransmission,
>>>>> dissemination, distribution, copying or other use of, or taking of any
>>>>> action in reliance upon this information is strictly prohibited. If you
>>>>> have received this communication in error, please contact the sender and
>>>>> delete the material from your computer.
>>>>>
>>>>>
>>>>>
>>>>>
>>> ------------------------------
>>
>> The information contained in this e-mail is confidential and/or
>> proprietary to Capital One and/or its affiliates and may only be used
>> solely in performance of work or services for Capital One. The information
>> transmitted herewith is intended only for use by the individual or entity
>> to which it is addressed. If the reader of this message is not the intended
>> recipient, you are hereby notified that any review, retransmission,
>> dissemination, distribution, copying or other use of, or taking of any
>> action in reliance upon this information is strictly prohibited. If you
>> have received this communication in error, please contact the sender and
>> delete the material from your computer.
>>
>>
>>
>>
______________________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: [External Sender] Re: Random Task executor shutdown
(java.lang.OutOfMemoryError: Metaspace)
Posted by Flavio Pompermaier <po...@okkam.it>.
Hi Kye, thanks a lot for the support! See my answers inline below
On Tue, Nov 17, 2020 at 7:22 PM Kye Bae <ky...@capitalone.com> wrote:
> It is possible, but I am not entirely sure about the load order affecting
> the metaspace usage.
>
> To find out why your taskmanager container is exceeding the metaspace, we
> would need to know what value the max metaspace size is set to and then
> find out how much of the metaspace is actually being used and what the
> classloading stats are (before Yarn kills the container).
>
> If possible, please share the current metaspace setting and the native
> memory stats from one of your taskmanager instances (Java process as a Yarn
> container). I think those run as the yarn user as opposed to the hadoop
> user (it depends on your environment).
>
>
Actually I don't use YARN, I use standalone cluster and MaxMetspaceSize is
automatically computed and it is set to 256 MB
(-XX:MaxMetaspaceSize=268435456)
# enable the native memory tracking referenced earlier - this may need to
> be passed in as the taskmanager JVM option parameter not the jobmanager.
> # log onto one of the taskmanager HW instances
> sudo su - yarn # or log on as the user that runs Yarn
> jps -v # grab the PID for one of the processes named YarnTaskExecutorRunner
> - this would be a taskmanager
> jcmd TM_PID VM.native_memory summary # size information for "Class" -
> metaspace
>
jcmd 6976 VM.native_memory summary
6976:
Native Memory Tracking:
Total: reserved=1473311KB, committed=954707KB
- Java Heap (reserved=524288KB, committed=524288KB)
(mmap: reserved=524288KB, committed=524288KB)
- Class (reserved=379640KB, committed=142024KB)
(classes #23605)
( instance classes #22058, array classes #1547)
(malloc=2808KB #48655)
(mmap: reserved=376832KB, committed=139216KB)
( Metadata: )
( reserved=122880KB, committed=121168KB)
( used=117529KB)
( free=3639KB)
( waste=0KB =0.00%)
( Class space:)
( reserved=253952KB, committed=18048KB)
( used=16516KB)
( free=1532KB)
( waste=0KB =0.00%)
- Thread (reserved=76359KB, committed=6867KB)
(thread #74)
(stack: reserved=76012KB, committed=6520KB)
(malloc=262KB #446)
(arena=85KB #146)
- Code (reserved=250407KB, committed=38911KB)
(malloc=2719KB #11989)
(mmap: reserved=247688KB, committed=36192KB)
- GC (reserved=71481KB, committed=71481KB)
(malloc=19121KB #17323)
(mmap: reserved=52360KB, committed=52360KB)
- Compiler (reserved=561KB, committed=561KB)
(malloc=429KB #846)
(arena=133KB #5)
- Internal (reserved=855KB, committed=855KB)
(malloc=823KB #2442)
(mmap: reserved=32KB, committed=32KB)
- Other (reserved=131857KB, committed=131857KB)
(malloc=131857KB #4128)
- Symbol (reserved=15702KB, committed=15702KB)
(malloc=14288KB #165850)
(arena=1415KB #1)
- Native Memory Tracking (reserved=4072KB, committed=4072KB)
(malloc=27KB #353)
(tracking overhead=4045KB)
- Shared class space (reserved=17044KB, committed=17044KB)
(mmap: reserved=17044KB, committed=17044KB)
- Arena Chunk (reserved=199KB, committed=199KB)
(malloc=199KB)
- Logging (reserved=4KB, committed=4KB)
(malloc=4KB #191)
- Arguments (reserved=20KB, committed=20KB)
(malloc=20KB #501)
- Module (reserved=568KB, committed=568KB)
(malloc=568KB #3919)
- Synchronizer (reserved=244KB, committed=244KB)
(malloc=244KB #2054)
- Safepoint (reserved=8KB, committed=8KB)
(mmap: reserved=8KB, committed=8KB)
> jcmd TM_PID VM.classloader_stats # how many classes were loaded by which
> classloader, sizes, etc.
>
jcmd 6976 VM.classloader_stats
6976:
ClassLoader Parent CLD* Classes
ChunkSz BlockSz Type
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d8140 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c38193d50 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00a9f10 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ab290 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c38011150 1
6144 4488 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b5d90 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c9004dec0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c380ae230 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c6402a2f0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c5803b430 1
4096 3232 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1d1c072ce0 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c800b7ee0 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc336610 1
6144 3960 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc8084aa0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cc0011ba0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c2f70 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c90021090 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1cc8085cf0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d3830 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00becf0 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812a7c0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1cc000c880 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1ccc018030 1
6144 4760 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8139a90 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c90020cd0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c9806f6c0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c9804c2d0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc80bcf80 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc347ec0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1ccc01aca0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c38576590 1
6144 3960 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c380f71f0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc40110d0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813c030 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c2440 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c8003c3a0 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c9816bb00 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c900cee70 1
6144 3960 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c98053d30 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc80b5a00 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e29b0 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c90259870 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc8053250 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c60057820 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c3867f350 1
6144 3960 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1d58d765c0 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cc4014c00 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e2f90 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ae450 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1ccc014ae0 1
10240 7392 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813e880 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036ef80 0x00007f1c380851e0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb811f7f0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb811f080 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1ccc0629c0 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1cc8064b40 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc14e8a0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813ab20 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c64026060 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1cc00078a0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc805dc50 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc80b45f0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c3867eeb0 1
6144 3960 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1cc0008590 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c98050230 1
6144 4488 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00cbfa0 1
6144 4760 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8135800 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c38060c30 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c0e20 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b0800 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c640399e0 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00cd350 1
6144 4760 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b3dc0 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c38089ee0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c98053b00 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c301da250 1
6144 4760 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1cc80f1050 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1ccc014bf0 1
6144 4760 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c902a1a60 1
6144 3960 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c100bc340 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c304d0370 1
6144 4464 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812bf60 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c64051870 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c38094a60 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c38068320 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc80b84d0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1ca4037920 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc8046890 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c30000dc0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00cbdd0 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c380c0e40 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00dfae0 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c90258ee0 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c9804f6f0 1
6144 4488 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c90045b60 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c3808b700 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc8065550 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc8060390 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b5e40 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc80e5cf0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc807f2c0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc805d8b0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c800adb40 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc8081720 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc400d760 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c30326b80 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1cc804a0c0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c9029f4a0 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c90259b80 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1cc00048c0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00af830 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c8550 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb811e7a0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c3808e490 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8129480 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x00000008405a5288 0x000000080036f648 0x00007f1d14005160 2547
15675392 15603040 org.apache.flink.util.ChildFirstClassLoader
61
172032 106152 + unsafe anonymous classes
0x000000080039f8b0 0x0000000000000000 0x00007f1c380ea520 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c3857a720 1
6144 3960 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c7930 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cac004710 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00df0f0 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c3861a300 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc80c6120 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c980d50c0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1cc00020d0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c64026190 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1ccc009490 1
6144 4488 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d5510 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x0000559cae5496b0 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c38081400 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d7110 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1cc8052210 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c64052a10 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1ccc008aa0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c1930 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8134380 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812d770 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc337240 1
6144 3960 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c58016fd0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8140140 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1ccc017460 1
6144 4760 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c98059040 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c38674d70 1
6144 3960 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c38013c50 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c30072cb0 1
6144 4032 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1cc0005930 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc80b4ae0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812fb30 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c38032ba0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1ccc0166a0 1
6144 4760 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000b9f0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc80b8ca0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b48d0 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cf00647f0 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc8053f00 1
6144 4488 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1d58d73980 1
6144 3960 jdk.internal.reflect.DelegatingClassLoader
0x000000080036f648 0x000000080036ef80 0x00007f1d5822a1b0 4979
34353152 34252632 jdk.internal.loader.ClassLoaders$AppClassLoader
564
2703360 1908736 + unsafe anonymous classes
0x000000080039f8b0 0x0000000000000000 0x00007f1cc804fc70 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00bf800 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c6402f6c0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c3808efa0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00cc960 1
6144 4760 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c380b0820 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c6003d940 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c3863da40 1
6144 3960 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8135b60 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c38078d80 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c902a0060 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x00000008405a5288 0x000000080036f648 0x00007f1ccc059790 2648
16330752 16287960 org.apache.flink.util.ChildFirstClassLoader
61
175104 109552 + unsafe anonymous classes
0x000000080039f8b0 0x000000080036f648 0x00007f1cc4013060 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00cb480 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c7b60 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c3054d770 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x00000008405a8e68 0x000000080036f648 0x00007f1c60007a50 1
6144 3024 sun.reflect.misc.MethodUtil
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00de270 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1ca0028160 1
6144 3960 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c381d6b10 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d4c80 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c38077ab0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c5660 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b8010 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1d58f59c10 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cac010bb0 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00bd6c0 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b53e0 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc400db60 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1d1404e940 1
6144 3960 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1ccc0147d0 1
6144 4760 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b3430 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c98093270 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c380bc7c0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cb8116dc0 1
6144 4760 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1d58d8aa50 1
6144 3960 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e49d0 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000f340 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cb8113720 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c980d21d0 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c88072820 1
6144 4024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c3808f610 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d67e0 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1cc80883a0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1ccc0134f0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c3868db90 1
6144 4520 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c303249f0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c3807aff0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1cc80e08d0 1
6144 4760 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1d1c045720 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c902475f0 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000fbb0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e2cd0 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb81297a0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00abd60 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8134860 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc000a6c0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c303246d0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c38099a70 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812f150 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000c610 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c900072d0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cd4052fc0 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c380140b0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c90257e30 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cd402bd50 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c38015370 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1d5857f7a0 1
6144 3960 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813b5d0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8136870 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb811a8c0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cf402a5a0 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c9804ebf0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c980aecb0 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc80c4330 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1cc80b7850 1
6144 4488 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc8084d80 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1ccc017d20 1
6144 4760 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc804c7a0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c68003140 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1ca4033900 1
6144 4760 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1d14042ba0 1
6144 3960 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc8046210 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c90006f20 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1d5857f230 1
6144 3960 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d7750 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ac830 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c38088e70 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d0350 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb81378e0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c30325540 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1d5a281630 1
6144 4768 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813cbd0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c8860 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c3808a920 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c380899f0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc803f350 1
6144 4488 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c6d90 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c380db5b0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d2130 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00adbc0 1
6144 4760 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb81312a0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1cc000ad00 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ca2c0 1
6144 4464 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00be210 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00bb5b0 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c386785e0 1
6144 3960 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cf0064e50 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c38035d90 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e2e60 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d36c0 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc0002210 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c385c6f00 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e53c0 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c3a80 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x0000000000000000 0x0000000000000000 0x00007f1d58223a40 3735
16637952 16601232 <boot class loader>
364
1014784 658032 + unsafe anonymous classes
0x000000080039f8b0 0x00000008405a5288 0x00007f1c90033cd0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b6a40 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc80a1520 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1d58f38e10 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d0a00 1
6144 4760 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8120170 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1d1c04fb30 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c100c9b10 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb81205e0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8122f30 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000d170 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c60045bc0 1
6144 3992 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc04a8f0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1d58d8ce60 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e3930 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb81346e0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c38079f60 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ad8c0 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c380bc900 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1ccc011de0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c9804e7f0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c7580 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c3009d730 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a8e68 0x00007f1cc0011e00 1
3072 2176 jdk.internal.reflect.DelegatingClassLoader
0x00000008405a5288 0x000000080036f648 0x00007f1c90004d90 2626
16461824 16391336 org.apache.flink.util.ChildFirstClassLoader
61
177152 111624 + unsafe anonymous classes
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c0310 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1ccc05d1f0 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c380e8400 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8130f80 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c9820e410 1
6144 3848 jdk.internal.reflect.DelegatingClassLoader
0x00000008405a5288 0x000000080036f648 0x00007f1c38685650 2648
16330752 16286608 org.apache.flink.util.ChildFirstClassLoader
61
175104 109552 + unsafe anonymous classes
0x000000080039f8b0 0x000000080036f648 0x00007f1cc8042c50 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb81406e0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c90034980 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000bd60 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc80839f0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c38083d50 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c90257460 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c3803d6f0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c98074170 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c380bb950 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c38096730 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d9bd0 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c780016c0 1
4096 3160 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1cc80e27c0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c38087f00 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c38086150 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e3fe0 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1d1c047930 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c303d77a0 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cd402bf60 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c38063580 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c9816bdf0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c9004e8c0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c5980 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c4570 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812b740 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1cc00074b0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c380c1560 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8134b70 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ad5b0 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00bcbd0 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc4016dc0 1
3072 2192 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d99c0 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b7500 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c380ad230 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c38051d00 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8137a20 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc80ba3d0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c980700f0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c386121b0 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ae770 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c9029fe30 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x00000008405a5288 0x000000080036f648 0x00007f1c3801e590 2651
16408576 16320192 org.apache.flink.util.ChildFirstClassLoader
61
175104 109552 + unsafe anonymous classes
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00ac470 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1cc0009c00 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c88061f30 1
6144 4024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c580154c0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c3807be20 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8139df0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00c5bd0 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8142d80 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8137710 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c60056030 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c380c1990 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c902a0420 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1d58d74a40 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c38061f80 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1cc80e05a0 1
4096 3160 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00bc0c0 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c981c06d0 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1ccc046f80 1
6144 3960 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c30328250 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1cc804a340 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c60057560 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c304fdc50 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812d450 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c900222f0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cc001e850 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1ccc00b6b0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc335880 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c38138510 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c3857d030 1
6144 3960 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c38086290 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e04d0 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c380870c0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e0830 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c980d5490 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1d58d75010 1
6144 3960 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc33bb70 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c900333a0 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c3807cbb0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1ccc019db0 1
6144 4760 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1cc000c510 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc000ba60 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8140870 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813a170 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8122d40 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c600092b0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c381d7610 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc80aa1c0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1cc00062f0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8131f90 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1c880739f0 1
6144 4024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d7420 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00b1310 1
3072 2024 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c3c20f0b0 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1ccc01b6b0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc02fd70 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb811ac10 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1ccc062900 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc80aa4c0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1cc80bb720 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00d9610 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8121e80 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c30327500 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc80e1ad0 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080036ef80 0x0000000000000000 0x00007f1d582879b0 112
843776 783312 jdk.internal.loader.ClassLoaders$PlatformClassLoader
3
9216 5432 + unsafe anonymous classes
0x000000080039f8b0 0x000000080036f648 0x00007f1cc8080100 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1cc80873d0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1ccc00b300 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1ccc2e7b90 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb813c350 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c90021200 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c3c210130 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1cc8044950 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1ca4031cc0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1d58d74ec0 1
6144 3960 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc8081b00 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1cc80ba9e0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1ccc011840 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00da8d0 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1c9000d5a0 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1bb00e57b0 1
6144 4048 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb812e180 1
6144 3952 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x0000000000000000 0x00007f1cc8047410 1
6144 4008 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x000000080036f648 0x00007f1c3807a130 1
6144 3872 jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0 0x00000008405a5288 0x00007f1cb8146e70 1
6144 3928 jdk.internal.reflect.DelegatingClassLoader
Total = 431 23605
140124160 137261400
ChunkSz: Total size of all allocated metaspace chunks
BlockSz: Total size of all allocated metaspace blocks (each chunk has
several blocks)
>
> There is a gentleman who posts blogs on Flink, and he goes into a bit more
> detail there. This was against Flink 1.9, but the foundational concepts
> still apply to 1.10.
> http://cloudsqale.com/2020/04/29/flink-1-9-off-heap-memory-on-yarn-troubleshooting-container-is-running-beyond-physical-memory-limits-errors/
>
> -K
>
> On Tue, Nov 17, 2020 at 12:47 PM Flavio Pompermaier <po...@okkam.it>
> wrote:
>
>> Another big potential candidate is the fact that JDBC libs I use in my
>> job are put into the Flink lib folder instead of putting them into the fat
>> jar..tomorrow I'll try to see if the metaspace is getting cleared correctly
>> after that change.
>> Unfortunately our jobs were written before the child-first / parent-first
>> classloading refactoring and at that time that was the way to go..but now
>> it can cause this kind of problems if using child-first policy.
>>
>> On Mon, Nov 16, 2020 at 8:44 PM Flavio Pompermaier <po...@okkam.it>
>> wrote:
>>
>>> Thank you Kye for your insights...in my mind, if the job runs without
>>> problems one or more times the heap size, and thus the medatadata-size, is
>>> big enough and I should not increase it (on the same data of course).
>>> So I'll try to understand who is leaking what..the advice to avoid the
>>> dynamic class loading is just a workaround to me..there's something wrong
>>> going on and tomorrow I'll try to understand the root cause of the
>>> problem using -XX:NativeMemoryTracking=summary as you suggested.
>>>
>>> I'll keep you up to date with my findings..
>>>
>>> Best,
>>> Flavio
>>>
>>> On Mon, Nov 16, 2020 at 8:22 PM Kye Bae <ky...@capitalone.com> wrote:
>>>
>>>> Hello!
>>>>
>>>> The JVM metaspace is where all the classes (not class instances or
>>>> objects) get loaded. jmap -histo is going to show you the heap space usage
>>>> info not the metaspace.
>>>>
>>>> You could inspect what is happening in the metaspace by using jcmd (e.g.,
>>>> jcmd JPID VM.native_memory summary) after restarting the cluster with "
>>>> *-XX:NativeMemoryTracking=summary"*
>>>>
>>>> *As the error message suggests, you may need to increase *taskmanager.memory.jvm-metaspace.size,
>>>> but you need to be slightly careful when specifying the memory parameters
>>>> in flink-conf.yaml in Flink 1.10 (they have an issue with a confusing error
>>>> message).
>>>>
>>>> Anothing thing to keep in mind is that you may want to avoid using
>>>> dynamic classloading (
>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/debugging_classloading.html#avoiding-dynamic-classloading-for-user-code
>>>> <https://urldefense.com/v3/__https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/debugging_classloading.html*avoiding-dynamic-classloading-for-user-code__;Iw!!EFVe01R3CjU!L48mrpYG-_Tcldpp26edL4MKDU3RIHFmX89E-BzmJk6_RiLUqQUFvy4RHE5kTFVDig$>):
>>>> when the job continuously fails for some temporary issues, it will load the
>>>> same class files into the metaspace multiple times and could exceed
>>>> whatever the limit you set it.
>>>>
>>>> -K
>>>>
>>>> On Mon, Nov 16, 2020 at 12:39 PM Jan Lukavský <je...@seznam.cz> wrote:
>>>>
>>>>> The exclusions should not have any impact on that, because what
>>>>> defines which classloader will load which class is not the presence or
>>>>> particular class in a specific jar, but the configuration of
>>>>> parent-first-patterns [1].
>>>>>
>>>>> If you don't use any flink internal imports, than it still might be
>>>>> the case, that a class in any of the packages defined by the
>>>>> parent-first-pattern to hold reference to your user-code classes, which
>>>>> would cause the leak. I'd recommend to inspect the heap dump after several
>>>>> restarts of the application and look for reference to Class objects from
>>>>> the root set.
>>>>>
>>>>> Jan
>>>>>
>>>>> [1]
>>>>> https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#class-loading
>>>>> <https://urldefense.com/v3/__https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html*class-loading__;Iw!!EFVe01R3CjU!NF2uHM8m-1kZSP7E3-7ZhdKcQa2U3wTqWKDA8zSI4727gH8ASTFc4h9qBaX4_W2wJA$>
>>>>> On 11/16/20 5:34 PM, Flavio Pompermaier wrote:
>>>>>
>>>>> I've tried to remove all possible imports of classes not contained in
>>>>> the fat jar but I still face the same problem.
>>>>> I've also tried to reduce as much as possible the exclude in the shade
>>>>> section of the maven plugin (I took the one at [1]) so now I exclude only
>>>>> few dependencies..could it be that I should include org.slf4j:* if I use
>>>>> static import of it?
>>>>>
>>>>> <artifactSet>
>>>>> <excludes>
>>>>> <exclude>com.google.code.findbugs:jsr305</exclude>
>>>>> <exclude>org.slf4j:*</exclude>
>>>>> <exclude>log4j:*</exclude>
>>>>> </excludes>
>>>>> </artifactSet>
>>>>>
>>>>> [1]
>>>>> https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html#appendix-template-for-building-a-jar-with-dependencies
>>>>> <https://urldefense.com/v3/__https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html*appendix-template-for-building-a-jar-with-dependencies__;Iw!!EFVe01R3CjU!NF2uHM8m-1kZSP7E3-7ZhdKcQa2U3wTqWKDA8zSI4727gH8ASTFc4h9qBaWGhZYoqQ$>
>>>>>
>>>>> On Mon, Nov 16, 2020 at 3:29 PM Jan Lukavský <je...@seznam.cz> wrote:
>>>>>
>>>>>> Yes, that could definitely cause this. You should probably avoid
>>>>>> using these flink-internal shaded classes and ship your own versions (not
>>>>>> shaded).
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>> Jan
>>>>>> On 11/16/20 3:22 PM, Flavio Pompermaier wrote:
>>>>>>
>>>>>> Thank you Jan for your valuable feedback.
>>>>>> Could it be that I should not use import shaded-jackson classes in my
>>>>>> user code?
>>>>>> For example import
>>>>>> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper?
>>>>>>
>>>>>> Bets,
>>>>>> Flavio
>>>>>>
>>>>>> On Mon, Nov 16, 2020 at 3:15 PM Jan Lukavský <je...@seznam.cz> wrote:
>>>>>>
>>>>>>> Hi Flavio,
>>>>>>>
>>>>>>> when I encountered quite similar problem that you describe, it was
>>>>>>> related to a static storage located in class that was loaded
>>>>>>> "parent-first". In my case it was it was in java.lang.ClassValue, but it
>>>>>>> might (and probably will be) different in your case. The problem is that if
>>>>>>> user-code registers something in some (static) storage located in class
>>>>>>> loaded with parent (TaskTracker) classloader, then its associated classes
>>>>>>> will never be GC'd and Metaspace will grow. A good starting point would be
>>>>>>> not to focus on biggest consumers of heap (in general), but to look at
>>>>>>> where the 15k objects of type Class are referenced from. That might help
>>>>>>> you figure this out. I'm not sure if there is something that can be done in
>>>>>>> general to prevent this type of leaks. That would be probably question on
>>>>>>> dev@ mailing list.
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> Jan
>>>>>>> On 11/16/20 2:27 PM, Flavio Pompermaier wrote:
>>>>>>>
>>>>>>> Hello everybody,
>>>>>>> I was writing this email when a similar thread on this mailing list
>>>>>>> appeared..
>>>>>>> The difference is that the other problem seems to be related
>>>>>>> with Flink 1.10 on YARN and does not output anything helpful in debugging
>>>>>>> the cause of the problem.
>>>>>>>
>>>>>>> Indeed, in my use case I use Flink 1.11.0 and Flink on a standalone
>>>>>>> session cluster (the job is submitted to the cluster using the CLI client).
>>>>>>> The problem arises when I submit the same job for about 20 times
>>>>>>> (this number unfortunately is not deterministic and can change a little
>>>>>>> bit). The error reported by the Task Executor is related to the ever
>>>>>>> growing Metaspace..the error seems to be pretty detailed [1].
>>>>>>>
>>>>>>> I found the same issue in some previous threads on this mailing list
>>>>>>> and I've tried to figure it out the cause of the problem. The issue is that
>>>>>>> looking at the objects allocated I don't really get an idea of the source
>>>>>>> of the problem because the type of objects that are consuming the memory
>>>>>>> are of general purpose (i.e. Bytes, Integers and Strings)...these are my
>>>>>>> "top" memory consumers if looking at the output of jmap -histo <PID>:
>>>>>>>
>>>>>>> At run 0:
>>>>>>>
>>>>>>> num #instances #bytes class name (module)
>>>>>>> -------------------------------------------------------
>>>>>>> 1: 46238 13224056 [B (java.base@11.0.9.1)
>>>>>>> 2: 3736 6536672 [I (java.base@11.0.9.1)
>>>>>>> 3: 38081 913944 java.lang.String (
>>>>>>> java.base@11.0.9.1)
>>>>>>> 4: 26 852384
>>>>>>> [Lakka.dispatch.forkjoin.ForkJoinTask;
>>>>>>> 5: 7146 844984 java.lang.Class (
>>>>>>> java.base@11.0.9.1)
>>>>>>>
>>>>>>> At run 1:
>>>>>>>
>>>>>>> 1: 77.608 25.317.496 [B (java.base@11.0.9.1)
>>>>>>> 2: 7.004 9.088.360 [I (java.base@11.0.9.1)
>>>>>>> 3: 15.814 1.887.256 java.lang.Class (
>>>>>>> java.base@11.0.9.1)
>>>>>>> 4: 67.381 1.617.144 java.lang.String (
>>>>>>> java.base@11.0.9.1)
>>>>>>> 5: 3.906 1.422.960 [Ljava.util.HashMap$Node; (
>>>>>>> java.base@11.0.9.1)
>>>>>>>
>>>>>>> At run 6:
>>>>>>>
>>>>>>> 1: 81.408 25.375.400 [B (java.base@11.0.9.1)
>>>>>>> 2: 12.479 7.249.392 [I (java.base@11.0.9.1)
>>>>>>> 3: 29.090 3.496.168 java.lang.Class (
>>>>>>> java.base@11.0.9.1)
>>>>>>> 4: 4.347 2.813.416 [Ljava.util.HashMap$Node; (
>>>>>>> java.base@11.0.9.1)
>>>>>>> 5: 71.584 1.718.016 java.lang.String (
>>>>>>> java.base@11.0.9.1)
>>>>>>>
>>>>>>> At run 8:
>>>>>>>
>>>>>>> 1: 985.979 127.193.256 [B (java.base@11.0.9.1)
>>>>>>> 2: 35.400 13.702.112 [I (java.base@11.0.9.1)
>>>>>>> 3: 260.387 6.249.288 java.lang.String (
>>>>>>> java.base@11.0.9.1)
>>>>>>> 4: 148.836 5.953.440 java.util.HashMap$KeyIterator
>>>>>>> (java.base@11.0.9.1)
>>>>>>> 5: 17.641 5.222.344 [Ljava.util.HashMap$Node; (
>>>>>>> java.base@11.0.9.1)
>>>>>>>
>>>>>>> Thanks in advance for any help,
>>>>>>> Flavio
>>>>>>>
>>>>>>> [1]
>>>>>>> --------------------------------------------------------------------------------------------------
>>>>>>> java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory
>>>>>>> error has occurred. This can mean two things: either the job requires a
>>>>>>> larger size of JVM metaspace to load classes or there is a class loading
>>>>>>> leak. In the first case 'taskmanager.memory.jvm-metaspace.size'
>>>>>>> configuration option should be increased. If the error persists (usually in
>>>>>>> cluster after several job (re-)submissions) then there is probably a class
>>>>>>> loading leak in user code or some of its dependencies which has to be
>>>>>>> investigated and fixed. The task executor has to be shutdown...
>>>>>>> at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
>>>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
>>>>>>> ~[?:?]
>>>>>>> at
>>>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
>>>>>>> ~[?:?]
>>>>>>> at
>>>>>>> java.net.URLClassLoader.defineClass(URLClassLoader.java:550) ~[?:?]
>>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:458)
>>>>>>> ~[?:?]
>>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:452)
>>>>>>> ~[?:?]
>>>>>>> at java.security.AccessController.doPrivileged(Native
>>>>>>> Method) ~[?:?]
>>>>>>> at
>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:451) ~[?:?]
>>>>>>> at
>>>>>>> org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71)
>>>>>>> ~[flink-dist_2.12-1.11.0.jar:1.11.0]
>>>>>>> at
>>>>>>> org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
>>>>>>> [flink-dist_2.12-1.11.0.jar:1.11.0]
>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:522)
>>>>>>> [?:?]
>>>>>>>
>>>>>>> ------------------------------
>>>>
>>>> The information contained in this e-mail is confidential and/or
>>>> proprietary to Capital One and/or its affiliates and may only be used
>>>> solely in performance of work or services for Capital One. The information
>>>> transmitted herewith is intended only for use by the individual or entity
>>>> to which it is addressed. If the reader of this message is not the intended
>>>> recipient, you are hereby notified that any review, retransmission,
>>>> dissemination, distribution, copying or other use of, or taking of any
>>>> action in reliance upon this information is strictly prohibited. If you
>>>> have received this communication in error, please contact the sender and
>>>> delete the material from your computer.
>>>>
>>>>
>>>>
>>>>
>> ------------------------------
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
>
>
>
Re: [External Sender] Re: Random Task executor shutdown
(java.lang.OutOfMemoryError: Metaspace)
Posted by Kye Bae <ky...@capitalone.com>.
It is possible, but I am not entirely sure about the load order affecting
the metaspace usage.
To find out why your taskmanager container is exceeding the metaspace, we
would need to know what value the max metaspace size is set to and then
find out how much of the metaspace is actually being used and what the
classloading stats are (before Yarn kills the container).
If possible, please share the current metaspace setting and the native
memory stats from one of your taskmanager instances (Java process as a Yarn
container). I think those run as the yarn user as opposed to the hadoop
user (it depends on your environment).
# enable the native memory tracking referenced earlier - this may need to
be passed in as the taskmanager JVM option parameter not the jobmanager.
# log onto one of the taskmanager HW instances
sudo su - yarn # or log on as the user that runs Yarn
jps -v # grab the PID for one of the processes named YarnTaskExecutorRunner
- this would be a taskmanager
jcmd TM_PID VM.native_memory summary # size information for "Class" -
metaspace
jcmd TM_PID VM.classloader_stats # how many classes were loaded by which
classloader, sizes, etc.
There is a gentleman who posts blogs on Flink, and he goes into a bit more
detail there. This was against Flink 1.9, but the foundational concepts
still apply to 1.10.
http://cloudsqale.com/2020/04/29/flink-1-9-off-heap-memory-on-yarn-troubleshooting-container-is-running-beyond-physical-memory-limits-errors/
-K
On Tue, Nov 17, 2020 at 12:47 PM Flavio Pompermaier <po...@okkam.it>
wrote:
> Another big potential candidate is the fact that JDBC libs I use in my job
> are put into the Flink lib folder instead of putting them into the fat
> jar..tomorrow I'll try to see if the metaspace is getting cleared correctly
> after that change.
> Unfortunately our jobs were written before the child-first / parent-first
> classloading refactoring and at that time that was the way to go..but now
> it can cause this kind of problems if using child-first policy.
>
> On Mon, Nov 16, 2020 at 8:44 PM Flavio Pompermaier <po...@okkam.it>
> wrote:
>
>> Thank you Kye for your insights...in my mind, if the job runs without
>> problems one or more times the heap size, and thus the medatadata-size, is
>> big enough and I should not increase it (on the same data of course).
>> So I'll try to understand who is leaking what..the advice to avoid the
>> dynamic class loading is just a workaround to me..there's something wrong
>> going on and tomorrow I'll try to understand the root cause of the
>> problem using -XX:NativeMemoryTracking=summary as you suggested.
>>
>> I'll keep you up to date with my findings..
>>
>> Best,
>> Flavio
>>
>> On Mon, Nov 16, 2020 at 8:22 PM Kye Bae <ky...@capitalone.com> wrote:
>>
>>> Hello!
>>>
>>> The JVM metaspace is where all the classes (not class instances or
>>> objects) get loaded. jmap -histo is going to show you the heap space usage
>>> info not the metaspace.
>>>
>>> You could inspect what is happening in the metaspace by using jcmd (e.g.,
>>> jcmd JPID VM.native_memory summary) after restarting the cluster with "
>>> *-XX:NativeMemoryTracking=summary"*
>>>
>>> *As the error message suggests, you may need to increase *taskmanager.memory.jvm-metaspace.size,
>>> but you need to be slightly careful when specifying the memory parameters
>>> in flink-conf.yaml in Flink 1.10 (they have an issue with a confusing error
>>> message).
>>>
>>> Anothing thing to keep in mind is that you may want to avoid using
>>> dynamic classloading (
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/debugging_classloading.html#avoiding-dynamic-classloading-for-user-code
>>> <https://urldefense.com/v3/__https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/debugging_classloading.html*avoiding-dynamic-classloading-for-user-code__;Iw!!EFVe01R3CjU!L48mrpYG-_Tcldpp26edL4MKDU3RIHFmX89E-BzmJk6_RiLUqQUFvy4RHE5kTFVDig$>):
>>> when the job continuously fails for some temporary issues, it will load the
>>> same class files into the metaspace multiple times and could exceed
>>> whatever the limit you set it.
>>>
>>> -K
>>>
>>> On Mon, Nov 16, 2020 at 12:39 PM Jan Lukavský <je...@seznam.cz> wrote:
>>>
>>>> The exclusions should not have any impact on that, because what defines
>>>> which classloader will load which class is not the presence or particular
>>>> class in a specific jar, but the configuration of parent-first-patterns [1].
>>>>
>>>> If you don't use any flink internal imports, than it still might be the
>>>> case, that a class in any of the packages defined by the
>>>> parent-first-pattern to hold reference to your user-code classes, which
>>>> would cause the leak. I'd recommend to inspect the heap dump after several
>>>> restarts of the application and look for reference to Class objects from
>>>> the root set.
>>>>
>>>> Jan
>>>>
>>>> [1]
>>>> https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#class-loading
>>>> <https://urldefense.com/v3/__https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html*class-loading__;Iw!!EFVe01R3CjU!NF2uHM8m-1kZSP7E3-7ZhdKcQa2U3wTqWKDA8zSI4727gH8ASTFc4h9qBaX4_W2wJA$>
>>>> On 11/16/20 5:34 PM, Flavio Pompermaier wrote:
>>>>
>>>> I've tried to remove all possible imports of classes not contained in
>>>> the fat jar but I still face the same problem.
>>>> I've also tried to reduce as much as possible the exclude in the shade
>>>> section of the maven plugin (I took the one at [1]) so now I exclude only
>>>> few dependencies..could it be that I should include org.slf4j:* if I use
>>>> static import of it?
>>>>
>>>> <artifactSet>
>>>> <excludes>
>>>> <exclude>com.google.code.findbugs:jsr305</exclude>
>>>> <exclude>org.slf4j:*</exclude>
>>>> <exclude>log4j:*</exclude>
>>>> </excludes>
>>>> </artifactSet>
>>>>
>>>> [1]
>>>> https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html#appendix-template-for-building-a-jar-with-dependencies
>>>> <https://urldefense.com/v3/__https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html*appendix-template-for-building-a-jar-with-dependencies__;Iw!!EFVe01R3CjU!NF2uHM8m-1kZSP7E3-7ZhdKcQa2U3wTqWKDA8zSI4727gH8ASTFc4h9qBaWGhZYoqQ$>
>>>>
>>>> On Mon, Nov 16, 2020 at 3:29 PM Jan Lukavský <je...@seznam.cz> wrote:
>>>>
>>>>> Yes, that could definitely cause this. You should probably avoid using
>>>>> these flink-internal shaded classes and ship your own versions (not shaded).
>>>>>
>>>>> Best,
>>>>>
>>>>> Jan
>>>>> On 11/16/20 3:22 PM, Flavio Pompermaier wrote:
>>>>>
>>>>> Thank you Jan for your valuable feedback.
>>>>> Could it be that I should not use import shaded-jackson classes in my
>>>>> user code?
>>>>> For example import
>>>>> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper?
>>>>>
>>>>> Bets,
>>>>> Flavio
>>>>>
>>>>> On Mon, Nov 16, 2020 at 3:15 PM Jan Lukavský <je...@seznam.cz> wrote:
>>>>>
>>>>>> Hi Flavio,
>>>>>>
>>>>>> when I encountered quite similar problem that you describe, it was
>>>>>> related to a static storage located in class that was loaded
>>>>>> "parent-first". In my case it was it was in java.lang.ClassValue, but it
>>>>>> might (and probably will be) different in your case. The problem is that if
>>>>>> user-code registers something in some (static) storage located in class
>>>>>> loaded with parent (TaskTracker) classloader, then its associated classes
>>>>>> will never be GC'd and Metaspace will grow. A good starting point would be
>>>>>> not to focus on biggest consumers of heap (in general), but to look at
>>>>>> where the 15k objects of type Class are referenced from. That might help
>>>>>> you figure this out. I'm not sure if there is something that can be done in
>>>>>> general to prevent this type of leaks. That would be probably question on
>>>>>> dev@ mailing list.
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>> Jan
>>>>>> On 11/16/20 2:27 PM, Flavio Pompermaier wrote:
>>>>>>
>>>>>> Hello everybody,
>>>>>> I was writing this email when a similar thread on this mailing list
>>>>>> appeared..
>>>>>> The difference is that the other problem seems to be related
>>>>>> with Flink 1.10 on YARN and does not output anything helpful in debugging
>>>>>> the cause of the problem.
>>>>>>
>>>>>> Indeed, in my use case I use Flink 1.11.0 and Flink on a standalone
>>>>>> session cluster (the job is submitted to the cluster using the CLI client).
>>>>>> The problem arises when I submit the same job for about 20 times
>>>>>> (this number unfortunately is not deterministic and can change a little
>>>>>> bit). The error reported by the Task Executor is related to the ever
>>>>>> growing Metaspace..the error seems to be pretty detailed [1].
>>>>>>
>>>>>> I found the same issue in some previous threads on this mailing list
>>>>>> and I've tried to figure it out the cause of the problem. The issue is that
>>>>>> looking at the objects allocated I don't really get an idea of the source
>>>>>> of the problem because the type of objects that are consuming the memory
>>>>>> are of general purpose (i.e. Bytes, Integers and Strings)...these are my
>>>>>> "top" memory consumers if looking at the output of jmap -histo <PID>:
>>>>>>
>>>>>> At run 0:
>>>>>>
>>>>>> num #instances #bytes class name (module)
>>>>>> -------------------------------------------------------
>>>>>> 1: 46238 13224056 [B (java.base@11.0.9.1)
>>>>>> 2: 3736 6536672 [I (java.base@11.0.9.1)
>>>>>> 3: 38081 913944 java.lang.String (
>>>>>> java.base@11.0.9.1)
>>>>>> 4: 26 852384
>>>>>> [Lakka.dispatch.forkjoin.ForkJoinTask;
>>>>>> 5: 7146 844984 java.lang.Class (
>>>>>> java.base@11.0.9.1)
>>>>>>
>>>>>> At run 1:
>>>>>>
>>>>>> 1: 77.608 25.317.496 [B (java.base@11.0.9.1)
>>>>>> 2: 7.004 9.088.360 [I (java.base@11.0.9.1)
>>>>>> 3: 15.814 1.887.256 java.lang.Class (
>>>>>> java.base@11.0.9.1)
>>>>>> 4: 67.381 1.617.144 java.lang.String (
>>>>>> java.base@11.0.9.1)
>>>>>> 5: 3.906 1.422.960 [Ljava.util.HashMap$Node; (
>>>>>> java.base@11.0.9.1)
>>>>>>
>>>>>> At run 6:
>>>>>>
>>>>>> 1: 81.408 25.375.400 [B (java.base@11.0.9.1)
>>>>>> 2: 12.479 7.249.392 [I (java.base@11.0.9.1)
>>>>>> 3: 29.090 3.496.168 java.lang.Class (
>>>>>> java.base@11.0.9.1)
>>>>>> 4: 4.347 2.813.416 [Ljava.util.HashMap$Node; (
>>>>>> java.base@11.0.9.1)
>>>>>> 5: 71.584 1.718.016 java.lang.String (
>>>>>> java.base@11.0.9.1)
>>>>>>
>>>>>> At run 8:
>>>>>>
>>>>>> 1: 985.979 127.193.256 [B (java.base@11.0.9.1)
>>>>>> 2: 35.400 13.702.112 [I (java.base@11.0.9.1)
>>>>>> 3: 260.387 6.249.288 java.lang.String (
>>>>>> java.base@11.0.9.1)
>>>>>> 4: 148.836 5.953.440 java.util.HashMap$KeyIterator (
>>>>>> java.base@11.0.9.1)
>>>>>> 5: 17.641 5.222.344 [Ljava.util.HashMap$Node; (
>>>>>> java.base@11.0.9.1)
>>>>>>
>>>>>> Thanks in advance for any help,
>>>>>> Flavio
>>>>>>
>>>>>> [1]
>>>>>> --------------------------------------------------------------------------------------------------
>>>>>> java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory
>>>>>> error has occurred. This can mean two things: either the job requires a
>>>>>> larger size of JVM metaspace to load classes or there is a class loading
>>>>>> leak. In the first case 'taskmanager.memory.jvm-metaspace.size'
>>>>>> configuration option should be increased. If the error persists (usually in
>>>>>> cluster after several job (re-)submissions) then there is probably a class
>>>>>> loading leak in user code or some of its dependencies which has to be
>>>>>> investigated and fixed. The task executor has to be shutdown...
>>>>>> at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
>>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
>>>>>> ~[?:?]
>>>>>> at
>>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
>>>>>> ~[?:?]
>>>>>> at
>>>>>> java.net.URLClassLoader.defineClass(URLClassLoader.java:550) ~[?:?]
>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:458)
>>>>>> ~[?:?]
>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:452)
>>>>>> ~[?:?]
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> ~[?:?]
>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:451)
>>>>>> ~[?:?]
>>>>>> at
>>>>>> org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71)
>>>>>> ~[flink-dist_2.12-1.11.0.jar:1.11.0]
>>>>>> at
>>>>>> org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
>>>>>> [flink-dist_2.12-1.11.0.jar:1.11.0]
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:522) [?:?]
>>>>>>
>>>>>> ------------------------------
>>>
>>> The information contained in this e-mail is confidential and/or
>>> proprietary to Capital One and/or its affiliates and may only be used
>>> solely in performance of work or services for Capital One. The information
>>> transmitted herewith is intended only for use by the individual or entity
>>> to which it is addressed. If the reader of this message is not the intended
>>> recipient, you are hereby notified that any review, retransmission,
>>> dissemination, distribution, copying or other use of, or taking of any
>>> action in reliance upon this information is strictly prohibited. If you
>>> have received this communication in error, please contact the sender and
>>> delete the material from your computer.
>>>
>>>
>>>
>>>
>
______________________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: [External Sender] Re: Random Task executor shutdown
(java.lang.OutOfMemoryError: Metaspace)
Posted by Flavio Pompermaier <po...@okkam.it>.
Another big potential candidate is the fact that JDBC libs I use in my job
are put into the Flink lib folder instead of putting them into the fat
jar..tomorrow I'll try to see if the metaspace is getting cleared correctly
after that change.
Unfortunately our jobs were written before the child-first / parent-first
classloading refactoring and at that time that was the way to go..but now
it can cause this kind of problems if using child-first policy.
On Mon, Nov 16, 2020 at 8:44 PM Flavio Pompermaier <po...@okkam.it>
wrote:
> Thank you Kye for your insights...in my mind, if the job runs without
> problems one or more times the heap size, and thus the medatadata-size, is
> big enough and I should not increase it (on the same data of course).
> So I'll try to understand who is leaking what..the advice to avoid the
> dynamic class loading is just a workaround to me..there's something wrong
> going on and tomorrow I'll try to understand the root cause of the
> problem using -XX:NativeMemoryTracking=summary as you suggested.
>
> I'll keep you up to date with my findings..
>
> Best,
> Flavio
>
> On Mon, Nov 16, 2020 at 8:22 PM Kye Bae <ky...@capitalone.com> wrote:
>
>> Hello!
>>
>> The JVM metaspace is where all the classes (not class instances or
>> objects) get loaded. jmap -histo is going to show you the heap space usage
>> info not the metaspace.
>>
>> You could inspect what is happening in the metaspace by using jcmd (e.g.,
>> jcmd JPID VM.native_memory summary) after restarting the cluster with "
>> *-XX:NativeMemoryTracking=summary"*
>>
>> *As the error message suggests, you may need to increase *taskmanager.memory.jvm-metaspace.size,
>> but you need to be slightly careful when specifying the memory parameters
>> in flink-conf.yaml in Flink 1.10 (they have an issue with a confusing error
>> message).
>>
>> Anothing thing to keep in mind is that you may want to avoid using
>> dynamic classloading (
>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/debugging_classloading.html#avoiding-dynamic-classloading-for-user-code):
>> when the job continuously fails for some temporary issues, it will load the
>> same class files into the metaspace multiple times and could exceed
>> whatever the limit you set it.
>>
>> -K
>>
>> On Mon, Nov 16, 2020 at 12:39 PM Jan Lukavský <je...@seznam.cz> wrote:
>>
>>> The exclusions should not have any impact on that, because what defines
>>> which classloader will load which class is not the presence or particular
>>> class in a specific jar, but the configuration of parent-first-patterns [1].
>>>
>>> If you don't use any flink internal imports, than it still might be the
>>> case, that a class in any of the packages defined by the
>>> parent-first-pattern to hold reference to your user-code classes, which
>>> would cause the leak. I'd recommend to inspect the heap dump after several
>>> restarts of the application and look for reference to Class objects from
>>> the root set.
>>>
>>> Jan
>>>
>>> [1]
>>> https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#class-loading
>>> <https://urldefense.com/v3/__https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html*class-loading__;Iw!!EFVe01R3CjU!NF2uHM8m-1kZSP7E3-7ZhdKcQa2U3wTqWKDA8zSI4727gH8ASTFc4h9qBaX4_W2wJA$>
>>> On 11/16/20 5:34 PM, Flavio Pompermaier wrote:
>>>
>>> I've tried to remove all possible imports of classes not contained in
>>> the fat jar but I still face the same problem.
>>> I've also tried to reduce as much as possible the exclude in the shade
>>> section of the maven plugin (I took the one at [1]) so now I exclude only
>>> few dependencies..could it be that I should include org.slf4j:* if I use
>>> static import of it?
>>>
>>> <artifactSet>
>>> <excludes>
>>> <exclude>com.google.code.findbugs:jsr305</exclude>
>>> <exclude>org.slf4j:*</exclude>
>>> <exclude>log4j:*</exclude>
>>> </excludes>
>>> </artifactSet>
>>>
>>> [1]
>>> https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html#appendix-template-for-building-a-jar-with-dependencies
>>> <https://urldefense.com/v3/__https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html*appendix-template-for-building-a-jar-with-dependencies__;Iw!!EFVe01R3CjU!NF2uHM8m-1kZSP7E3-7ZhdKcQa2U3wTqWKDA8zSI4727gH8ASTFc4h9qBaWGhZYoqQ$>
>>>
>>> On Mon, Nov 16, 2020 at 3:29 PM Jan Lukavský <je...@seznam.cz> wrote:
>>>
>>>> Yes, that could definitely cause this. You should probably avoid using
>>>> these flink-internal shaded classes and ship your own versions (not shaded).
>>>>
>>>> Best,
>>>>
>>>> Jan
>>>> On 11/16/20 3:22 PM, Flavio Pompermaier wrote:
>>>>
>>>> Thank you Jan for your valuable feedback.
>>>> Could it be that I should not use import shaded-jackson classes in my
>>>> user code?
>>>> For example import
>>>> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper?
>>>>
>>>> Bets,
>>>> Flavio
>>>>
>>>> On Mon, Nov 16, 2020 at 3:15 PM Jan Lukavský <je...@seznam.cz> wrote:
>>>>
>>>>> Hi Flavio,
>>>>>
>>>>> when I encountered quite similar problem that you describe, it was
>>>>> related to a static storage located in class that was loaded
>>>>> "parent-first". In my case it was it was in java.lang.ClassValue, but it
>>>>> might (and probably will be) different in your case. The problem is that if
>>>>> user-code registers something in some (static) storage located in class
>>>>> loaded with parent (TaskTracker) classloader, then its associated classes
>>>>> will never be GC'd and Metaspace will grow. A good starting point would be
>>>>> not to focus on biggest consumers of heap (in general), but to look at
>>>>> where the 15k objects of type Class are referenced from. That might help
>>>>> you figure this out. I'm not sure if there is something that can be done in
>>>>> general to prevent this type of leaks. That would be probably question on
>>>>> dev@ mailing list.
>>>>>
>>>>> Best,
>>>>>
>>>>> Jan
>>>>> On 11/16/20 2:27 PM, Flavio Pompermaier wrote:
>>>>>
>>>>> Hello everybody,
>>>>> I was writing this email when a similar thread on this mailing list
>>>>> appeared..
>>>>> The difference is that the other problem seems to be related
>>>>> with Flink 1.10 on YARN and does not output anything helpful in debugging
>>>>> the cause of the problem.
>>>>>
>>>>> Indeed, in my use case I use Flink 1.11.0 and Flink on a standalone
>>>>> session cluster (the job is submitted to the cluster using the CLI client).
>>>>> The problem arises when I submit the same job for about 20 times (this
>>>>> number unfortunately is not deterministic and can change a little bit). The
>>>>> error reported by the Task Executor is related to the ever growing
>>>>> Metaspace..the error seems to be pretty detailed [1].
>>>>>
>>>>> I found the same issue in some previous threads on this mailing list
>>>>> and I've tried to figure it out the cause of the problem. The issue is that
>>>>> looking at the objects allocated I don't really get an idea of the source
>>>>> of the problem because the type of objects that are consuming the memory
>>>>> are of general purpose (i.e. Bytes, Integers and Strings)...these are my
>>>>> "top" memory consumers if looking at the output of jmap -histo <PID>:
>>>>>
>>>>> At run 0:
>>>>>
>>>>> num #instances #bytes class name (module)
>>>>> -------------------------------------------------------
>>>>> 1: 46238 13224056 [B (java.base@11.0.9.1)
>>>>> 2: 3736 6536672 [I (java.base@11.0.9.1)
>>>>> 3: 38081 913944 java.lang.String (
>>>>> java.base@11.0.9.1)
>>>>> 4: 26 852384
>>>>> [Lakka.dispatch.forkjoin.ForkJoinTask;
>>>>> 5: 7146 844984 java.lang.Class (
>>>>> java.base@11.0.9.1)
>>>>>
>>>>> At run 1:
>>>>>
>>>>> 1: 77.608 25.317.496 [B (java.base@11.0.9.1)
>>>>> 2: 7.004 9.088.360 [I (java.base@11.0.9.1)
>>>>> 3: 15.814 1.887.256 java.lang.Class (
>>>>> java.base@11.0.9.1)
>>>>> 4: 67.381 1.617.144 java.lang.String (
>>>>> java.base@11.0.9.1)
>>>>> 5: 3.906 1.422.960 [Ljava.util.HashMap$Node; (
>>>>> java.base@11.0.9.1)
>>>>>
>>>>> At run 6:
>>>>>
>>>>> 1: 81.408 25.375.400 [B (java.base@11.0.9.1)
>>>>> 2: 12.479 7.249.392 [I (java.base@11.0.9.1)
>>>>> 3: 29.090 3.496.168 java.lang.Class (
>>>>> java.base@11.0.9.1)
>>>>> 4: 4.347 2.813.416 [Ljava.util.HashMap$Node; (
>>>>> java.base@11.0.9.1)
>>>>> 5: 71.584 1.718.016 java.lang.String (
>>>>> java.base@11.0.9.1)
>>>>>
>>>>> At run 8:
>>>>>
>>>>> 1: 985.979 127.193.256 [B (java.base@11.0.9.1)
>>>>> 2: 35.400 13.702.112 [I (java.base@11.0.9.1)
>>>>> 3: 260.387 6.249.288 java.lang.String (
>>>>> java.base@11.0.9.1)
>>>>> 4: 148.836 5.953.440 java.util.HashMap$KeyIterator (
>>>>> java.base@11.0.9.1)
>>>>> 5: 17.641 5.222.344 [Ljava.util.HashMap$Node; (
>>>>> java.base@11.0.9.1)
>>>>>
>>>>> Thanks in advance for any help,
>>>>> Flavio
>>>>>
>>>>> [1]
>>>>> --------------------------------------------------------------------------------------------------
>>>>> java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory
>>>>> error has occurred. This can mean two things: either the job requires a
>>>>> larger size of JVM metaspace to load classes or there is a class loading
>>>>> leak. In the first case 'taskmanager.memory.jvm-metaspace.size'
>>>>> configuration option should be increased. If the error persists (usually in
>>>>> cluster after several job (re-)submissions) then there is probably a class
>>>>> loading leak in user code or some of its dependencies which has to be
>>>>> investigated and fixed. The task executor has to be shutdown...
>>>>> at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
>>>>> ~[?:?]
>>>>> at
>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
>>>>> ~[?:?]
>>>>> at
>>>>> java.net.URLClassLoader.defineClass(URLClassLoader.java:550) ~[?:?]
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:458)
>>>>> ~[?:?]
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:452)
>>>>> ~[?:?]
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> ~[?:?]
>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:451)
>>>>> ~[?:?]
>>>>> at
>>>>> org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71)
>>>>> ~[flink-dist_2.12-1.11.0.jar:1.11.0]
>>>>> at
>>>>> org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
>>>>> [flink-dist_2.12-1.11.0.jar:1.11.0]
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:522) [?:?]
>>>>>
>>>>> ------------------------------
>>
>> The information contained in this e-mail is confidential and/or
>> proprietary to Capital One and/or its affiliates and may only be used
>> solely in performance of work or services for Capital One. The information
>> transmitted herewith is intended only for use by the individual or entity
>> to which it is addressed. If the reader of this message is not the intended
>> recipient, you are hereby notified that any review, retransmission,
>> dissemination, distribution, copying or other use of, or taking of any
>> action in reliance upon this information is strictly prohibited. If you
>> have received this communication in error, please contact the sender and
>> delete the material from your computer.
>>
>>
>>
>>
Re: [External Sender] Re: Random Task executor shutdown
(java.lang.OutOfMemoryError: Metaspace)
Posted by Flavio Pompermaier <po...@okkam.it>.
Thank you Kye for your insights...in my mind, if the job runs without
problems one or more times the heap size, and thus the medatadata-size, is
big enough and I should not increase it (on the same data of course).
So I'll try to understand who is leaking what..the advice to avoid the
dynamic class loading is just a workaround to me..there's something wrong
going on and tomorrow I'll try to understand the root cause of the
problem using -XX:NativeMemoryTracking=summary as you suggested.
I'll keep you up to date with my findings..
Best,
Flavio
On Mon, Nov 16, 2020 at 8:22 PM Kye Bae <ky...@capitalone.com> wrote:
> Hello!
>
> The JVM metaspace is where all the classes (not class instances or
> objects) get loaded. jmap -histo is going to show you the heap space usage
> info not the metaspace.
>
> You could inspect what is happening in the metaspace by using jcmd (e.g.,
> jcmd JPID VM.native_memory summary) after restarting the cluster with "
> *-XX:NativeMemoryTracking=summary"*
>
> *As the error message suggests, you may need to increase *taskmanager.memory.jvm-metaspace.size,
> but you need to be slightly careful when specifying the memory parameters
> in flink-conf.yaml in Flink 1.10 (they have an issue with a confusing error
> message).
>
> Anothing thing to keep in mind is that you may want to avoid using dynamic
> classloading (
> https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/debugging_classloading.html#avoiding-dynamic-classloading-for-user-code):
> when the job continuously fails for some temporary issues, it will load the
> same class files into the metaspace multiple times and could exceed
> whatever the limit you set it.
>
> -K
>
> On Mon, Nov 16, 2020 at 12:39 PM Jan Lukavský <je...@seznam.cz> wrote:
>
>> The exclusions should not have any impact on that, because what defines
>> which classloader will load which class is not the presence or particular
>> class in a specific jar, but the configuration of parent-first-patterns [1].
>>
>> If you don't use any flink internal imports, than it still might be the
>> case, that a class in any of the packages defined by the
>> parent-first-pattern to hold reference to your user-code classes, which
>> would cause the leak. I'd recommend to inspect the heap dump after several
>> restarts of the application and look for reference to Class objects from
>> the root set.
>>
>> Jan
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#class-loading
>> <https://urldefense.com/v3/__https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html*class-loading__;Iw!!EFVe01R3CjU!NF2uHM8m-1kZSP7E3-7ZhdKcQa2U3wTqWKDA8zSI4727gH8ASTFc4h9qBaX4_W2wJA$>
>> On 11/16/20 5:34 PM, Flavio Pompermaier wrote:
>>
>> I've tried to remove all possible imports of classes not contained in the
>> fat jar but I still face the same problem.
>> I've also tried to reduce as much as possible the exclude in the shade
>> section of the maven plugin (I took the one at [1]) so now I exclude only
>> few dependencies..could it be that I should include org.slf4j:* if I use
>> static import of it?
>>
>> <artifactSet>
>> <excludes>
>> <exclude>com.google.code.findbugs:jsr305</exclude>
>> <exclude>org.slf4j:*</exclude>
>> <exclude>log4j:*</exclude>
>> </excludes>
>> </artifactSet>
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html#appendix-template-for-building-a-jar-with-dependencies
>> <https://urldefense.com/v3/__https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html*appendix-template-for-building-a-jar-with-dependencies__;Iw!!EFVe01R3CjU!NF2uHM8m-1kZSP7E3-7ZhdKcQa2U3wTqWKDA8zSI4727gH8ASTFc4h9qBaWGhZYoqQ$>
>>
>> On Mon, Nov 16, 2020 at 3:29 PM Jan Lukavský <je...@seznam.cz> wrote:
>>
>>> Yes, that could definitely cause this. You should probably avoid using
>>> these flink-internal shaded classes and ship your own versions (not shaded).
>>>
>>> Best,
>>>
>>> Jan
>>> On 11/16/20 3:22 PM, Flavio Pompermaier wrote:
>>>
>>> Thank you Jan for your valuable feedback.
>>> Could it be that I should not use import shaded-jackson classes in my
>>> user code?
>>> For example import
>>> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper?
>>>
>>> Bets,
>>> Flavio
>>>
>>> On Mon, Nov 16, 2020 at 3:15 PM Jan Lukavský <je...@seznam.cz> wrote:
>>>
>>>> Hi Flavio,
>>>>
>>>> when I encountered quite similar problem that you describe, it was
>>>> related to a static storage located in class that was loaded
>>>> "parent-first". In my case it was it was in java.lang.ClassValue, but it
>>>> might (and probably will be) different in your case. The problem is that if
>>>> user-code registers something in some (static) storage located in class
>>>> loaded with parent (TaskTracker) classloader, then its associated classes
>>>> will never be GC'd and Metaspace will grow. A good starting point would be
>>>> not to focus on biggest consumers of heap (in general), but to look at
>>>> where the 15k objects of type Class are referenced from. That might help
>>>> you figure this out. I'm not sure if there is something that can be done in
>>>> general to prevent this type of leaks. That would be probably question on
>>>> dev@ mailing list.
>>>>
>>>> Best,
>>>>
>>>> Jan
>>>> On 11/16/20 2:27 PM, Flavio Pompermaier wrote:
>>>>
>>>> Hello everybody,
>>>> I was writing this email when a similar thread on this mailing list
>>>> appeared..
>>>> The difference is that the other problem seems to be related with Flink
>>>> 1.10 on YARN and does not output anything helpful in debugging the cause of
>>>> the problem.
>>>>
>>>> Indeed, in my use case I use Flink 1.11.0 and Flink on a standalone
>>>> session cluster (the job is submitted to the cluster using the CLI client).
>>>> The problem arises when I submit the same job for about 20 times (this
>>>> number unfortunately is not deterministic and can change a little bit). The
>>>> error reported by the Task Executor is related to the ever growing
>>>> Metaspace..the error seems to be pretty detailed [1].
>>>>
>>>> I found the same issue in some previous threads on this mailing list
>>>> and I've tried to figure it out the cause of the problem. The issue is that
>>>> looking at the objects allocated I don't really get an idea of the source
>>>> of the problem because the type of objects that are consuming the memory
>>>> are of general purpose (i.e. Bytes, Integers and Strings)...these are my
>>>> "top" memory consumers if looking at the output of jmap -histo <PID>:
>>>>
>>>> At run 0:
>>>>
>>>> num #instances #bytes class name (module)
>>>> -------------------------------------------------------
>>>> 1: 46238 13224056 [B (java.base@11.0.9.1)
>>>> 2: 3736 6536672 [I (java.base@11.0.9.1)
>>>> 3: 38081 913944 java.lang.String (
>>>> java.base@11.0.9.1)
>>>> 4: 26 852384
>>>> [Lakka.dispatch.forkjoin.ForkJoinTask;
>>>> 5: 7146 844984 java.lang.Class (java.base@11.0.9.1
>>>> )
>>>>
>>>> At run 1:
>>>>
>>>> 1: 77.608 25.317.496 [B (java.base@11.0.9.1)
>>>> 2: 7.004 9.088.360 [I (java.base@11.0.9.1)
>>>> 3: 15.814 1.887.256 java.lang.Class (
>>>> java.base@11.0.9.1)
>>>> 4: 67.381 1.617.144 java.lang.String (
>>>> java.base@11.0.9.1)
>>>> 5: 3.906 1.422.960 [Ljava.util.HashMap$Node; (
>>>> java.base@11.0.9.1)
>>>>
>>>> At run 6:
>>>>
>>>> 1: 81.408 25.375.400 [B (java.base@11.0.9.1)
>>>> 2: 12.479 7.249.392 [I (java.base@11.0.9.1)
>>>> 3: 29.090 3.496.168 java.lang.Class (
>>>> java.base@11.0.9.1)
>>>> 4: 4.347 2.813.416 [Ljava.util.HashMap$Node; (
>>>> java.base@11.0.9.1)
>>>> 5: 71.584 1.718.016 java.lang.String (
>>>> java.base@11.0.9.1)
>>>>
>>>> At run 8:
>>>>
>>>> 1: 985.979 127.193.256 [B (java.base@11.0.9.1)
>>>> 2: 35.400 13.702.112 [I (java.base@11.0.9.1)
>>>> 3: 260.387 6.249.288 java.lang.String (
>>>> java.base@11.0.9.1)
>>>> 4: 148.836 5.953.440 java.util.HashMap$KeyIterator (
>>>> java.base@11.0.9.1)
>>>> 5: 17.641 5.222.344 [Ljava.util.HashMap$Node; (
>>>> java.base@11.0.9.1)
>>>>
>>>> Thanks in advance for any help,
>>>> Flavio
>>>>
>>>> [1]
>>>> --------------------------------------------------------------------------------------------------
>>>> java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory
>>>> error has occurred. This can mean two things: either the job requires a
>>>> larger size of JVM metaspace to load classes or there is a class loading
>>>> leak. In the first case 'taskmanager.memory.jvm-metaspace.size'
>>>> configuration option should be increased. If the error persists (usually in
>>>> cluster after several job (re-)submissions) then there is probably a class
>>>> loading leak in user code or some of its dependencies which has to be
>>>> investigated and fixed. The task executor has to be shutdown...
>>>> at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
>>>> ~[?:?]
>>>> at
>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
>>>> ~[?:?]
>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:550)
>>>> ~[?:?]
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:458) ~[?:?]
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:452) ~[?:?]
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> ~[?:?]
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:451)
>>>> ~[?:?]
>>>> at
>>>> org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71)
>>>> ~[flink-dist_2.12-1.11.0.jar:1.11.0]
>>>> at
>>>> org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
>>>> [flink-dist_2.12-1.11.0.jar:1.11.0]
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:522) [?:?]
>>>>
>>>> ------------------------------
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
>
>
>
Re: [External Sender] Re: Random Task executor shutdown
(java.lang.OutOfMemoryError: Metaspace)
Posted by Kye Bae <ky...@capitalone.com>.
Hello!
The JVM metaspace is where all the classes (not class instances or objects)
get loaded. jmap -histo is going to show you the heap space usage info not
the metaspace.
You could inspect what is happening in the metaspace by using jcmd (e.g.,
jcmd JPID VM.native_memory summary) after restarting the cluster with "
*-XX:NativeMemoryTracking=summary"*
*As the error message suggests, you may need to increase
*taskmanager.memory.jvm-metaspace.size,
but you need to be slightly careful when specifying the memory parameters
in flink-conf.yaml in Flink 1.10 (they have an issue with a confusing error
message).
Anothing thing to keep in mind is that you may want to avoid using dynamic
classloading (
https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/debugging_classloading.html#avoiding-dynamic-classloading-for-user-code):
when the job continuously fails for some temporary issues, it will load the
same class files into the metaspace multiple times and could exceed
whatever the limit you set it.
-K
On Mon, Nov 16, 2020 at 12:39 PM Jan Lukavský <je...@seznam.cz> wrote:
> The exclusions should not have any impact on that, because what defines
> which classloader will load which class is not the presence or particular
> class in a specific jar, but the configuration of parent-first-patterns [1].
>
> If you don't use any flink internal imports, than it still might be the
> case, that a class in any of the packages defined by the
> parent-first-pattern to hold reference to your user-code classes, which
> would cause the leak. I'd recommend to inspect the heap dump after several
> restarts of the application and look for reference to Class objects from
> the root set.
>
> Jan
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#class-loading
> <https://urldefense.com/v3/__https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html*class-loading__;Iw!!EFVe01R3CjU!NF2uHM8m-1kZSP7E3-7ZhdKcQa2U3wTqWKDA8zSI4727gH8ASTFc4h9qBaX4_W2wJA$>
> On 11/16/20 5:34 PM, Flavio Pompermaier wrote:
>
> I've tried to remove all possible imports of classes not contained in the
> fat jar but I still face the same problem.
> I've also tried to reduce as much as possible the exclude in the shade
> section of the maven plugin (I took the one at [1]) so now I exclude only
> few dependencies..could it be that I should include org.slf4j:* if I use
> static import of it?
>
> <artifactSet>
> <excludes>
> <exclude>com.google.code.findbugs:jsr305</exclude>
> <exclude>org.slf4j:*</exclude>
> <exclude>log4j:*</exclude>
> </excludes>
> </artifactSet>
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html#appendix-template-for-building-a-jar-with-dependencies
> <https://urldefense.com/v3/__https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html*appendix-template-for-building-a-jar-with-dependencies__;Iw!!EFVe01R3CjU!NF2uHM8m-1kZSP7E3-7ZhdKcQa2U3wTqWKDA8zSI4727gH8ASTFc4h9qBaWGhZYoqQ$>
>
> On Mon, Nov 16, 2020 at 3:29 PM Jan Lukavský <je...@seznam.cz> wrote:
>
>> Yes, that could definitely cause this. You should probably avoid using
>> these flink-internal shaded classes and ship your own versions (not shaded).
>>
>> Best,
>>
>> Jan
>> On 11/16/20 3:22 PM, Flavio Pompermaier wrote:
>>
>> Thank you Jan for your valuable feedback.
>> Could it be that I should not use import shaded-jackson classes in my
>> user code?
>> For example import
>> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper?
>>
>> Bets,
>> Flavio
>>
>> On Mon, Nov 16, 2020 at 3:15 PM Jan Lukavský <je...@seznam.cz> wrote:
>>
>>> Hi Flavio,
>>>
>>> when I encountered quite similar problem that you describe, it was
>>> related to a static storage located in class that was loaded
>>> "parent-first". In my case it was it was in java.lang.ClassValue, but it
>>> might (and probably will be) different in your case. The problem is that if
>>> user-code registers something in some (static) storage located in class
>>> loaded with parent (TaskTracker) classloader, then its associated classes
>>> will never be GC'd and Metaspace will grow. A good starting point would be
>>> not to focus on biggest consumers of heap (in general), but to look at
>>> where the 15k objects of type Class are referenced from. That might help
>>> you figure this out. I'm not sure if there is something that can be done in
>>> general to prevent this type of leaks. That would be probably question on
>>> dev@ mailing list.
>>>
>>> Best,
>>>
>>> Jan
>>> On 11/16/20 2:27 PM, Flavio Pompermaier wrote:
>>>
>>> Hello everybody,
>>> I was writing this email when a similar thread on this mailing list
>>> appeared..
>>> The difference is that the other problem seems to be related with Flink
>>> 1.10 on YARN and does not output anything helpful in debugging the cause of
>>> the problem.
>>>
>>> Indeed, in my use case I use Flink 1.11.0 and Flink on a standalone
>>> session cluster (the job is submitted to the cluster using the CLI client).
>>> The problem arises when I submit the same job for about 20 times (this
>>> number unfortunately is not deterministic and can change a little bit). The
>>> error reported by the Task Executor is related to the ever growing
>>> Metaspace..the error seems to be pretty detailed [1].
>>>
>>> I found the same issue in some previous threads on this mailing list and
>>> I've tried to figure it out the cause of the problem. The issue is that
>>> looking at the objects allocated I don't really get an idea of the source
>>> of the problem because the type of objects that are consuming the memory
>>> are of general purpose (i.e. Bytes, Integers and Strings)...these are my
>>> "top" memory consumers if looking at the output of jmap -histo <PID>:
>>>
>>> At run 0:
>>>
>>> num #instances #bytes class name (module)
>>> -------------------------------------------------------
>>> 1: 46238 13224056 [B (java.base@11.0.9.1)
>>> 2: 3736 6536672 [I (java.base@11.0.9.1)
>>> 3: 38081 913944 java.lang.String (java.base@11.0.9.1
>>> )
>>> 4: 26 852384
>>> [Lakka.dispatch.forkjoin.ForkJoinTask;
>>> 5: 7146 844984 java.lang.Class (java.base@11.0.9.1)
>>>
>>> At run 1:
>>>
>>> 1: 77.608 25.317.496 [B (java.base@11.0.9.1)
>>> 2: 7.004 9.088.360 [I (java.base@11.0.9.1)
>>> 3: 15.814 1.887.256 java.lang.Class (
>>> java.base@11.0.9.1)
>>> 4: 67.381 1.617.144 java.lang.String (
>>> java.base@11.0.9.1)
>>> 5: 3.906 1.422.960 [Ljava.util.HashMap$Node; (
>>> java.base@11.0.9.1)
>>>
>>> At run 6:
>>>
>>> 1: 81.408 25.375.400 [B (java.base@11.0.9.1)
>>> 2: 12.479 7.249.392 [I (java.base@11.0.9.1)
>>> 3: 29.090 3.496.168 java.lang.Class (
>>> java.base@11.0.9.1)
>>> 4: 4.347 2.813.416 [Ljava.util.HashMap$Node; (
>>> java.base@11.0.9.1)
>>> 5: 71.584 1.718.016 java.lang.String (
>>> java.base@11.0.9.1)
>>>
>>> At run 8:
>>>
>>> 1: 985.979 127.193.256 [B (java.base@11.0.9.1)
>>> 2: 35.400 13.702.112 [I (java.base@11.0.9.1)
>>> 3: 260.387 6.249.288 java.lang.String (
>>> java.base@11.0.9.1)
>>> 4: 148.836 5.953.440 java.util.HashMap$KeyIterator (
>>> java.base@11.0.9.1)
>>> 5: 17.641 5.222.344 [Ljava.util.HashMap$Node; (
>>> java.base@11.0.9.1)
>>>
>>> Thanks in advance for any help,
>>> Flavio
>>>
>>> [1]
>>> --------------------------------------------------------------------------------------------------
>>> java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error
>>> has occurred. This can mean two things: either the job requires a larger
>>> size of JVM metaspace to load classes or there is a class loading leak. In
>>> the first case 'taskmanager.memory.jvm-metaspace.size' configuration option
>>> should be increased. If the error persists (usually in cluster after
>>> several job (re-)submissions) then there is probably a class loading leak
>>> in user code or some of its dependencies which has to be investigated and
>>> fixed. The task executor has to be shutdown...
>>> at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
>>> ~[?:?]
>>> at
>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
>>> ~[?:?]
>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:550)
>>> ~[?:?]
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:458) ~[?:?]
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:452) ~[?:?]
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> ~[?:?]
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:451)
>>> ~[?:?]
>>> at
>>> org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71)
>>> ~[flink-dist_2.12-1.11.0.jar:1.11.0]
>>> at
>>> org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
>>> [flink-dist_2.12-1.11.0.jar:1.11.0]
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:522) [?:?]
>>>
>>>
______________________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: Random Task executor shutdown (java.lang.OutOfMemoryError:
Metaspace)
Posted by Jan Lukavský <je...@seznam.cz>.
The exclusions should not have any impact on that, because what defines
which classloader will load which class is not the presence or
particular class in a specific jar, but the configuration of
parent-first-patterns [1].
If you don't use any flink internal imports, than it still might be the
case, that a class in any of the packages defined by the
parent-first-pattern to hold reference to your user-code classes, which
would cause the leak. I'd recommend to inspect the heap dump after
several restarts of the application and look for reference to Class
objects from the root set.
Jan
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#class-loading
On 11/16/20 5:34 PM, Flavio Pompermaier wrote:
> I've tried to remove all possible imports of classes not contained in
> the fat jar but I still face the same problem.
> I've also tried to reduce as much as possible the exclude in the shade
> section of the maven plugin (I took the one at [1]) so now I exclude
> only few dependencies..could it be that I should include org.slf4j:*
> if I use static import of it?
>
> <artifactSet>
> <excludes>
> <exclude>com.google.code.findbugs:jsr305</exclude>
> <exclude>org.slf4j:*</exclude>
> <exclude>log4j:*</exclude>
> </excludes>
> </artifactSet>
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html#appendix-template-for-building-a-jar-with-dependencies
>
> On Mon, Nov 16, 2020 at 3:29 PM Jan Lukavský <je.ik@seznam.cz
> <ma...@seznam.cz>> wrote:
>
> Yes, that could definitely cause this. You should probably avoid
> using these flink-internal shaded classes and ship your own
> versions (not shaded).
>
> Best,
>
> Jan
>
> On 11/16/20 3:22 PM, Flavio Pompermaier wrote:
>> Thank you Jan for your valuable feedback.
>> Could it be that I should not use import shaded-jackson classes
>> in my user code?
>> For example import
>> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper?
>>
>> Bets,
>> Flavio
>>
>> On Mon, Nov 16, 2020 at 3:15 PM Jan Lukavský <je.ik@seznam.cz
>> <ma...@seznam.cz>> wrote:
>>
>> Hi Flavio,
>>
>> when I encountered quite similar problem that you describe,
>> it was related to a static storage located in class that was
>> loaded "parent-first". In my case it was it was in
>> java.lang.ClassValue, but it might (and probably will be)
>> different in your case. The problem is that if user-code
>> registers something in some (static) storage located in class
>> loaded with parent (TaskTracker) classloader, then its
>> associated classes will never be GC'd and Metaspace will
>> grow. A good starting point would be not to focus on biggest
>> consumers of heap (in general), but to look at where the 15k
>> objects of type Class are referenced from. That might help
>> you figure this out. I'm not sure if there is something that
>> can be done in general to prevent this type of leaks. That
>> would be probably question on dev@ mailing list.
>>
>> Best,
>>
>> Jan
>>
>> On 11/16/20 2:27 PM, Flavio Pompermaier wrote:
>>> Hello everybody,
>>> I was writing this email when a similar thread on this
>>> mailing list appeared..
>>> The difference is that the other problem seems to be related
>>> with Flink 1.10 on YARN and does not output anything helpful
>>> in debugging the cause of the problem.
>>>
>>> Indeed, in my use case I use Flink 1.11.0 and Flink on a
>>> standalone session cluster (the job is submitted to the
>>> cluster using the CLI client).
>>> The problem arises when I submit the same job for about 20
>>> times (this number unfortunately is not deterministic and
>>> can change a little bit). The error reported by the Task
>>> Executor is related to the ever growing Metaspace..the error
>>> seems to be pretty detailed [1].
>>>
>>> I found the same issue in some previous threads on this
>>> mailing list and I've tried to figure it out the cause of
>>> the problem. The issue is that looking at the objects
>>> allocated I don't really get an idea of the source of the
>>> problem because the type of objects that are consuming the
>>> memory are of general purpose (i.e. Bytes, Integers and
>>> Strings)...these are my "top" memory consumers if looking at
>>> the output of jmap -histo <PID>:
>>>
>>> At run 0:
>>>
>>> num #instances #bytes class name (module)
>>> -------------------------------------------------------
>>> 1: 46238 13224056 [B (java.base@11.0.9.1
>>> <ma...@11.0.9.1>)
>>> 2: 3736 6536672 [I (java.base@11.0.9.1
>>> <ma...@11.0.9.1>)
>>> 3: 38081 913944 java.lang.String
>>> (java.base@11.0.9.1 <ma...@11.0.9.1>)
>>> 4: 26 852384
>>> [Lakka.dispatch.forkjoin.ForkJoinTask;
>>> 5: 7146 844984 java.lang.Class
>>> (java.base@11.0.9.1 <ma...@11.0.9.1>)
>>>
>>> At run 1:
>>>
>>> 1: 77.608 25.317.496 [B
>>> (java.base@11.0.9.1 <ma...@11.0.9.1>)
>>> 2: 7.004 9.088.360 [I
>>> (java.base@11.0.9.1 <ma...@11.0.9.1>)
>>> 3: 15.814 1.887.256 java.lang.Class
>>> (java.base@11.0.9.1 <ma...@11.0.9.1>)
>>> 4: 67.381 1.617.144 java.lang.String
>>> (java.base@11.0.9.1 <ma...@11.0.9.1>)
>>> 5: 3.906 1.422.960
>>> [Ljava.util.HashMap$Node; (java.base@11.0.9.1
>>> <ma...@11.0.9.1>)
>>>
>>> At run 6:
>>>
>>> 1: 81.408 25.375.400 [B
>>> (java.base@11.0.9.1 <ma...@11.0.9.1>)
>>> 2: 12.479 7.249.392 [I
>>> (java.base@11.0.9.1 <ma...@11.0.9.1>)
>>> 3: 29.090 3.496.168 java.lang.Class
>>> (java.base@11.0.9.1 <ma...@11.0.9.1>)
>>> 4: 4.347 2.813.416
>>> [Ljava.util.HashMap$Node; (java.base@11.0.9.1
>>> <ma...@11.0.9.1>)
>>> 5: 71.584 1.718.016 java.lang.String
>>> (java.base@11.0.9.1 <ma...@11.0.9.1>)
>>>
>>> At run 8:
>>>
>>> 1: 985.979 127.193.256 [B
>>> (java.base@11.0.9.1 <ma...@11.0.9.1>)
>>> 2: 35.400 13.702.112 [I
>>> (java.base@11.0.9.1 <ma...@11.0.9.1>)
>>> 3: 260.387 6.249.288 java.lang.String
>>> (java.base@11.0.9.1 <ma...@11.0.9.1>)
>>> 4: 148.836 5.953.440
>>> java.util.HashMap$KeyIterator (java.base@11.0.9.1
>>> <ma...@11.0.9.1>)
>>> 5: 17.641 5.222.344
>>> [Ljava.util.HashMap$Node; (java.base@11.0.9.1
>>> <ma...@11.0.9.1>)
>>>
>>> Thanks in advance for any help,
>>> Flavio
>>>
>>> [1]
>>> --------------------------------------------------------------------------------------------------
>>> java.lang.OutOfMemoryError: Metaspace. The metaspace
>>> out-of-memory error has occurred. This can mean two things:
>>> either the job requires a larger size of JVM metaspace to
>>> load classes or there is a class loading leak. In the first
>>> case 'taskmanager.memory.jvm-metaspace.size' configuration
>>> option should be increased. If the error persists (usually
>>> in cluster after several job (re-)submissions) then there is
>>> probably a class loading leak in user code or some of its
>>> dependencies which has to be investigated and fixed. The
>>> task executor has to be shutdown...
>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>> ~[?:?]
>>> at
>>> java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
>>> at
>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
>>> ~[?:?]
>>> at
>>> java.net.URLClassLoader.defineClass(URLClassLoader.java:550)
>>> ~[?:?]
>>> at
>>> java.net.URLClassLoader$1.run(URLClassLoader.java:458) ~[?:?]
>>> at
>>> java.net.URLClassLoader$1.run(URLClassLoader.java:452) ~[?:?]
>>> at
>>> java.security.AccessController.doPrivileged(Native Method)
>>> ~[?:?]
>>> at
>>> java.net.URLClassLoader.findClass(URLClassLoader.java:451)
>>> ~[?:?]
>>> at
>>> org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71)
>>> ~[flink-dist_2.12-1.11.0.jar:1.11.0]
>>> at
>>> org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
>>> [flink-dist_2.12-1.11.0.jar:1.11.0]
>>> at
>>> java.lang.ClassLoader.loadClass(ClassLoader.java:522) [?:?]
>>
Re: Random Task executor shutdown (java.lang.OutOfMemoryError: Metaspace)
Posted by Flavio Pompermaier <po...@okkam.it>.
I've tried to remove all possible imports of classes not contained in the
fat jar but I still face the same problem.
I've also tried to reduce as much as possible the exclude in the shade
section of the maven plugin (I took the one at [1]) so now I exclude only
few dependencies..could it be that I should include org.slf4j:* if I use
static import of it?
<artifactSet>
<excludes>
<exclude>com.google.code.findbugs:jsr305</exclude>
<exclude>org.slf4j:*</exclude>
<exclude>log4j:*</exclude>
</excludes>
</artifactSet>
[1]
https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html#appendix-template-for-building-a-jar-with-dependencies
On Mon, Nov 16, 2020 at 3:29 PM Jan Lukavský <je...@seznam.cz> wrote:
> Yes, that could definitely cause this. You should probably avoid using
> these flink-internal shaded classes and ship your own versions (not shaded).
>
> Best,
>
> Jan
> On 11/16/20 3:22 PM, Flavio Pompermaier wrote:
>
> Thank you Jan for your valuable feedback.
> Could it be that I should not use import shaded-jackson classes in my user
> code?
> For example import
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper?
>
> Bets,
> Flavio
>
> On Mon, Nov 16, 2020 at 3:15 PM Jan Lukavský <je...@seznam.cz> wrote:
>
>> Hi Flavio,
>>
>> when I encountered quite similar problem that you describe, it was
>> related to a static storage located in class that was loaded
>> "parent-first". In my case it was it was in java.lang.ClassValue, but it
>> might (and probably will be) different in your case. The problem is that if
>> user-code registers something in some (static) storage located in class
>> loaded with parent (TaskTracker) classloader, then its associated classes
>> will never be GC'd and Metaspace will grow. A good starting point would be
>> not to focus on biggest consumers of heap (in general), but to look at
>> where the 15k objects of type Class are referenced from. That might help
>> you figure this out. I'm not sure if there is something that can be done in
>> general to prevent this type of leaks. That would be probably question on
>> dev@ mailing list.
>>
>> Best,
>>
>> Jan
>> On 11/16/20 2:27 PM, Flavio Pompermaier wrote:
>>
>> Hello everybody,
>> I was writing this email when a similar thread on this mailing list
>> appeared..
>> The difference is that the other problem seems to be related with Flink
>> 1.10 on YARN and does not output anything helpful in debugging the cause of
>> the problem.
>>
>> Indeed, in my use case I use Flink 1.11.0 and Flink on a standalone
>> session cluster (the job is submitted to the cluster using the CLI client).
>> The problem arises when I submit the same job for about 20 times (this
>> number unfortunately is not deterministic and can change a little bit). The
>> error reported by the Task Executor is related to the ever growing
>> Metaspace..the error seems to be pretty detailed [1].
>>
>> I found the same issue in some previous threads on this mailing list and
>> I've tried to figure it out the cause of the problem. The issue is that
>> looking at the objects allocated I don't really get an idea of the source
>> of the problem because the type of objects that are consuming the memory
>> are of general purpose (i.e. Bytes, Integers and Strings)...these are my
>> "top" memory consumers if looking at the output of jmap -histo <PID>:
>>
>> At run 0:
>>
>> num #instances #bytes class name (module)
>> -------------------------------------------------------
>> 1: 46238 13224056 [B (java.base@11.0.9.1)
>> 2: 3736 6536672 [I (java.base@11.0.9.1)
>> 3: 38081 913944 java.lang.String (java.base@11.0.9.1)
>> 4: 26 852384 [Lakka.dispatch.forkjoin.ForkJoinTask;
>> 5: 7146 844984 java.lang.Class (java.base@11.0.9.1)
>>
>> At run 1:
>>
>> 1: 77.608 25.317.496 [B (java.base@11.0.9.1)
>> 2: 7.004 9.088.360 [I (java.base@11.0.9.1)
>> 3: 15.814 1.887.256 java.lang.Class (
>> java.base@11.0.9.1)
>> 4: 67.381 1.617.144 java.lang.String (
>> java.base@11.0.9.1)
>> 5: 3.906 1.422.960 [Ljava.util.HashMap$Node; (
>> java.base@11.0.9.1)
>>
>> At run 6:
>>
>> 1: 81.408 25.375.400 [B (java.base@11.0.9.1)
>> 2: 12.479 7.249.392 [I (java.base@11.0.9.1)
>> 3: 29.090 3.496.168 java.lang.Class (
>> java.base@11.0.9.1)
>> 4: 4.347 2.813.416 [Ljava.util.HashMap$Node; (
>> java.base@11.0.9.1)
>> 5: 71.584 1.718.016 java.lang.String (
>> java.base@11.0.9.1)
>>
>> At run 8:
>>
>> 1: 985.979 127.193.256 [B (java.base@11.0.9.1)
>> 2: 35.400 13.702.112 [I (java.base@11.0.9.1)
>> 3: 260.387 6.249.288 java.lang.String (
>> java.base@11.0.9.1)
>> 4: 148.836 5.953.440 java.util.HashMap$KeyIterator (
>> java.base@11.0.9.1)
>> 5: 17.641 5.222.344 [Ljava.util.HashMap$Node; (
>> java.base@11.0.9.1)
>>
>> Thanks in advance for any help,
>> Flavio
>>
>> [1]
>> --------------------------------------------------------------------------------------------------
>> java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error
>> has occurred. This can mean two things: either the job requires a larger
>> size of JVM metaspace to load classes or there is a class loading leak. In
>> the first case 'taskmanager.memory.jvm-metaspace.size' configuration option
>> should be increased. If the error persists (usually in cluster after
>> several job (re-)submissions) then there is probably a class loading leak
>> in user code or some of its dependencies which has to be investigated and
>> fixed. The task executor has to be shutdown...
>> at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
>> at java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
>> at
>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
>> ~[?:?]
>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:550)
>> ~[?:?]
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:458) ~[?:?]
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:452) ~[?:?]
>> at java.security.AccessController.doPrivileged(Native Method)
>> ~[?:?]
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:451)
>> ~[?:?]
>> at
>> org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71)
>> ~[flink-dist_2.12-1.11.0.jar:1.11.0]
>> at
>> org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
>> [flink-dist_2.12-1.11.0.jar:1.11.0]
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:522) [?:?]
>>
>>
Re: Random Task executor shutdown (java.lang.OutOfMemoryError:
Metaspace)
Posted by Jan Lukavský <je...@seznam.cz>.
Yes, that could definitely cause this. You should probably avoid using
these flink-internal shaded classes and ship your own versions (not shaded).
Best,
Jan
On 11/16/20 3:22 PM, Flavio Pompermaier wrote:
> Thank you Jan for your valuable feedback.
> Could it be that I should not use import shaded-jackson classes in my
> user code?
> For example import
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper?
>
> Bets,
> Flavio
>
> On Mon, Nov 16, 2020 at 3:15 PM Jan Lukavský <je.ik@seznam.cz
> <ma...@seznam.cz>> wrote:
>
> Hi Flavio,
>
> when I encountered quite similar problem that you describe, it was
> related to a static storage located in class that was loaded
> "parent-first". In my case it was it was in java.lang.ClassValue,
> but it might (and probably will be) different in your case. The
> problem is that if user-code registers something in some (static)
> storage located in class loaded with parent (TaskTracker)
> classloader, then its associated classes will never be GC'd and
> Metaspace will grow. A good starting point would be not to focus
> on biggest consumers of heap (in general), but to look at where
> the 15k objects of type Class are referenced from. That might help
> you figure this out. I'm not sure if there is something that can
> be done in general to prevent this type of leaks. That would be
> probably question on dev@ mailing list.
>
> Best,
>
> Jan
>
> On 11/16/20 2:27 PM, Flavio Pompermaier wrote:
>> Hello everybody,
>> I was writing this email when a similar thread on this mailing
>> list appeared..
>> The difference is that the other problem seems to be related
>> with Flink 1.10 on YARN and does not output anything helpful in
>> debugging the cause of the problem.
>>
>> Indeed, in my use case I use Flink 1.11.0 and Flink on a
>> standalone session cluster (the job is submitted to the cluster
>> using the CLI client).
>> The problem arises when I submit the same job for about 20 times
>> (this number unfortunately is not deterministic and can change a
>> little bit). The error reported by the Task Executor is related
>> to the ever growing Metaspace..the error seems to be pretty
>> detailed [1].
>>
>> I found the same issue in some previous threads on this mailing
>> list and I've tried to figure it out the cause of the problem.
>> The issue is that looking at the objects allocated I don't really
>> get an idea of the source of the problem because the type of
>> objects that are consuming the memory are of general purpose
>> (i.e. Bytes, Integers and Strings)...these are my "top" memory
>> consumers if looking at the output of jmap -histo <PID>:
>>
>> At run 0:
>>
>> num #instances #bytes class name (module)
>> -------------------------------------------------------
>> 1: 46238 13224056 [B (java.base@11.0.9.1
>> <ma...@11.0.9.1>)
>> 2: 3736 6536672 [I (java.base@11.0.9.1
>> <ma...@11.0.9.1>)
>> 3: 38081 913944 java.lang.String
>> (java.base@11.0.9.1 <ma...@11.0.9.1>)
>> 4: 26 852384
>> [Lakka.dispatch.forkjoin.ForkJoinTask;
>> 5: 7146 844984 java.lang.Class
>> (java.base@11.0.9.1 <ma...@11.0.9.1>)
>>
>> At run 1:
>>
>> 1: 77.608 25.317.496 [B (java.base@11.0.9.1
>> <ma...@11.0.9.1>)
>> 2: 7.004 9.088.360 [I (java.base@11.0.9.1
>> <ma...@11.0.9.1>)
>> 3: 15.814 1.887.256 java.lang.Class
>> (java.base@11.0.9.1 <ma...@11.0.9.1>)
>> 4: 67.381 1.617.144 java.lang.String
>> (java.base@11.0.9.1 <ma...@11.0.9.1>)
>> 5: 3.906 1.422.960 [Ljava.util.HashMap$Node;
>> (java.base@11.0.9.1 <ma...@11.0.9.1>)
>>
>> At run 6:
>>
>> 1: 81.408 25.375.400 [B (java.base@11.0.9.1
>> <ma...@11.0.9.1>)
>> 2: 12.479 7.249.392 [I (java.base@11.0.9.1
>> <ma...@11.0.9.1>)
>> 3: 29.090 3.496.168 java.lang.Class
>> (java.base@11.0.9.1 <ma...@11.0.9.1>)
>> 4: 4.347 2.813.416 [Ljava.util.HashMap$Node;
>> (java.base@11.0.9.1 <ma...@11.0.9.1>)
>> 5: 71.584 1.718.016 java.lang.String
>> (java.base@11.0.9.1 <ma...@11.0.9.1>)
>>
>> At run 8:
>>
>> 1: 985.979 127.193.256 [B (java.base@11.0.9.1
>> <ma...@11.0.9.1>)
>> 2: 35.400 13.702.112 [I (java.base@11.0.9.1
>> <ma...@11.0.9.1>)
>> 3: 260.387 6.249.288 java.lang.String
>> (java.base@11.0.9.1 <ma...@11.0.9.1>)
>> 4: 148.836 5.953.440
>> java.util.HashMap$KeyIterator (java.base@11.0.9.1
>> <ma...@11.0.9.1>)
>> 5: 17.641 5.222.344 [Ljava.util.HashMap$Node;
>> (java.base@11.0.9.1 <ma...@11.0.9.1>)
>>
>> Thanks in advance for any help,
>> Flavio
>>
>> [1]
>> --------------------------------------------------------------------------------------------------
>> java.lang.OutOfMemoryError: Metaspace. The metaspace
>> out-of-memory error has occurred. This can mean two things:
>> either the job requires a larger size of JVM metaspace to load
>> classes or there is a class loading leak. In the first case
>> 'taskmanager.memory.jvm-metaspace.size' configuration option
>> should be increased. If the error persists (usually in cluster
>> after several job (re-)submissions) then there is probably a
>> class loading leak in user code or some of its dependencies which
>> has to be investigated and fixed. The task executor has to be
>> shutdown...
>> at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
>> at
>> java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
>> at
>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
>> ~[?:?]
>> at
>> java.net.URLClassLoader.defineClass(URLClassLoader.java:550) ~[?:?]
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:458)
>> ~[?:?]
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:452)
>> ~[?:?]
>> at java.security.AccessController.doPrivileged(Native
>> Method) ~[?:?]
>> at
>> java.net.URLClassLoader.findClass(URLClassLoader.java:451) ~[?:?]
>> at
>> org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71)
>> ~[flink-dist_2.12-1.11.0.jar:1.11.0]
>> at
>> org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
>> [flink-dist_2.12-1.11.0.jar:1.11.0]
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:522)
>> [?:?]
>
Re: Random Task executor shutdown (java.lang.OutOfMemoryError: Metaspace)
Posted by Flavio Pompermaier <po...@okkam.it>.
Thank you Jan for your valuable feedback.
Could it be that I should not use import shaded-jackson classes in my user
code?
For example import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper?
Bets,
Flavio
On Mon, Nov 16, 2020 at 3:15 PM Jan Lukavský <je...@seznam.cz> wrote:
> Hi Flavio,
>
> when I encountered quite similar problem that you describe, it was related
> to a static storage located in class that was loaded "parent-first". In my
> case it was it was in java.lang.ClassValue, but it might (and probably will
> be) different in your case. The problem is that if user-code registers
> something in some (static) storage located in class loaded with parent
> (TaskTracker) classloader, then its associated classes will never be GC'd
> and Metaspace will grow. A good starting point would be not to focus on
> biggest consumers of heap (in general), but to look at where the 15k
> objects of type Class are referenced from. That might help you figure this
> out. I'm not sure if there is something that can be done in general to
> prevent this type of leaks. That would be probably question on dev@
> mailing list.
>
> Best,
>
> Jan
> On 11/16/20 2:27 PM, Flavio Pompermaier wrote:
>
> Hello everybody,
> I was writing this email when a similar thread on this mailing list
> appeared..
> The difference is that the other problem seems to be related with Flink
> 1.10 on YARN and does not output anything helpful in debugging the cause of
> the problem.
>
> Indeed, in my use case I use Flink 1.11.0 and Flink on a standalone
> session cluster (the job is submitted to the cluster using the CLI client).
> The problem arises when I submit the same job for about 20 times (this
> number unfortunately is not deterministic and can change a little bit). The
> error reported by the Task Executor is related to the ever growing
> Metaspace..the error seems to be pretty detailed [1].
>
> I found the same issue in some previous threads on this mailing list and
> I've tried to figure it out the cause of the problem. The issue is that
> looking at the objects allocated I don't really get an idea of the source
> of the problem because the type of objects that are consuming the memory
> are of general purpose (i.e. Bytes, Integers and Strings)...these are my
> "top" memory consumers if looking at the output of jmap -histo <PID>:
>
> At run 0:
>
> num #instances #bytes class name (module)
> -------------------------------------------------------
> 1: 46238 13224056 [B (java.base@11.0.9.1)
> 2: 3736 6536672 [I (java.base@11.0.9.1)
> 3: 38081 913944 java.lang.String (java.base@11.0.9.1)
> 4: 26 852384 [Lakka.dispatch.forkjoin.ForkJoinTask;
> 5: 7146 844984 java.lang.Class (java.base@11.0.9.1)
>
> At run 1:
>
> 1: 77.608 25.317.496 [B (java.base@11.0.9.1)
> 2: 7.004 9.088.360 [I (java.base@11.0.9.1)
> 3: 15.814 1.887.256 java.lang.Class (java.base@11.0.9.1
> )
> 4: 67.381 1.617.144 java.lang.String (
> java.base@11.0.9.1)
> 5: 3.906 1.422.960 [Ljava.util.HashMap$Node; (
> java.base@11.0.9.1)
>
> At run 6:
>
> 1: 81.408 25.375.400 [B (java.base@11.0.9.1)
> 2: 12.479 7.249.392 [I (java.base@11.0.9.1)
> 3: 29.090 3.496.168 java.lang.Class (java.base@11.0.9.1
> )
> 4: 4.347 2.813.416 [Ljava.util.HashMap$Node; (
> java.base@11.0.9.1)
> 5: 71.584 1.718.016 java.lang.String (
> java.base@11.0.9.1)
>
> At run 8:
>
> 1: 985.979 127.193.256 [B (java.base@11.0.9.1)
> 2: 35.400 13.702.112 [I (java.base@11.0.9.1)
> 3: 260.387 6.249.288 java.lang.String (
> java.base@11.0.9.1)
> 4: 148.836 5.953.440 java.util.HashMap$KeyIterator (
> java.base@11.0.9.1)
> 5: 17.641 5.222.344 [Ljava.util.HashMap$Node; (
> java.base@11.0.9.1)
>
> Thanks in advance for any help,
> Flavio
>
> [1]
> --------------------------------------------------------------------------------------------------
> java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error
> has occurred. This can mean two things: either the job requires a larger
> size of JVM metaspace to load classes or there is a class loading leak. In
> the first case 'taskmanager.memory.jvm-metaspace.size' configuration option
> should be increased. If the error persists (usually in cluster after
> several job (re-)submissions) then there is probably a class loading leak
> in user code or some of its dependencies which has to be investigated and
> fixed. The task executor has to be shutdown...
> at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
> at java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
> ~[?:?]
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:550)
> ~[?:?]
> at java.net.URLClassLoader$1.run(URLClassLoader.java:458) ~[?:?]
> at java.net.URLClassLoader$1.run(URLClassLoader.java:452) ~[?:?]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:?]
> at java.net.URLClassLoader.findClass(URLClassLoader.java:451)
> ~[?:?]
> at
> org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71)
> ~[flink-dist_2.12-1.11.0.jar:1.11.0]
> at
> org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
> [flink-dist_2.12-1.11.0.jar:1.11.0]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:522) [?:?]
>
>
Re: Random Task executor shutdown (java.lang.OutOfMemoryError:
Metaspace)
Posted by Jan Lukavský <je...@seznam.cz>.
Hi Flavio,
when I encountered quite similar problem that you describe, it was
related to a static storage located in class that was loaded
"parent-first". In my case it was it was in java.lang.ClassValue, but it
might (and probably will be) different in your case. The problem is that
if user-code registers something in some (static) storage located in
class loaded with parent (TaskTracker) classloader, then its associated
classes will never be GC'd and Metaspace will grow. A good starting
point would be not to focus on biggest consumers of heap (in general),
but to look at where the 15k objects of type Class are referenced from.
That might help you figure this out. I'm not sure if there is something
that can be done in general to prevent this type of leaks. That would be
probably question on dev@ mailing list.
Best,
Jan
On 11/16/20 2:27 PM, Flavio Pompermaier wrote:
> Hello everybody,
> I was writing this email when a similar thread on this mailing list
> appeared..
> The difference is that the other problem seems to be related
> with Flink 1.10 on YARN and does not output anything helpful in
> debugging the cause of the problem.
>
> Indeed, in my use case I use Flink 1.11.0 and Flink on a standalone
> session cluster (the job is submitted to the cluster using the CLI
> client).
> The problem arises when I submit the same job for about 20 times (this
> number unfortunately is not deterministic and can change a little
> bit). The error reported by the Task Executor is related to the ever
> growing Metaspace..the error seems to be pretty detailed [1].
>
> I found the same issue in some previous threads on this mailing list
> and I've tried to figure it out the cause of the problem. The issue is
> that looking at the objects allocated I don't really get an idea of
> the source of the problem because the type of objects that are
> consuming the memory are of general purpose (i.e. Bytes, Integers and
> Strings)...these are my "top" memory consumers if looking at the
> output of jmap -histo <PID>:
>
> At run 0:
>
> num #instances #bytes class name (module)
> -------------------------------------------------------
> 1: 46238 13224056 [B (java.base@11.0.9.1
> <ma...@11.0.9.1>)
> 2: 3736 6536672 [I (java.base@11.0.9.1
> <ma...@11.0.9.1>)
> 3: 38081 913944 java.lang.String
> (java.base@11.0.9.1 <ma...@11.0.9.1>)
> 4: 26 852384 [Lakka.dispatch.forkjoin.ForkJoinTask;
> 5: 7146 844984 java.lang.Class
> (java.base@11.0.9.1 <ma...@11.0.9.1>)
>
> At run 1:
>
> 1: 77.608 25.317.496 [B (java.base@11.0.9.1
> <ma...@11.0.9.1>)
> 2: 7.004 9.088.360 [I (java.base@11.0.9.1
> <ma...@11.0.9.1>)
> 3: 15.814 1.887.256 java.lang.Class
> (java.base@11.0.9.1 <ma...@11.0.9.1>)
> 4: 67.381 1.617.144 java.lang.String
> (java.base@11.0.9.1 <ma...@11.0.9.1>)
> 5: 3.906 1.422.960 [Ljava.util.HashMap$Node;
> (java.base@11.0.9.1 <ma...@11.0.9.1>)
>
> At run 6:
>
> 1: 81.408 25.375.400 [B (java.base@11.0.9.1
> <ma...@11.0.9.1>)
> 2: 12.479 7.249.392 [I (java.base@11.0.9.1
> <ma...@11.0.9.1>)
> 3: 29.090 3.496.168 java.lang.Class
> (java.base@11.0.9.1 <ma...@11.0.9.1>)
> 4: 4.347 2.813.416 [Ljava.util.HashMap$Node;
> (java.base@11.0.9.1 <ma...@11.0.9.1>)
> 5: 71.584 1.718.016 java.lang.String
> (java.base@11.0.9.1 <ma...@11.0.9.1>)
>
> At run 8:
>
> 1: 985.979 127.193.256 [B (java.base@11.0.9.1
> <ma...@11.0.9.1>)
> 2: 35.400 13.702.112 [I (java.base@11.0.9.1
> <ma...@11.0.9.1>)
> 3: 260.387 6.249.288 java.lang.String
> (java.base@11.0.9.1 <ma...@11.0.9.1>)
> 4: 148.836 5.953.440 java.util.HashMap$KeyIterator
> (java.base@11.0.9.1 <ma...@11.0.9.1>)
> 5: 17.641 5.222.344 [Ljava.util.HashMap$Node;
> (java.base@11.0.9.1 <ma...@11.0.9.1>)
>
> Thanks in advance for any help,
> Flavio
>
> [1]
> --------------------------------------------------------------------------------------------------
> java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory
> error has occurred. This can mean two things: either the job requires
> a larger size of JVM metaspace to load classes or there is a class
> loading leak. In the first case
> 'taskmanager.memory.jvm-metaspace.size' configuration option should be
> increased. If the error persists (usually in cluster after several job
> (re-)submissions) then there is probably a class loading leak in user
> code or some of its dependencies which has to be investigated and
> fixed. The task executor has to be shutdown...
> at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
> at java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
> ~[?:?]
> at
> java.net.URLClassLoader.defineClass(URLClassLoader.java:550) ~[?:?]
> at java.net.URLClassLoader$1.run(URLClassLoader.java:458) ~[?:?]
> at java.net.URLClassLoader$1.run(URLClassLoader.java:452) ~[?:?]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:?]
> at java.net.URLClassLoader.findClass(URLClassLoader.java:451)
> ~[?:?]
> at
> org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71)
> ~[flink-dist_2.12-1.11.0.jar:1.11.0]
> at
> org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
> [flink-dist_2.12-1.11.0.jar:1.11.0]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:522) [?:?]