You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomee.apache.org by Jonathan Gallimore <jo...@gmail.com> on 2019/12/16 14:48:03 UTC
Re: Old Gen Full of Annotation Finder Index
Further update - I think my changes work with respect to this issue, but I
do have a bunch of unit test failures which I need to look into. For info,
here's the diff:
diff --git
a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
index a6e2bdc4f5..c0d1a8c98f 100644
---
a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
+++
b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
@@ -69,10 +69,7 @@ import org.apache.openejb.cdi.ThreadSingletonService;
import org.apache.openejb.classloader.ClassLoaderConfigurer;
import org.apache.openejb.classloader.CompositeClassLoaderConfigurer;
import org.apache.openejb.component.ClassLoaderEnricher;
-import org.apache.openejb.config.ConfigurationFactory;
-import org.apache.openejb.config.NewLoaderLogic;
-import org.apache.openejb.config.QuickJarsTxtParser;
-import org.apache.openejb.config.TldScanner;
+import org.apache.openejb.config.*;
import org.apache.openejb.core.ConnectorReference;
import org.apache.openejb.core.CoreContainerSystem;
import org.apache.openejb.core.CoreUserTransaction;
@@ -829,6 +826,14 @@ public class Assembler extends AssemblerTool
implements org.apache.openejb.spi.A
final AppContext appContext = new
AppContext(appInfo.appId, SystemInstance.get(), classLoader,
globalJndiContext, appJndiContext, appInfo.standaloneModule);
appContext.getProperties().putAll(appInfo.properties);
+
+ final Set<Object> keys =
appContext.getProperties().keySet();
+ for (final Object key : keys) {
+ if (! Module.class.isInstance(key)) {
+ appContext.getProperties().remove(key);
+ }
+ }
+
appContext.getInjections().addAll(injections);
appContext.getBindings().putAll(globalBindings);
appContext.getBindings().putAll(appBindings);
diff --git
a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
index feff954d40..aed0fda941 100644
---
a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
+++
b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
@@ -384,12 +384,7 @@ public class JndiEncBuilder {
reference = new LinkRef(jndiName);
} else if (BeanManager.class.equals(type)) {
- reference = new LazyObjectReference<BeanManager>(new
Callable<BeanManager>() {
- @Override
- public BeanManager call() throws Exception {
- return new
InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
- }
- });
+ reference = new LazyObjectReference<>(new
BeanManagerLazyReference());
} else if (UserTransaction.class.equals(type)) {
reference = new
IntraVmJndiReference("comp/UserTransaction");
@@ -684,4 +679,11 @@ public class JndiEncBuilder {
}
}
}
+
+ public static class BeanManagerLazyReference implements
Callable<BeanManager> {
+ @Override
+ public BeanManager call() throws Exception {
+ return new
InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
+ }
+ }
}
diff --git
a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
index 2cf387e112..285007925f 100644
---
a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
+++
b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
@@ -59,6 +59,7 @@ public class TempClassLoader extends URLClassLoader {
private final ClassLoader system;
private final boolean embedded;
private final boolean parentURLClassLoader;
+ private final StackTraceElement[] created;
// 80% of class files are smaller then 6k
private final ByteArrayOutputStream bout = new ByteArrayOutputStream(6
* 1024);
@@ -69,6 +70,7 @@ public class TempClassLoader extends URLClassLoader {
this.system = ClassLoader.getSystemClassLoader();
this.embedded = this.getClass().getClassLoader() == this.system;
this.parentURLClassLoader =
URLClassLoader.class.isInstance(parent);
+ this.created = new Throwable().getStackTrace();
}
/*
In essence, this does 2 things:
1. Defines the Callable for the LazyObjectReference for BeanManager as a
static class. This stops that object from having a reference to
JndiEncBuilder and everything that has attached to it.
2. Strips off the EjbModule / WebModule from the SuperProperties on
AppContext. This is set when deploying through the apps folder and lives
forever. It has the TempClassLoader attached, and therefore all the
AnnotationFinder stuff.
I suspect (2) is too aggressive and is causing the test failures, but will
confirm.
Jon
On Mon, Dec 16, 2019 at 11:27 AM Jonathan Gallimore <
jonathan.gallimore@gmail.com> wrote:
> Thanks for sending those over. I'm digging through this - I *think* I have
> pinned down a specific reference that stops that classloader from being
> released. The behaviour for a war/ear deployed in apps/ seems different to
> deploying in webapps/ - I'm assuming that's what you're doing. If you can
> confirm, that would help. I'll test out a patch with some bigger .ear files
> today.
>
> Thanks
>
> Jon
>
> On Thu, Dec 12, 2019 at 9:08 PM Jonathan Gallimore <
> jonathan.gallimore@gmail.com> wrote:
>
>> The mailing list seems to be stripping off your images - can you resend
>> and include my email address as well? I'd be keen to dig into this a little
>> more.
>>
>> Thanks
>>
>> Jon
>>
>> On Thu, Dec 12, 2019 at 9:00 PM Paul Carter-Brown
>> <pa...@jini.guru> wrote:
>>
>>> Hi,
>>>
>>> The code does seem to use org.apache.openejb.core.TempClassLoader. I can
>>> see one instance in the heap for every war in my ear. + 1.
>>>
>>> The TempClassLoader however still has 100's of strong references to it.
>>> E.g:
>>>
>>> [image: image.png]
>>>
>>> [image: image.png]
>>>
>>>
>>> Paul Carter-Brown
>>> Director
>>> Jini Guru
>>> m: +27 (0) 83 442 7179 <+27834427179>
>>> a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
>>> Johannesburg, South Africa
>>> w: jini.guru e: paul@jini.guru
>>>
>>> Disclaimer: This message and/or attachment(s) may contain
>>> privileged, confidential and/or personal information. If you are not the
>>> intended recipient you may not disclose or distribute any of
>>> the information contained within this message. In such case you must
>>> destroy this message and inform the sender of the error. Jini Guru may not
>>> accept liability for any errors, omissions, information and viruses
>>> contained in the transmission of this message. Any opinions, conclusions
>>> and other information contained within this message not related to Jini
>>> Guru official business is deemed to be that of the individual only and is
>>> not endorsed by Jini Guru.
>>>
>>>
>>>
>>> On Thu, Dec 12, 2019 at 9:29 AM Mark Struberg <st...@yahoo.de.invalid>
>>> wrote:
>>>
>>>> Hi!
>>>>
>>>> All this is just a workaround imo.
>>>>
>>>> Afair we (Romain) introduced a temporary throw-away ClassLoader for
>>>> scanning. That means after doing all the class scans we only keep the
>>>> extracted information but do not keep the Classes in memory. Doesn't this
>>>> work anymore?
>>>>
>>>> LieGrue,
>>>> strub
>>>>
>>>>
>>>> > Am 12.12.2019 um 00:45 schrieb Paul Carter-Brown
>>>> <pa...@jini.guru>:
>>>> >
>>>> > FYI. Graph of the change in heap size on a prod instance after this
>>>> change:
>>>> >
>>>> >
>>>> > Paul Carter-Brown
>>>> > Director
>>>> > Jini Guru
>>>> > m: +27 (0) 83 442 7179
>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>> Leslie
>>>> > Johannesburg, South Africa
>>>> > w: jini.guru e: paul@jini.guru
>>>> >
>>>> > Disclaimer: This message and/or attachment(s) may contain privileged,
>>>> confidential and/or personal information. If you are not the intended
>>>> recipient you may not disclose or distribute any of the information
>>>> contained within this message. In such case you must destroy this message
>>>> and inform the sender of the error. Jini Guru may not accept liability for
>>>> any errors, omissions, information and viruses contained in the
>>>> transmission of this message. Any opinions, conclusions and other
>>>> information contained within this message not related to Jini Guru official
>>>> business is deemed to be that of the individual only and is not endorsed by
>>>> Jini Guru.
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On Thu, Dec 12, 2019 at 1:17 AM Paul Carter-Brown
>>>> <pa...@jini.guru> wrote:
>>>> > Hi Jon,
>>>> >
>>>> > I took a look at the source and worked out that I could add my own
>>>> filters using openejb.additional.exclude.
>>>> >
>>>> > I set it to openejb.additional.exclude=com,net,io,org and the JVM
>>>> heap now sits at around 150MB whereas it would never go below 450MB
>>>> previously!!!
>>>> >
>>>> > Our EJB's and code is all in jars prefixed with guru.jini and hence
>>>> they are not filtered out and the system works 100%.
>>>> >
>>>> > My sense is that there should be an easier way to specify what jars
>>>> and/or packages to process for annotations. I have come across the
>>>> following settings but can't really work out what fits where:
>>>> >
>>>> > openejb.additional.exclude
>>>> > openejb.additional.include
>>>> > openejb.deployments.classpath.filter.descriptors
>>>> > openejb.deployments.classpath.filter.systemapps
>>>> > openejb.deployments.classpath.include
>>>> > openejb.deployments.classpath.exclude
>>>> > openejb.deployments.package.include
>>>> > openejb.deployments.package.exclude
>>>> >
>>>> >
>>>> > P.S. Perhaps also add these to the standard exclusion list as they
>>>> are common and yet won't need to be processed for annotations I guess?
>>>> > com.amazonaws
>>>> > com.fasterxml
>>>> > com.google
>>>> > com.hazelcast
>>>> > io.grpc
>>>> > io.netty
>>>> > org.docx4j
>>>> > org.mongodb
>>>> > org.rocksdb
>>>> > org.asynchttpclient
>>>> > org.apache
>>>> > org.aspectj
>>>> >
>>>> > But then maybe some are too broad and I don't really understand what
>>>> the annotation index/cache is really needed for at runtime? Can it not be
>>>> lazy loaded or discarded if unused? Just thinking out loud here :-( Seems
>>>> like the cache uses a significant amount of heap when considering the drive
>>>> towards tiny micro services.
>>>> >
>>>> > Paul Carter-Brown
>>>> > Director
>>>> > Jini Guru
>>>> > m: +27 (0) 83 442 7179
>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>> Leslie
>>>> > Johannesburg, South Africa
>>>> > w: jini.guru e: paul@jini.guru
>>>> >
>>>> > Disclaimer: This message and/or attachment(s) may contain privileged,
>>>> confidential and/or personal information. If you are not the intended
>>>> recipient you may not disclose or distribute any of the information
>>>> contained within this message. In such case you must destroy this message
>>>> and inform the sender of the error. Jini Guru may not accept liability for
>>>> any errors, omissions, information and viruses contained in the
>>>> transmission of this message. Any opinions, conclusions and other
>>>> information contained within this message not related to Jini Guru official
>>>> business is deemed to be that of the individual only and is not endorsed by
>>>> Jini Guru.
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On Wed, Dec 11, 2019 at 7:18 PM Paul Carter-Brown
>>>> <pa...@jini.guru> wrote:
>>>> > Hi Jon,
>>>> >
>>>> > Unfortunately the snapshot behaves exactly the same way
>>>> >
>>>> > Paul Carter-Brown
>>>> > Director
>>>> > Jini Guru
>>>> > m: +27 (0) 83 442 7179
>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>> Leslie
>>>> > Johannesburg, South Africa
>>>> > w: jini.guru e: paul@jini.guru
>>>> >
>>>> > Disclaimer: This message and/or attachment(s) may contain privileged,
>>>> confidential and/or personal information. If you are not the intended
>>>> recipient you may not disclose or distribute any of the information
>>>> contained within this message. In such case you must destroy this message
>>>> and inform the sender of the error. Jini Guru may not accept liability for
>>>> any errors, omissions, information and viruses contained in the
>>>> transmission of this message. Any opinions, conclusions and other
>>>> information contained within this message not related to Jini Guru official
>>>> business is deemed to be that of the individual only and is not endorsed by
>>>> Jini Guru.
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On Wed, Dec 11, 2019 at 3:39 PM Jonathan Gallimore <
>>>> jonathan.gallimore@gmail.com> wrote:
>>>> > Hi Paul
>>>> >
>>>> > Would you mind trying your application with a recent snapshot?
>>>> >
>>>> https://repository.apache.org/content/repositories/snapshots/org/apache/tomee/apache-tomee/8.0.1-SNAPSHOT/
>>>> .
>>>> > I recently updated the exclude list.
>>>> >
>>>> > Many thanks
>>>> >
>>>> > Jon
>>>> >
>>>> > On Wed, Dec 11, 2019 at 12:31 PM Paul Carter-Brown
>>>> > <pa...@jini.guru> wrote:
>>>> >
>>>> > > Hi,
>>>> > >
>>>> > > I've been trying to lower the memory footprint of an EAR deployed
>>>> to TomEE
>>>> > > 8.0.0.
>>>> > >
>>>> > > Here is a screenshot of the heap histogram. The total Old Gen is
>>>> about
>>>> > > 450MB (after forcing multiple GC's). If I boot TomEE without my EAR
>>>> then
>>>> > > the old gen is about 6MB.
>>>> > >
>>>> > > [image: Screenshot from 2019-12-11 12-53-12.png]
>>>> > >
>>>> > > The entire ear is 140MB zip, most of which is in the ears /lib
>>>> directory
>>>> > > which contains libs such as Kafka, hazelcast, apache POI, Google
>>>> cloud
>>>> > > APIs, AWS client APIs etc etc. In total our code has about 100
>>>> actual
>>>> > > EJB's. If i remove files from the lib folder in the ear then I can
>>>> see that
>>>> > > the memory used by the annotation finder is lowered.
>>>> > >
>>>> > > Is there any way I can tell TomEE that it need not bother scanning
>>>> > > everything in the /lib folder of my EAR for annotations and fulling
>>>> up the
>>>> > > heap. I already
>>>> > > set openejb.deployments.classpath.include=.*jg-arch-core-impl.* to
>>>> scan
>>>> > > only the one jar in /lib which does have EJB's in it and it seems
>>>> to obey
>>>> > > this property but it doesn't seem to mean that annotation
>>>> processing is
>>>> > > skipped for all these other jars in /lib
>>>> > >
>>>> > > Thanks!
>>>> > >
>>>> > > Paul Carter-Brown
>>>> > > Director
>>>> > > Jini Guru
>>>> > > m: +27 (0) 83 442 7179 <+27834427179>
>>>> > > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>> Leslie
>>>> > > Johannesburg, South Africa
>>>> > > w: jini.guru e: paul@jini.guru
>>>> > >
>>>> > > Disclaimer: This message and/or attachment(s) may contain
>>>> > > privileged, confidential and/or personal information. If you are
>>>> not the
>>>> > > intended recipient you may not disclose or distribute any of
>>>> > > the information contained within this message. In such case you must
>>>> > > destroy this message and inform the sender of the error. Jini Guru
>>>> may not
>>>> > > accept liability for any errors, omissions, information and viruses
>>>> > > contained in the transmission of this message. Any opinions,
>>>> conclusions
>>>> > > and other information contained within this message not related to
>>>> Jini
>>>> > > Guru official business is deemed to be that of the individual only
>>>> and is
>>>> > > not endorsed by Jini Guru.
>>>> > >
>>>> > >
>>>>
>>>>
Re: Old Gen Full of Annotation Finder Index
Posted by Jonathan Gallimore <jo...@gmail.com>.
Hi Paul
Thanks for your feedback, that's great news. FYI, TomEE 8.0.1 is up for
vote (feel free to vote on it!), and this change is included.
Cheers
Jon
On Fri, Dec 20, 2019 at 7:10 AM Paul Carter-Brown
<pa...@jini.guru> wrote:
> Hi Jon
>
> Latest snapshot worked a charm. Old gen went from 500MB to 110MB after
> deploy completed. Thanks so much
>
> On Tue, 17 Dec 2019, 11:33 Paul Carter-Brown, <pa...@jini.guru>
> wrote:
>
>> Thanks Jon.
>>
>> I'm away at the moment but will give it a spin in the next few days and
>> let you know. Thanks again
>>
>> On Mon, 16 Dec 2019, 22:53 Jonathan Gallimore, <
>> jonathan.gallimore@gmail.com> wrote:
>>
>>> Hi Paul
>>>
>>> I've pushed a new snapshot which (hopefully) should address this. The CI
>>> build is running on it now. Would you mind giving it a try? If it looks
>>> good, I'll finish rolling the release. Fingers crossed it looks ok for you.
>>>
>>> Jon
>>>
>>> On Mon, Dec 16, 2019 at 2:48 PM Jonathan Gallimore <
>>> jonathan.gallimore@gmail.com> wrote:
>>>
>>>> Further update - I think my changes work with respect to this issue,
>>>> but I do have a bunch of unit test failures which I need to look into. For
>>>> info, here's the diff:
>>>>
>>>> diff --git
>>>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>>>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>>>> index a6e2bdc4f5..c0d1a8c98f 100644
>>>> ---
>>>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>>>> +++
>>>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>>>> @@ -69,10 +69,7 @@ import org.apache.openejb.cdi.ThreadSingletonService;
>>>> import org.apache.openejb.classloader.ClassLoaderConfigurer;
>>>> import org.apache.openejb.classloader.CompositeClassLoaderConfigurer;
>>>> import org.apache.openejb.component.ClassLoaderEnricher;
>>>> -import org.apache.openejb.config.ConfigurationFactory;
>>>> -import org.apache.openejb.config.NewLoaderLogic;
>>>> -import org.apache.openejb.config.QuickJarsTxtParser;
>>>> -import org.apache.openejb.config.TldScanner;
>>>> +import org.apache.openejb.config.*;
>>>> import org.apache.openejb.core.ConnectorReference;
>>>> import org.apache.openejb.core.CoreContainerSystem;
>>>> import org.apache.openejb.core.CoreUserTransaction;
>>>> @@ -829,6 +826,14 @@ public class Assembler extends AssemblerTool
>>>> implements org.apache.openejb.spi.A
>>>>
>>>> final AppContext appContext = new
>>>> AppContext(appInfo.appId, SystemInstance.get(), classLoader,
>>>> globalJndiContext, appJndiContext, appInfo.standaloneModule);
>>>> appContext.getProperties().putAll(appInfo.properties);
>>>> +
>>>> + final Set<Object> keys =
>>>> appContext.getProperties().keySet();
>>>> + for (final Object key : keys) {
>>>> + if (! Module.class.isInstance(key)) {
>>>> + appContext.getProperties().remove(key);
>>>> + }
>>>> + }
>>>> +
>>>> appContext.getInjections().addAll(injections);
>>>> appContext.getBindings().putAll(globalBindings);
>>>> appContext.getBindings().putAll(appBindings);
>>>> diff --git
>>>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>>>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>>>> index feff954d40..aed0fda941 100644
>>>> ---
>>>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>>>> +++
>>>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>>>> @@ -384,12 +384,7 @@ public class JndiEncBuilder {
>>>> reference = new LinkRef(jndiName);
>>>>
>>>> } else if (BeanManager.class.equals(type)) {
>>>> - reference = new LazyObjectReference<BeanManager>(new
>>>> Callable<BeanManager>() {
>>>> - @Override
>>>> - public BeanManager call() throws Exception {
>>>> - return new
>>>> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
>>>> - }
>>>> - });
>>>> + reference = new LazyObjectReference<>(new
>>>> BeanManagerLazyReference());
>>>>
>>>> } else if (UserTransaction.class.equals(type)) {
>>>> reference = new
>>>> IntraVmJndiReference("comp/UserTransaction");
>>>> @@ -684,4 +679,11 @@ public class JndiEncBuilder {
>>>> }
>>>> }
>>>> }
>>>> +
>>>> + public static class BeanManagerLazyReference implements
>>>> Callable<BeanManager> {
>>>> + @Override
>>>> + public BeanManager call() throws Exception {
>>>> + return new
>>>> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
>>>> + }
>>>> + }
>>>> }
>>>> diff --git
>>>> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>>>> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>>>> index 2cf387e112..285007925f 100644
>>>> ---
>>>> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>>>> +++
>>>> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>>>> @@ -59,6 +59,7 @@ public class TempClassLoader extends URLClassLoader {
>>>> private final ClassLoader system;
>>>> private final boolean embedded;
>>>> private final boolean parentURLClassLoader;
>>>> + private final StackTraceElement[] created;
>>>>
>>>> // 80% of class files are smaller then 6k
>>>> private final ByteArrayOutputStream bout = new
>>>> ByteArrayOutputStream(6 * 1024);
>>>> @@ -69,6 +70,7 @@ public class TempClassLoader extends URLClassLoader {
>>>> this.system = ClassLoader.getSystemClassLoader();
>>>> this.embedded = this.getClass().getClassLoader() ==
>>>> this.system;
>>>> this.parentURLClassLoader =
>>>> URLClassLoader.class.isInstance(parent);
>>>> + this.created = new Throwable().getStackTrace();
>>>> }
>>>>
>>>> /*
>>>>
>>>> In essence, this does 2 things:
>>>>
>>>> 1. Defines the Callable for the LazyObjectReference for BeanManager as
>>>> a static class. This stops that object from having a reference to
>>>> JndiEncBuilder and everything that has attached to it.
>>>> 2. Strips off the EjbModule / WebModule from the SuperProperties on
>>>> AppContext. This is set when deploying through the apps folder and lives
>>>> forever. It has the TempClassLoader attached, and therefore all the
>>>> AnnotationFinder stuff.
>>>>
>>>> I suspect (2) is too aggressive and is causing the test failures, but
>>>> will confirm.
>>>>
>>>> Jon
>>>>
>>>> On Mon, Dec 16, 2019 at 11:27 AM Jonathan Gallimore <
>>>> jonathan.gallimore@gmail.com> wrote:
>>>>
>>>>> Thanks for sending those over. I'm digging through this - I *think* I
>>>>> have pinned down a specific reference that stops that classloader from
>>>>> being released. The behaviour for a war/ear deployed in apps/ seems
>>>>> different to deploying in webapps/ - I'm assuming that's what you're doing.
>>>>> If you can confirm, that would help. I'll test out a patch with some bigger
>>>>> .ear files today.
>>>>>
>>>>> Thanks
>>>>>
>>>>> Jon
>>>>>
>>>>> On Thu, Dec 12, 2019 at 9:08 PM Jonathan Gallimore <
>>>>> jonathan.gallimore@gmail.com> wrote:
>>>>>
>>>>>> The mailing list seems to be stripping off your images - can you
>>>>>> resend and include my email address as well? I'd be keen to dig into this a
>>>>>> little more.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Jon
>>>>>>
>>>>>> On Thu, Dec 12, 2019 at 9:00 PM Paul Carter-Brown
>>>>>> <pa...@jini.guru> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> The code does seem to use org.apache.openejb.core.TempClassLoader. I
>>>>>>> can see one instance in the heap for every war in my ear. + 1.
>>>>>>>
>>>>>>> The TempClassLoader however still has 100's of strong references to
>>>>>>> it. E.g:
>>>>>>>
>>>>>>> [image: image.png]
>>>>>>>
>>>>>>> [image: image.png]
>>>>>>>
>>>>>>>
>>>>>>> Paul Carter-Brown
>>>>>>> Director
>>>>>>> Jini Guru
>>>>>>> m: +27 (0) 83 442 7179 <+27834427179>
>>>>>>> a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>>>> Leslie
>>>>>>> Johannesburg, South Africa
>>>>>>> w: jini.guru e: paul@jini.guru
>>>>>>>
>>>>>>> Disclaimer: This message and/or attachment(s) may contain
>>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>>> intended recipient you may not disclose or distribute any of
>>>>>>> the information contained within this message. In such case you must
>>>>>>> destroy this message and inform the sender of the error. Jini Guru may not
>>>>>>> accept liability for any errors, omissions, information and viruses
>>>>>>> contained in the transmission of this message. Any opinions, conclusions
>>>>>>> and other information contained within this message not related to Jini
>>>>>>> Guru official business is deemed to be that of the individual only and is
>>>>>>> not endorsed by Jini Guru.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Dec 12, 2019 at 9:29 AM Mark Struberg
>>>>>>> <st...@yahoo.de.invalid> wrote:
>>>>>>>
>>>>>>>> Hi!
>>>>>>>>
>>>>>>>> All this is just a workaround imo.
>>>>>>>>
>>>>>>>> Afair we (Romain) introduced a temporary throw-away ClassLoader for
>>>>>>>> scanning. That means after doing all the class scans we only keep the
>>>>>>>> extracted information but do not keep the Classes in memory. Doesn't this
>>>>>>>> work anymore?
>>>>>>>>
>>>>>>>> LieGrue,
>>>>>>>> strub
>>>>>>>>
>>>>>>>>
>>>>>>>> > Am 12.12.2019 um 00:45 schrieb Paul Carter-Brown
>>>>>>>> <pa...@jini.guru>:
>>>>>>>> >
>>>>>>>> > FYI. Graph of the change in heap size on a prod instance after
>>>>>>>> this change:
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > Paul Carter-Brown
>>>>>>>> > Director
>>>>>>>> > Jini Guru
>>>>>>>> > m: +27 (0) 83 442 7179
>>>>>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol
>>>>>>>> and Leslie
>>>>>>>> > Johannesburg, South Africa
>>>>>>>> > w: jini.guru e: paul@jini.guru
>>>>>>>> >
>>>>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>>>> intended recipient you may not disclose or distribute any of the
>>>>>>>> information contained within this message. In such case you must destroy
>>>>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>>>>> liability for any errors, omissions, information and viruses contained in
>>>>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>>>>> information contained within this message not related to Jini Guru official
>>>>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>>>>> Jini Guru.
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > On Thu, Dec 12, 2019 at 1:17 AM Paul Carter-Brown
>>>>>>>> <pa...@jini.guru> wrote:
>>>>>>>> > Hi Jon,
>>>>>>>> >
>>>>>>>> > I took a look at the source and worked out that I could add my
>>>>>>>> own filters using openejb.additional.exclude.
>>>>>>>> >
>>>>>>>> > I set it to openejb.additional.exclude=com,net,io,org and the JVM
>>>>>>>> heap now sits at around 150MB whereas it would never go below 450MB
>>>>>>>> previously!!!
>>>>>>>> >
>>>>>>>> > Our EJB's and code is all in jars prefixed with guru.jini and
>>>>>>>> hence they are not filtered out and the system works 100%.
>>>>>>>> >
>>>>>>>> > My sense is that there should be an easier way to specify what
>>>>>>>> jars and/or packages to process for annotations. I have come across the
>>>>>>>> following settings but can't really work out what fits where:
>>>>>>>> >
>>>>>>>> > openejb.additional.exclude
>>>>>>>> > openejb.additional.include
>>>>>>>> > openejb.deployments.classpath.filter.descriptors
>>>>>>>> > openejb.deployments.classpath.filter.systemapps
>>>>>>>> > openejb.deployments.classpath.include
>>>>>>>> > openejb.deployments.classpath.exclude
>>>>>>>> > openejb.deployments.package.include
>>>>>>>> > openejb.deployments.package.exclude
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > P.S. Perhaps also add these to the standard exclusion list as
>>>>>>>> they are common and yet won't need to be processed for annotations I guess?
>>>>>>>> > com.amazonaws
>>>>>>>> > com.fasterxml
>>>>>>>> > com.google
>>>>>>>> > com.hazelcast
>>>>>>>> > io.grpc
>>>>>>>> > io.netty
>>>>>>>> > org.docx4j
>>>>>>>> > org.mongodb
>>>>>>>> > org.rocksdb
>>>>>>>> > org.asynchttpclient
>>>>>>>> > org.apache
>>>>>>>> > org.aspectj
>>>>>>>> >
>>>>>>>> > But then maybe some are too broad and I don't really understand
>>>>>>>> what the annotation index/cache is really needed for at runtime? Can it not
>>>>>>>> be lazy loaded or discarded if unused? Just thinking out loud here :-(
>>>>>>>> Seems like the cache uses a significant amount of heap when considering the
>>>>>>>> drive towards tiny micro services.
>>>>>>>> >
>>>>>>>> > Paul Carter-Brown
>>>>>>>> > Director
>>>>>>>> > Jini Guru
>>>>>>>> > m: +27 (0) 83 442 7179
>>>>>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol
>>>>>>>> and Leslie
>>>>>>>> > Johannesburg, South Africa
>>>>>>>> > w: jini.guru e: paul@jini.guru
>>>>>>>> >
>>>>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>>>> intended recipient you may not disclose or distribute any of the
>>>>>>>> information contained within this message. In such case you must destroy
>>>>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>>>>> liability for any errors, omissions, information and viruses contained in
>>>>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>>>>> information contained within this message not related to Jini Guru official
>>>>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>>>>> Jini Guru.
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > On Wed, Dec 11, 2019 at 7:18 PM Paul Carter-Brown
>>>>>>>> <pa...@jini.guru> wrote:
>>>>>>>> > Hi Jon,
>>>>>>>> >
>>>>>>>> > Unfortunately the snapshot behaves exactly the same way
>>>>>>>> >
>>>>>>>> > Paul Carter-Brown
>>>>>>>> > Director
>>>>>>>> > Jini Guru
>>>>>>>> > m: +27 (0) 83 442 7179
>>>>>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol
>>>>>>>> and Leslie
>>>>>>>> > Johannesburg, South Africa
>>>>>>>> > w: jini.guru e: paul@jini.guru
>>>>>>>> >
>>>>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>>>> intended recipient you may not disclose or distribute any of the
>>>>>>>> information contained within this message. In such case you must destroy
>>>>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>>>>> liability for any errors, omissions, information and viruses contained in
>>>>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>>>>> information contained within this message not related to Jini Guru official
>>>>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>>>>> Jini Guru.
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > On Wed, Dec 11, 2019 at 3:39 PM Jonathan Gallimore <
>>>>>>>> jonathan.gallimore@gmail.com> wrote:
>>>>>>>> > Hi Paul
>>>>>>>> >
>>>>>>>> > Would you mind trying your application with a recent snapshot?
>>>>>>>> >
>>>>>>>> https://repository.apache.org/content/repositories/snapshots/org/apache/tomee/apache-tomee/8.0.1-SNAPSHOT/
>>>>>>>> .
>>>>>>>> > I recently updated the exclude list.
>>>>>>>> >
>>>>>>>> > Many thanks
>>>>>>>> >
>>>>>>>> > Jon
>>>>>>>> >
>>>>>>>> > On Wed, Dec 11, 2019 at 12:31 PM Paul Carter-Brown
>>>>>>>> > <pa...@jini.guru> wrote:
>>>>>>>> >
>>>>>>>> > > Hi,
>>>>>>>> > >
>>>>>>>> > > I've been trying to lower the memory footprint of an EAR
>>>>>>>> deployed to TomEE
>>>>>>>> > > 8.0.0.
>>>>>>>> > >
>>>>>>>> > > Here is a screenshot of the heap histogram. The total Old Gen
>>>>>>>> is about
>>>>>>>> > > 450MB (after forcing multiple GC's). If I boot TomEE without my
>>>>>>>> EAR then
>>>>>>>> > > the old gen is about 6MB.
>>>>>>>> > >
>>>>>>>> > > [image: Screenshot from 2019-12-11 12-53-12.png]
>>>>>>>> > >
>>>>>>>> > > The entire ear is 140MB zip, most of which is in the ears /lib
>>>>>>>> directory
>>>>>>>> > > which contains libs such as Kafka, hazelcast, apache POI,
>>>>>>>> Google cloud
>>>>>>>> > > APIs, AWS client APIs etc etc. In total our code has about 100
>>>>>>>> actual
>>>>>>>> > > EJB's. If i remove files from the lib folder in the ear then I
>>>>>>>> can see that
>>>>>>>> > > the memory used by the annotation finder is lowered.
>>>>>>>> > >
>>>>>>>> > > Is there any way I can tell TomEE that it need not bother
>>>>>>>> scanning
>>>>>>>> > > everything in the /lib folder of my EAR for annotations and
>>>>>>>> fulling up the
>>>>>>>> > > heap. I already
>>>>>>>> > > set openejb.deployments.classpath.include=.*jg-arch-core-impl.*
>>>>>>>> to scan
>>>>>>>> > > only the one jar in /lib which does have EJB's in it and it
>>>>>>>> seems to obey
>>>>>>>> > > this property but it doesn't seem to mean that annotation
>>>>>>>> processing is
>>>>>>>> > > skipped for all these other jars in /lib
>>>>>>>> > >
>>>>>>>> > > Thanks!
>>>>>>>> > >
>>>>>>>> > > Paul Carter-Brown
>>>>>>>> > > Director
>>>>>>>> > > Jini Guru
>>>>>>>> > > m: +27 (0) 83 442 7179 <+27834427179>
>>>>>>>> > > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol
>>>>>>>> and Leslie
>>>>>>>> > > Johannesburg, South Africa
>>>>>>>> > > w: jini.guru e: paul@jini.guru
>>>>>>>> > >
>>>>>>>> > > Disclaimer: This message and/or attachment(s) may contain
>>>>>>>> > > privileged, confidential and/or personal information. If you
>>>>>>>> are not the
>>>>>>>> > > intended recipient you may not disclose or distribute any of
>>>>>>>> > > the information contained within this message. In such case you
>>>>>>>> must
>>>>>>>> > > destroy this message and inform the sender of the error. Jini
>>>>>>>> Guru may not
>>>>>>>> > > accept liability for any errors, omissions, information and
>>>>>>>> viruses
>>>>>>>> > > contained in the transmission of this message. Any opinions,
>>>>>>>> conclusions
>>>>>>>> > > and other information contained within this message not related
>>>>>>>> to Jini
>>>>>>>> > > Guru official business is deemed to be that of the individual
>>>>>>>> only and is
>>>>>>>> > > not endorsed by Jini Guru.
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>>
>>>>>>>>
Re: Old Gen Full of Annotation Finder Index
Posted by Jonathan Gallimore <jo...@gmail.com>.
Hi Paul
Thanks for your feedback, that's great news. FYI, TomEE 8.0.1 is up for
vote (feel free to vote on it!), and this change is included.
Cheers
Jon
On Fri, Dec 20, 2019 at 7:10 AM Paul Carter-Brown
<pa...@jini.guru> wrote:
> Hi Jon
>
> Latest snapshot worked a charm. Old gen went from 500MB to 110MB after
> deploy completed. Thanks so much
>
> On Tue, 17 Dec 2019, 11:33 Paul Carter-Brown, <pa...@jini.guru>
> wrote:
>
>> Thanks Jon.
>>
>> I'm away at the moment but will give it a spin in the next few days and
>> let you know. Thanks again
>>
>> On Mon, 16 Dec 2019, 22:53 Jonathan Gallimore, <
>> jonathan.gallimore@gmail.com> wrote:
>>
>>> Hi Paul
>>>
>>> I've pushed a new snapshot which (hopefully) should address this. The CI
>>> build is running on it now. Would you mind giving it a try? If it looks
>>> good, I'll finish rolling the release. Fingers crossed it looks ok for you.
>>>
>>> Jon
>>>
>>> On Mon, Dec 16, 2019 at 2:48 PM Jonathan Gallimore <
>>> jonathan.gallimore@gmail.com> wrote:
>>>
>>>> Further update - I think my changes work with respect to this issue,
>>>> but I do have a bunch of unit test failures which I need to look into. For
>>>> info, here's the diff:
>>>>
>>>> diff --git
>>>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>>>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>>>> index a6e2bdc4f5..c0d1a8c98f 100644
>>>> ---
>>>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>>>> +++
>>>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>>>> @@ -69,10 +69,7 @@ import org.apache.openejb.cdi.ThreadSingletonService;
>>>> import org.apache.openejb.classloader.ClassLoaderConfigurer;
>>>> import org.apache.openejb.classloader.CompositeClassLoaderConfigurer;
>>>> import org.apache.openejb.component.ClassLoaderEnricher;
>>>> -import org.apache.openejb.config.ConfigurationFactory;
>>>> -import org.apache.openejb.config.NewLoaderLogic;
>>>> -import org.apache.openejb.config.QuickJarsTxtParser;
>>>> -import org.apache.openejb.config.TldScanner;
>>>> +import org.apache.openejb.config.*;
>>>> import org.apache.openejb.core.ConnectorReference;
>>>> import org.apache.openejb.core.CoreContainerSystem;
>>>> import org.apache.openejb.core.CoreUserTransaction;
>>>> @@ -829,6 +826,14 @@ public class Assembler extends AssemblerTool
>>>> implements org.apache.openejb.spi.A
>>>>
>>>> final AppContext appContext = new
>>>> AppContext(appInfo.appId, SystemInstance.get(), classLoader,
>>>> globalJndiContext, appJndiContext, appInfo.standaloneModule);
>>>> appContext.getProperties().putAll(appInfo.properties);
>>>> +
>>>> + final Set<Object> keys =
>>>> appContext.getProperties().keySet();
>>>> + for (final Object key : keys) {
>>>> + if (! Module.class.isInstance(key)) {
>>>> + appContext.getProperties().remove(key);
>>>> + }
>>>> + }
>>>> +
>>>> appContext.getInjections().addAll(injections);
>>>> appContext.getBindings().putAll(globalBindings);
>>>> appContext.getBindings().putAll(appBindings);
>>>> diff --git
>>>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>>>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>>>> index feff954d40..aed0fda941 100644
>>>> ---
>>>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>>>> +++
>>>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>>>> @@ -384,12 +384,7 @@ public class JndiEncBuilder {
>>>> reference = new LinkRef(jndiName);
>>>>
>>>> } else if (BeanManager.class.equals(type)) {
>>>> - reference = new LazyObjectReference<BeanManager>(new
>>>> Callable<BeanManager>() {
>>>> - @Override
>>>> - public BeanManager call() throws Exception {
>>>> - return new
>>>> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
>>>> - }
>>>> - });
>>>> + reference = new LazyObjectReference<>(new
>>>> BeanManagerLazyReference());
>>>>
>>>> } else if (UserTransaction.class.equals(type)) {
>>>> reference = new
>>>> IntraVmJndiReference("comp/UserTransaction");
>>>> @@ -684,4 +679,11 @@ public class JndiEncBuilder {
>>>> }
>>>> }
>>>> }
>>>> +
>>>> + public static class BeanManagerLazyReference implements
>>>> Callable<BeanManager> {
>>>> + @Override
>>>> + public BeanManager call() throws Exception {
>>>> + return new
>>>> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
>>>> + }
>>>> + }
>>>> }
>>>> diff --git
>>>> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>>>> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>>>> index 2cf387e112..285007925f 100644
>>>> ---
>>>> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>>>> +++
>>>> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>>>> @@ -59,6 +59,7 @@ public class TempClassLoader extends URLClassLoader {
>>>> private final ClassLoader system;
>>>> private final boolean embedded;
>>>> private final boolean parentURLClassLoader;
>>>> + private final StackTraceElement[] created;
>>>>
>>>> // 80% of class files are smaller then 6k
>>>> private final ByteArrayOutputStream bout = new
>>>> ByteArrayOutputStream(6 * 1024);
>>>> @@ -69,6 +70,7 @@ public class TempClassLoader extends URLClassLoader {
>>>> this.system = ClassLoader.getSystemClassLoader();
>>>> this.embedded = this.getClass().getClassLoader() ==
>>>> this.system;
>>>> this.parentURLClassLoader =
>>>> URLClassLoader.class.isInstance(parent);
>>>> + this.created = new Throwable().getStackTrace();
>>>> }
>>>>
>>>> /*
>>>>
>>>> In essence, this does 2 things:
>>>>
>>>> 1. Defines the Callable for the LazyObjectReference for BeanManager as
>>>> a static class. This stops that object from having a reference to
>>>> JndiEncBuilder and everything that has attached to it.
>>>> 2. Strips off the EjbModule / WebModule from the SuperProperties on
>>>> AppContext. This is set when deploying through the apps folder and lives
>>>> forever. It has the TempClassLoader attached, and therefore all the
>>>> AnnotationFinder stuff.
>>>>
>>>> I suspect (2) is too aggressive and is causing the test failures, but
>>>> will confirm.
>>>>
>>>> Jon
>>>>
>>>> On Mon, Dec 16, 2019 at 11:27 AM Jonathan Gallimore <
>>>> jonathan.gallimore@gmail.com> wrote:
>>>>
>>>>> Thanks for sending those over. I'm digging through this - I *think* I
>>>>> have pinned down a specific reference that stops that classloader from
>>>>> being released. The behaviour for a war/ear deployed in apps/ seems
>>>>> different to deploying in webapps/ - I'm assuming that's what you're doing.
>>>>> If you can confirm, that would help. I'll test out a patch with some bigger
>>>>> .ear files today.
>>>>>
>>>>> Thanks
>>>>>
>>>>> Jon
>>>>>
>>>>> On Thu, Dec 12, 2019 at 9:08 PM Jonathan Gallimore <
>>>>> jonathan.gallimore@gmail.com> wrote:
>>>>>
>>>>>> The mailing list seems to be stripping off your images - can you
>>>>>> resend and include my email address as well? I'd be keen to dig into this a
>>>>>> little more.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Jon
>>>>>>
>>>>>> On Thu, Dec 12, 2019 at 9:00 PM Paul Carter-Brown
>>>>>> <pa...@jini.guru> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> The code does seem to use org.apache.openejb.core.TempClassLoader. I
>>>>>>> can see one instance in the heap for every war in my ear. + 1.
>>>>>>>
>>>>>>> The TempClassLoader however still has 100's of strong references to
>>>>>>> it. E.g:
>>>>>>>
>>>>>>> [image: image.png]
>>>>>>>
>>>>>>> [image: image.png]
>>>>>>>
>>>>>>>
>>>>>>> Paul Carter-Brown
>>>>>>> Director
>>>>>>> Jini Guru
>>>>>>> m: +27 (0) 83 442 7179 <+27834427179>
>>>>>>> a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>>>> Leslie
>>>>>>> Johannesburg, South Africa
>>>>>>> w: jini.guru e: paul@jini.guru
>>>>>>>
>>>>>>> Disclaimer: This message and/or attachment(s) may contain
>>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>>> intended recipient you may not disclose or distribute any of
>>>>>>> the information contained within this message. In such case you must
>>>>>>> destroy this message and inform the sender of the error. Jini Guru may not
>>>>>>> accept liability for any errors, omissions, information and viruses
>>>>>>> contained in the transmission of this message. Any opinions, conclusions
>>>>>>> and other information contained within this message not related to Jini
>>>>>>> Guru official business is deemed to be that of the individual only and is
>>>>>>> not endorsed by Jini Guru.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Dec 12, 2019 at 9:29 AM Mark Struberg
>>>>>>> <st...@yahoo.de.invalid> wrote:
>>>>>>>
>>>>>>>> Hi!
>>>>>>>>
>>>>>>>> All this is just a workaround imo.
>>>>>>>>
>>>>>>>> Afair we (Romain) introduced a temporary throw-away ClassLoader for
>>>>>>>> scanning. That means after doing all the class scans we only keep the
>>>>>>>> extracted information but do not keep the Classes in memory. Doesn't this
>>>>>>>> work anymore?
>>>>>>>>
>>>>>>>> LieGrue,
>>>>>>>> strub
>>>>>>>>
>>>>>>>>
>>>>>>>> > Am 12.12.2019 um 00:45 schrieb Paul Carter-Brown
>>>>>>>> <pa...@jini.guru>:
>>>>>>>> >
>>>>>>>> > FYI. Graph of the change in heap size on a prod instance after
>>>>>>>> this change:
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > Paul Carter-Brown
>>>>>>>> > Director
>>>>>>>> > Jini Guru
>>>>>>>> > m: +27 (0) 83 442 7179
>>>>>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol
>>>>>>>> and Leslie
>>>>>>>> > Johannesburg, South Africa
>>>>>>>> > w: jini.guru e: paul@jini.guru
>>>>>>>> >
>>>>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>>>> intended recipient you may not disclose or distribute any of the
>>>>>>>> information contained within this message. In such case you must destroy
>>>>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>>>>> liability for any errors, omissions, information and viruses contained in
>>>>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>>>>> information contained within this message not related to Jini Guru official
>>>>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>>>>> Jini Guru.
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > On Thu, Dec 12, 2019 at 1:17 AM Paul Carter-Brown
>>>>>>>> <pa...@jini.guru> wrote:
>>>>>>>> > Hi Jon,
>>>>>>>> >
>>>>>>>> > I took a look at the source and worked out that I could add my
>>>>>>>> own filters using openejb.additional.exclude.
>>>>>>>> >
>>>>>>>> > I set it to openejb.additional.exclude=com,net,io,org and the JVM
>>>>>>>> heap now sits at around 150MB whereas it would never go below 450MB
>>>>>>>> previously!!!
>>>>>>>> >
>>>>>>>> > Our EJB's and code is all in jars prefixed with guru.jini and
>>>>>>>> hence they are not filtered out and the system works 100%.
>>>>>>>> >
>>>>>>>> > My sense is that there should be an easier way to specify what
>>>>>>>> jars and/or packages to process for annotations. I have come across the
>>>>>>>> following settings but can't really work out what fits where:
>>>>>>>> >
>>>>>>>> > openejb.additional.exclude
>>>>>>>> > openejb.additional.include
>>>>>>>> > openejb.deployments.classpath.filter.descriptors
>>>>>>>> > openejb.deployments.classpath.filter.systemapps
>>>>>>>> > openejb.deployments.classpath.include
>>>>>>>> > openejb.deployments.classpath.exclude
>>>>>>>> > openejb.deployments.package.include
>>>>>>>> > openejb.deployments.package.exclude
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > P.S. Perhaps also add these to the standard exclusion list as
>>>>>>>> they are common and yet won't need to be processed for annotations I guess?
>>>>>>>> > com.amazonaws
>>>>>>>> > com.fasterxml
>>>>>>>> > com.google
>>>>>>>> > com.hazelcast
>>>>>>>> > io.grpc
>>>>>>>> > io.netty
>>>>>>>> > org.docx4j
>>>>>>>> > org.mongodb
>>>>>>>> > org.rocksdb
>>>>>>>> > org.asynchttpclient
>>>>>>>> > org.apache
>>>>>>>> > org.aspectj
>>>>>>>> >
>>>>>>>> > But then maybe some are too broad and I don't really understand
>>>>>>>> what the annotation index/cache is really needed for at runtime? Can it not
>>>>>>>> be lazy loaded or discarded if unused? Just thinking out loud here :-(
>>>>>>>> Seems like the cache uses a significant amount of heap when considering the
>>>>>>>> drive towards tiny micro services.
>>>>>>>> >
>>>>>>>> > Paul Carter-Brown
>>>>>>>> > Director
>>>>>>>> > Jini Guru
>>>>>>>> > m: +27 (0) 83 442 7179
>>>>>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol
>>>>>>>> and Leslie
>>>>>>>> > Johannesburg, South Africa
>>>>>>>> > w: jini.guru e: paul@jini.guru
>>>>>>>> >
>>>>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>>>> intended recipient you may not disclose or distribute any of the
>>>>>>>> information contained within this message. In such case you must destroy
>>>>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>>>>> liability for any errors, omissions, information and viruses contained in
>>>>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>>>>> information contained within this message not related to Jini Guru official
>>>>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>>>>> Jini Guru.
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > On Wed, Dec 11, 2019 at 7:18 PM Paul Carter-Brown
>>>>>>>> <pa...@jini.guru> wrote:
>>>>>>>> > Hi Jon,
>>>>>>>> >
>>>>>>>> > Unfortunately the snapshot behaves exactly the same way
>>>>>>>> >
>>>>>>>> > Paul Carter-Brown
>>>>>>>> > Director
>>>>>>>> > Jini Guru
>>>>>>>> > m: +27 (0) 83 442 7179
>>>>>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol
>>>>>>>> and Leslie
>>>>>>>> > Johannesburg, South Africa
>>>>>>>> > w: jini.guru e: paul@jini.guru
>>>>>>>> >
>>>>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>>>> intended recipient you may not disclose or distribute any of the
>>>>>>>> information contained within this message. In such case you must destroy
>>>>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>>>>> liability for any errors, omissions, information and viruses contained in
>>>>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>>>>> information contained within this message not related to Jini Guru official
>>>>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>>>>> Jini Guru.
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > On Wed, Dec 11, 2019 at 3:39 PM Jonathan Gallimore <
>>>>>>>> jonathan.gallimore@gmail.com> wrote:
>>>>>>>> > Hi Paul
>>>>>>>> >
>>>>>>>> > Would you mind trying your application with a recent snapshot?
>>>>>>>> >
>>>>>>>> https://repository.apache.org/content/repositories/snapshots/org/apache/tomee/apache-tomee/8.0.1-SNAPSHOT/
>>>>>>>> .
>>>>>>>> > I recently updated the exclude list.
>>>>>>>> >
>>>>>>>> > Many thanks
>>>>>>>> >
>>>>>>>> > Jon
>>>>>>>> >
>>>>>>>> > On Wed, Dec 11, 2019 at 12:31 PM Paul Carter-Brown
>>>>>>>> > <pa...@jini.guru> wrote:
>>>>>>>> >
>>>>>>>> > > Hi,
>>>>>>>> > >
>>>>>>>> > > I've been trying to lower the memory footprint of an EAR
>>>>>>>> deployed to TomEE
>>>>>>>> > > 8.0.0.
>>>>>>>> > >
>>>>>>>> > > Here is a screenshot of the heap histogram. The total Old Gen
>>>>>>>> is about
>>>>>>>> > > 450MB (after forcing multiple GC's). If I boot TomEE without my
>>>>>>>> EAR then
>>>>>>>> > > the old gen is about 6MB.
>>>>>>>> > >
>>>>>>>> > > [image: Screenshot from 2019-12-11 12-53-12.png]
>>>>>>>> > >
>>>>>>>> > > The entire ear is 140MB zip, most of which is in the ears /lib
>>>>>>>> directory
>>>>>>>> > > which contains libs such as Kafka, hazelcast, apache POI,
>>>>>>>> Google cloud
>>>>>>>> > > APIs, AWS client APIs etc etc. In total our code has about 100
>>>>>>>> actual
>>>>>>>> > > EJB's. If i remove files from the lib folder in the ear then I
>>>>>>>> can see that
>>>>>>>> > > the memory used by the annotation finder is lowered.
>>>>>>>> > >
>>>>>>>> > > Is there any way I can tell TomEE that it need not bother
>>>>>>>> scanning
>>>>>>>> > > everything in the /lib folder of my EAR for annotations and
>>>>>>>> fulling up the
>>>>>>>> > > heap. I already
>>>>>>>> > > set openejb.deployments.classpath.include=.*jg-arch-core-impl.*
>>>>>>>> to scan
>>>>>>>> > > only the one jar in /lib which does have EJB's in it and it
>>>>>>>> seems to obey
>>>>>>>> > > this property but it doesn't seem to mean that annotation
>>>>>>>> processing is
>>>>>>>> > > skipped for all these other jars in /lib
>>>>>>>> > >
>>>>>>>> > > Thanks!
>>>>>>>> > >
>>>>>>>> > > Paul Carter-Brown
>>>>>>>> > > Director
>>>>>>>> > > Jini Guru
>>>>>>>> > > m: +27 (0) 83 442 7179 <+27834427179>
>>>>>>>> > > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol
>>>>>>>> and Leslie
>>>>>>>> > > Johannesburg, South Africa
>>>>>>>> > > w: jini.guru e: paul@jini.guru
>>>>>>>> > >
>>>>>>>> > > Disclaimer: This message and/or attachment(s) may contain
>>>>>>>> > > privileged, confidential and/or personal information. If you
>>>>>>>> are not the
>>>>>>>> > > intended recipient you may not disclose or distribute any of
>>>>>>>> > > the information contained within this message. In such case you
>>>>>>>> must
>>>>>>>> > > destroy this message and inform the sender of the error. Jini
>>>>>>>> Guru may not
>>>>>>>> > > accept liability for any errors, omissions, information and
>>>>>>>> viruses
>>>>>>>> > > contained in the transmission of this message. Any opinions,
>>>>>>>> conclusions
>>>>>>>> > > and other information contained within this message not related
>>>>>>>> to Jini
>>>>>>>> > > Guru official business is deemed to be that of the individual
>>>>>>>> only and is
>>>>>>>> > > not endorsed by Jini Guru.
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>>
>>>>>>>>
Re: Old Gen Full of Annotation Finder Index
Posted by Paul Carter-Brown <pa...@jini.guru>.
Hi Jon
Latest snapshot worked a charm. Old gen went from 500MB to 110MB after
deploy completed. Thanks so much
On Tue, 17 Dec 2019, 11:33 Paul Carter-Brown, <pa...@jini.guru>
wrote:
> Thanks Jon.
>
> I'm away at the moment but will give it a spin in the next few days and
> let you know. Thanks again
>
> On Mon, 16 Dec 2019, 22:53 Jonathan Gallimore, <
> jonathan.gallimore@gmail.com> wrote:
>
>> Hi Paul
>>
>> I've pushed a new snapshot which (hopefully) should address this. The CI
>> build is running on it now. Would you mind giving it a try? If it looks
>> good, I'll finish rolling the release. Fingers crossed it looks ok for you.
>>
>> Jon
>>
>> On Mon, Dec 16, 2019 at 2:48 PM Jonathan Gallimore <
>> jonathan.gallimore@gmail.com> wrote:
>>
>>> Further update - I think my changes work with respect to this issue, but
>>> I do have a bunch of unit test failures which I need to look into. For
>>> info, here's the diff:
>>>
>>> diff --git
>>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>>> index a6e2bdc4f5..c0d1a8c98f 100644
>>> ---
>>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>>> +++
>>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>>> @@ -69,10 +69,7 @@ import org.apache.openejb.cdi.ThreadSingletonService;
>>> import org.apache.openejb.classloader.ClassLoaderConfigurer;
>>> import org.apache.openejb.classloader.CompositeClassLoaderConfigurer;
>>> import org.apache.openejb.component.ClassLoaderEnricher;
>>> -import org.apache.openejb.config.ConfigurationFactory;
>>> -import org.apache.openejb.config.NewLoaderLogic;
>>> -import org.apache.openejb.config.QuickJarsTxtParser;
>>> -import org.apache.openejb.config.TldScanner;
>>> +import org.apache.openejb.config.*;
>>> import org.apache.openejb.core.ConnectorReference;
>>> import org.apache.openejb.core.CoreContainerSystem;
>>> import org.apache.openejb.core.CoreUserTransaction;
>>> @@ -829,6 +826,14 @@ public class Assembler extends AssemblerTool
>>> implements org.apache.openejb.spi.A
>>>
>>> final AppContext appContext = new
>>> AppContext(appInfo.appId, SystemInstance.get(), classLoader,
>>> globalJndiContext, appJndiContext, appInfo.standaloneModule);
>>> appContext.getProperties().putAll(appInfo.properties);
>>> +
>>> + final Set<Object> keys =
>>> appContext.getProperties().keySet();
>>> + for (final Object key : keys) {
>>> + if (! Module.class.isInstance(key)) {
>>> + appContext.getProperties().remove(key);
>>> + }
>>> + }
>>> +
>>> appContext.getInjections().addAll(injections);
>>> appContext.getBindings().putAll(globalBindings);
>>> appContext.getBindings().putAll(appBindings);
>>> diff --git
>>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>>> index feff954d40..aed0fda941 100644
>>> ---
>>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>>> +++
>>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>>> @@ -384,12 +384,7 @@ public class JndiEncBuilder {
>>> reference = new LinkRef(jndiName);
>>>
>>> } else if (BeanManager.class.equals(type)) {
>>> - reference = new LazyObjectReference<BeanManager>(new
>>> Callable<BeanManager>() {
>>> - @Override
>>> - public BeanManager call() throws Exception {
>>> - return new
>>> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
>>> - }
>>> - });
>>> + reference = new LazyObjectReference<>(new
>>> BeanManagerLazyReference());
>>>
>>> } else if (UserTransaction.class.equals(type)) {
>>> reference = new
>>> IntraVmJndiReference("comp/UserTransaction");
>>> @@ -684,4 +679,11 @@ public class JndiEncBuilder {
>>> }
>>> }
>>> }
>>> +
>>> + public static class BeanManagerLazyReference implements
>>> Callable<BeanManager> {
>>> + @Override
>>> + public BeanManager call() throws Exception {
>>> + return new
>>> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
>>> + }
>>> + }
>>> }
>>> diff --git
>>> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>>> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>>> index 2cf387e112..285007925f 100644
>>> ---
>>> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>>> +++
>>> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>>> @@ -59,6 +59,7 @@ public class TempClassLoader extends URLClassLoader {
>>> private final ClassLoader system;
>>> private final boolean embedded;
>>> private final boolean parentURLClassLoader;
>>> + private final StackTraceElement[] created;
>>>
>>> // 80% of class files are smaller then 6k
>>> private final ByteArrayOutputStream bout = new
>>> ByteArrayOutputStream(6 * 1024);
>>> @@ -69,6 +70,7 @@ public class TempClassLoader extends URLClassLoader {
>>> this.system = ClassLoader.getSystemClassLoader();
>>> this.embedded = this.getClass().getClassLoader() == this.system;
>>> this.parentURLClassLoader =
>>> URLClassLoader.class.isInstance(parent);
>>> + this.created = new Throwable().getStackTrace();
>>> }
>>>
>>> /*
>>>
>>> In essence, this does 2 things:
>>>
>>> 1. Defines the Callable for the LazyObjectReference for BeanManager as a
>>> static class. This stops that object from having a reference to
>>> JndiEncBuilder and everything that has attached to it.
>>> 2. Strips off the EjbModule / WebModule from the SuperProperties on
>>> AppContext. This is set when deploying through the apps folder and lives
>>> forever. It has the TempClassLoader attached, and therefore all the
>>> AnnotationFinder stuff.
>>>
>>> I suspect (2) is too aggressive and is causing the test failures, but
>>> will confirm.
>>>
>>> Jon
>>>
>>> On Mon, Dec 16, 2019 at 11:27 AM Jonathan Gallimore <
>>> jonathan.gallimore@gmail.com> wrote:
>>>
>>>> Thanks for sending those over. I'm digging through this - I *think* I
>>>> have pinned down a specific reference that stops that classloader from
>>>> being released. The behaviour for a war/ear deployed in apps/ seems
>>>> different to deploying in webapps/ - I'm assuming that's what you're doing.
>>>> If you can confirm, that would help. I'll test out a patch with some bigger
>>>> .ear files today.
>>>>
>>>> Thanks
>>>>
>>>> Jon
>>>>
>>>> On Thu, Dec 12, 2019 at 9:08 PM Jonathan Gallimore <
>>>> jonathan.gallimore@gmail.com> wrote:
>>>>
>>>>> The mailing list seems to be stripping off your images - can you
>>>>> resend and include my email address as well? I'd be keen to dig into this a
>>>>> little more.
>>>>>
>>>>> Thanks
>>>>>
>>>>> Jon
>>>>>
>>>>> On Thu, Dec 12, 2019 at 9:00 PM Paul Carter-Brown
>>>>> <pa...@jini.guru> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> The code does seem to use org.apache.openejb.core.TempClassLoader. I
>>>>>> can see one instance in the heap for every war in my ear. + 1.
>>>>>>
>>>>>> The TempClassLoader however still has 100's of strong references to
>>>>>> it. E.g:
>>>>>>
>>>>>> [image: image.png]
>>>>>>
>>>>>> [image: image.png]
>>>>>>
>>>>>>
>>>>>> Paul Carter-Brown
>>>>>> Director
>>>>>> Jini Guru
>>>>>> m: +27 (0) 83 442 7179 <+27834427179>
>>>>>> a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>>> Leslie
>>>>>> Johannesburg, South Africa
>>>>>> w: jini.guru e: paul@jini.guru
>>>>>>
>>>>>> Disclaimer: This message and/or attachment(s) may contain
>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>> intended recipient you may not disclose or distribute any of
>>>>>> the information contained within this message. In such case you must
>>>>>> destroy this message and inform the sender of the error. Jini Guru may not
>>>>>> accept liability for any errors, omissions, information and viruses
>>>>>> contained in the transmission of this message. Any opinions, conclusions
>>>>>> and other information contained within this message not related to Jini
>>>>>> Guru official business is deemed to be that of the individual only and is
>>>>>> not endorsed by Jini Guru.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Dec 12, 2019 at 9:29 AM Mark Struberg
>>>>>> <st...@yahoo.de.invalid> wrote:
>>>>>>
>>>>>>> Hi!
>>>>>>>
>>>>>>> All this is just a workaround imo.
>>>>>>>
>>>>>>> Afair we (Romain) introduced a temporary throw-away ClassLoader for
>>>>>>> scanning. That means after doing all the class scans we only keep the
>>>>>>> extracted information but do not keep the Classes in memory. Doesn't this
>>>>>>> work anymore?
>>>>>>>
>>>>>>> LieGrue,
>>>>>>> strub
>>>>>>>
>>>>>>>
>>>>>>> > Am 12.12.2019 um 00:45 schrieb Paul Carter-Brown
>>>>>>> <pa...@jini.guru>:
>>>>>>> >
>>>>>>> > FYI. Graph of the change in heap size on a prod instance after
>>>>>>> this change:
>>>>>>> >
>>>>>>> >
>>>>>>> > Paul Carter-Brown
>>>>>>> > Director
>>>>>>> > Jini Guru
>>>>>>> > m: +27 (0) 83 442 7179
>>>>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol
>>>>>>> and Leslie
>>>>>>> > Johannesburg, South Africa
>>>>>>> > w: jini.guru e: paul@jini.guru
>>>>>>> >
>>>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>>> intended recipient you may not disclose or distribute any of the
>>>>>>> information contained within this message. In such case you must destroy
>>>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>>>> liability for any errors, omissions, information and viruses contained in
>>>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>>>> information contained within this message not related to Jini Guru official
>>>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>>>> Jini Guru.
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > On Thu, Dec 12, 2019 at 1:17 AM Paul Carter-Brown
>>>>>>> <pa...@jini.guru> wrote:
>>>>>>> > Hi Jon,
>>>>>>> >
>>>>>>> > I took a look at the source and worked out that I could add my own
>>>>>>> filters using openejb.additional.exclude.
>>>>>>> >
>>>>>>> > I set it to openejb.additional.exclude=com,net,io,org and the JVM
>>>>>>> heap now sits at around 150MB whereas it would never go below 450MB
>>>>>>> previously!!!
>>>>>>> >
>>>>>>> > Our EJB's and code is all in jars prefixed with guru.jini and
>>>>>>> hence they are not filtered out and the system works 100%.
>>>>>>> >
>>>>>>> > My sense is that there should be an easier way to specify what
>>>>>>> jars and/or packages to process for annotations. I have come across the
>>>>>>> following settings but can't really work out what fits where:
>>>>>>> >
>>>>>>> > openejb.additional.exclude
>>>>>>> > openejb.additional.include
>>>>>>> > openejb.deployments.classpath.filter.descriptors
>>>>>>> > openejb.deployments.classpath.filter.systemapps
>>>>>>> > openejb.deployments.classpath.include
>>>>>>> > openejb.deployments.classpath.exclude
>>>>>>> > openejb.deployments.package.include
>>>>>>> > openejb.deployments.package.exclude
>>>>>>> >
>>>>>>> >
>>>>>>> > P.S. Perhaps also add these to the standard exclusion list as they
>>>>>>> are common and yet won't need to be processed for annotations I guess?
>>>>>>> > com.amazonaws
>>>>>>> > com.fasterxml
>>>>>>> > com.google
>>>>>>> > com.hazelcast
>>>>>>> > io.grpc
>>>>>>> > io.netty
>>>>>>> > org.docx4j
>>>>>>> > org.mongodb
>>>>>>> > org.rocksdb
>>>>>>> > org.asynchttpclient
>>>>>>> > org.apache
>>>>>>> > org.aspectj
>>>>>>> >
>>>>>>> > But then maybe some are too broad and I don't really understand
>>>>>>> what the annotation index/cache is really needed for at runtime? Can it not
>>>>>>> be lazy loaded or discarded if unused? Just thinking out loud here :-(
>>>>>>> Seems like the cache uses a significant amount of heap when considering the
>>>>>>> drive towards tiny micro services.
>>>>>>> >
>>>>>>> > Paul Carter-Brown
>>>>>>> > Director
>>>>>>> > Jini Guru
>>>>>>> > m: +27 (0) 83 442 7179
>>>>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol
>>>>>>> and Leslie
>>>>>>> > Johannesburg, South Africa
>>>>>>> > w: jini.guru e: paul@jini.guru
>>>>>>> >
>>>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>>> intended recipient you may not disclose or distribute any of the
>>>>>>> information contained within this message. In such case you must destroy
>>>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>>>> liability for any errors, omissions, information and viruses contained in
>>>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>>>> information contained within this message not related to Jini Guru official
>>>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>>>> Jini Guru.
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > On Wed, Dec 11, 2019 at 7:18 PM Paul Carter-Brown
>>>>>>> <pa...@jini.guru> wrote:
>>>>>>> > Hi Jon,
>>>>>>> >
>>>>>>> > Unfortunately the snapshot behaves exactly the same way
>>>>>>> >
>>>>>>> > Paul Carter-Brown
>>>>>>> > Director
>>>>>>> > Jini Guru
>>>>>>> > m: +27 (0) 83 442 7179
>>>>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol
>>>>>>> and Leslie
>>>>>>> > Johannesburg, South Africa
>>>>>>> > w: jini.guru e: paul@jini.guru
>>>>>>> >
>>>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>>> intended recipient you may not disclose or distribute any of the
>>>>>>> information contained within this message. In such case you must destroy
>>>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>>>> liability for any errors, omissions, information and viruses contained in
>>>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>>>> information contained within this message not related to Jini Guru official
>>>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>>>> Jini Guru.
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > On Wed, Dec 11, 2019 at 3:39 PM Jonathan Gallimore <
>>>>>>> jonathan.gallimore@gmail.com> wrote:
>>>>>>> > Hi Paul
>>>>>>> >
>>>>>>> > Would you mind trying your application with a recent snapshot?
>>>>>>> >
>>>>>>> https://repository.apache.org/content/repositories/snapshots/org/apache/tomee/apache-tomee/8.0.1-SNAPSHOT/
>>>>>>> .
>>>>>>> > I recently updated the exclude list.
>>>>>>> >
>>>>>>> > Many thanks
>>>>>>> >
>>>>>>> > Jon
>>>>>>> >
>>>>>>> > On Wed, Dec 11, 2019 at 12:31 PM Paul Carter-Brown
>>>>>>> > <pa...@jini.guru> wrote:
>>>>>>> >
>>>>>>> > > Hi,
>>>>>>> > >
>>>>>>> > > I've been trying to lower the memory footprint of an EAR
>>>>>>> deployed to TomEE
>>>>>>> > > 8.0.0.
>>>>>>> > >
>>>>>>> > > Here is a screenshot of the heap histogram. The total Old Gen is
>>>>>>> about
>>>>>>> > > 450MB (after forcing multiple GC's). If I boot TomEE without my
>>>>>>> EAR then
>>>>>>> > > the old gen is about 6MB.
>>>>>>> > >
>>>>>>> > > [image: Screenshot from 2019-12-11 12-53-12.png]
>>>>>>> > >
>>>>>>> > > The entire ear is 140MB zip, most of which is in the ears /lib
>>>>>>> directory
>>>>>>> > > which contains libs such as Kafka, hazelcast, apache POI, Google
>>>>>>> cloud
>>>>>>> > > APIs, AWS client APIs etc etc. In total our code has about 100
>>>>>>> actual
>>>>>>> > > EJB's. If i remove files from the lib folder in the ear then I
>>>>>>> can see that
>>>>>>> > > the memory used by the annotation finder is lowered.
>>>>>>> > >
>>>>>>> > > Is there any way I can tell TomEE that it need not bother
>>>>>>> scanning
>>>>>>> > > everything in the /lib folder of my EAR for annotations and
>>>>>>> fulling up the
>>>>>>> > > heap. I already
>>>>>>> > > set openejb.deployments.classpath.include=.*jg-arch-core-impl.*
>>>>>>> to scan
>>>>>>> > > only the one jar in /lib which does have EJB's in it and it
>>>>>>> seems to obey
>>>>>>> > > this property but it doesn't seem to mean that annotation
>>>>>>> processing is
>>>>>>> > > skipped for all these other jars in /lib
>>>>>>> > >
>>>>>>> > > Thanks!
>>>>>>> > >
>>>>>>> > > Paul Carter-Brown
>>>>>>> > > Director
>>>>>>> > > Jini Guru
>>>>>>> > > m: +27 (0) 83 442 7179 <+27834427179>
>>>>>>> > > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>>>> Leslie
>>>>>>> > > Johannesburg, South Africa
>>>>>>> > > w: jini.guru e: paul@jini.guru
>>>>>>> > >
>>>>>>> > > Disclaimer: This message and/or attachment(s) may contain
>>>>>>> > > privileged, confidential and/or personal information. If you are
>>>>>>> not the
>>>>>>> > > intended recipient you may not disclose or distribute any of
>>>>>>> > > the information contained within this message. In such case you
>>>>>>> must
>>>>>>> > > destroy this message and inform the sender of the error. Jini
>>>>>>> Guru may not
>>>>>>> > > accept liability for any errors, omissions, information and
>>>>>>> viruses
>>>>>>> > > contained in the transmission of this message. Any opinions,
>>>>>>> conclusions
>>>>>>> > > and other information contained within this message not related
>>>>>>> to Jini
>>>>>>> > > Guru official business is deemed to be that of the individual
>>>>>>> only and is
>>>>>>> > > not endorsed by Jini Guru.
>>>>>>> > >
>>>>>>> > >
>>>>>>>
>>>>>>>
Re: Old Gen Full of Annotation Finder Index
Posted by Paul Carter-Brown <pa...@jini.guru>.
Thanks Jon.
I'm away at the moment but will give it a spin in the next few days and
let you know. Thanks again
On Mon, 16 Dec 2019, 22:53 Jonathan Gallimore, <jo...@gmail.com>
wrote:
> Hi Paul
>
> I've pushed a new snapshot which (hopefully) should address this. The CI
> build is running on it now. Would you mind giving it a try? If it looks
> good, I'll finish rolling the release. Fingers crossed it looks ok for you.
>
> Jon
>
> On Mon, Dec 16, 2019 at 2:48 PM Jonathan Gallimore <
> jonathan.gallimore@gmail.com> wrote:
>
>> Further update - I think my changes work with respect to this issue, but
>> I do have a bunch of unit test failures which I need to look into. For
>> info, here's the diff:
>>
>> diff --git
>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>> index a6e2bdc4f5..c0d1a8c98f 100644
>> ---
>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>> +++
>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>> @@ -69,10 +69,7 @@ import org.apache.openejb.cdi.ThreadSingletonService;
>> import org.apache.openejb.classloader.ClassLoaderConfigurer;
>> import org.apache.openejb.classloader.CompositeClassLoaderConfigurer;
>> import org.apache.openejb.component.ClassLoaderEnricher;
>> -import org.apache.openejb.config.ConfigurationFactory;
>> -import org.apache.openejb.config.NewLoaderLogic;
>> -import org.apache.openejb.config.QuickJarsTxtParser;
>> -import org.apache.openejb.config.TldScanner;
>> +import org.apache.openejb.config.*;
>> import org.apache.openejb.core.ConnectorReference;
>> import org.apache.openejb.core.CoreContainerSystem;
>> import org.apache.openejb.core.CoreUserTransaction;
>> @@ -829,6 +826,14 @@ public class Assembler extends AssemblerTool
>> implements org.apache.openejb.spi.A
>>
>> final AppContext appContext = new
>> AppContext(appInfo.appId, SystemInstance.get(), classLoader,
>> globalJndiContext, appJndiContext, appInfo.standaloneModule);
>> appContext.getProperties().putAll(appInfo.properties);
>> +
>> + final Set<Object> keys =
>> appContext.getProperties().keySet();
>> + for (final Object key : keys) {
>> + if (! Module.class.isInstance(key)) {
>> + appContext.getProperties().remove(key);
>> + }
>> + }
>> +
>> appContext.getInjections().addAll(injections);
>> appContext.getBindings().putAll(globalBindings);
>> appContext.getBindings().putAll(appBindings);
>> diff --git
>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>> index feff954d40..aed0fda941 100644
>> ---
>> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>> +++
>> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>> @@ -384,12 +384,7 @@ public class JndiEncBuilder {
>> reference = new LinkRef(jndiName);
>>
>> } else if (BeanManager.class.equals(type)) {
>> - reference = new LazyObjectReference<BeanManager>(new
>> Callable<BeanManager>() {
>> - @Override
>> - public BeanManager call() throws Exception {
>> - return new
>> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
>> - }
>> - });
>> + reference = new LazyObjectReference<>(new
>> BeanManagerLazyReference());
>>
>> } else if (UserTransaction.class.equals(type)) {
>> reference = new
>> IntraVmJndiReference("comp/UserTransaction");
>> @@ -684,4 +679,11 @@ public class JndiEncBuilder {
>> }
>> }
>> }
>> +
>> + public static class BeanManagerLazyReference implements
>> Callable<BeanManager> {
>> + @Override
>> + public BeanManager call() throws Exception {
>> + return new
>> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
>> + }
>> + }
>> }
>> diff --git
>> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>> index 2cf387e112..285007925f 100644
>> ---
>> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>> +++
>> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>> @@ -59,6 +59,7 @@ public class TempClassLoader extends URLClassLoader {
>> private final ClassLoader system;
>> private final boolean embedded;
>> private final boolean parentURLClassLoader;
>> + private final StackTraceElement[] created;
>>
>> // 80% of class files are smaller then 6k
>> private final ByteArrayOutputStream bout = new
>> ByteArrayOutputStream(6 * 1024);
>> @@ -69,6 +70,7 @@ public class TempClassLoader extends URLClassLoader {
>> this.system = ClassLoader.getSystemClassLoader();
>> this.embedded = this.getClass().getClassLoader() == this.system;
>> this.parentURLClassLoader =
>> URLClassLoader.class.isInstance(parent);
>> + this.created = new Throwable().getStackTrace();
>> }
>>
>> /*
>>
>> In essence, this does 2 things:
>>
>> 1. Defines the Callable for the LazyObjectReference for BeanManager as a
>> static class. This stops that object from having a reference to
>> JndiEncBuilder and everything that has attached to it.
>> 2. Strips off the EjbModule / WebModule from the SuperProperties on
>> AppContext. This is set when deploying through the apps folder and lives
>> forever. It has the TempClassLoader attached, and therefore all the
>> AnnotationFinder stuff.
>>
>> I suspect (2) is too aggressive and is causing the test failures, but
>> will confirm.
>>
>> Jon
>>
>> On Mon, Dec 16, 2019 at 11:27 AM Jonathan Gallimore <
>> jonathan.gallimore@gmail.com> wrote:
>>
>>> Thanks for sending those over. I'm digging through this - I *think* I
>>> have pinned down a specific reference that stops that classloader from
>>> being released. The behaviour for a war/ear deployed in apps/ seems
>>> different to deploying in webapps/ - I'm assuming that's what you're doing.
>>> If you can confirm, that would help. I'll test out a patch with some bigger
>>> .ear files today.
>>>
>>> Thanks
>>>
>>> Jon
>>>
>>> On Thu, Dec 12, 2019 at 9:08 PM Jonathan Gallimore <
>>> jonathan.gallimore@gmail.com> wrote:
>>>
>>>> The mailing list seems to be stripping off your images - can you resend
>>>> and include my email address as well? I'd be keen to dig into this a little
>>>> more.
>>>>
>>>> Thanks
>>>>
>>>> Jon
>>>>
>>>> On Thu, Dec 12, 2019 at 9:00 PM Paul Carter-Brown
>>>> <pa...@jini.guru> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> The code does seem to use org.apache.openejb.core.TempClassLoader. I
>>>>> can see one instance in the heap for every war in my ear. + 1.
>>>>>
>>>>> The TempClassLoader however still has 100's of strong references to
>>>>> it. E.g:
>>>>>
>>>>> [image: image.png]
>>>>>
>>>>> [image: image.png]
>>>>>
>>>>>
>>>>> Paul Carter-Brown
>>>>> Director
>>>>> Jini Guru
>>>>> m: +27 (0) 83 442 7179 <+27834427179>
>>>>> a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>> Leslie
>>>>> Johannesburg, South Africa
>>>>> w: jini.guru e: paul@jini.guru
>>>>>
>>>>> Disclaimer: This message and/or attachment(s) may contain
>>>>> privileged, confidential and/or personal information. If you are not the
>>>>> intended recipient you may not disclose or distribute any of
>>>>> the information contained within this message. In such case you must
>>>>> destroy this message and inform the sender of the error. Jini Guru may not
>>>>> accept liability for any errors, omissions, information and viruses
>>>>> contained in the transmission of this message. Any opinions, conclusions
>>>>> and other information contained within this message not related to Jini
>>>>> Guru official business is deemed to be that of the individual only and is
>>>>> not endorsed by Jini Guru.
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Dec 12, 2019 at 9:29 AM Mark Struberg
>>>>> <st...@yahoo.de.invalid> wrote:
>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> All this is just a workaround imo.
>>>>>>
>>>>>> Afair we (Romain) introduced a temporary throw-away ClassLoader for
>>>>>> scanning. That means after doing all the class scans we only keep the
>>>>>> extracted information but do not keep the Classes in memory. Doesn't this
>>>>>> work anymore?
>>>>>>
>>>>>> LieGrue,
>>>>>> strub
>>>>>>
>>>>>>
>>>>>> > Am 12.12.2019 um 00:45 schrieb Paul Carter-Brown
>>>>>> <pa...@jini.guru>:
>>>>>> >
>>>>>> > FYI. Graph of the change in heap size on a prod instance after this
>>>>>> change:
>>>>>> >
>>>>>> >
>>>>>> > Paul Carter-Brown
>>>>>> > Director
>>>>>> > Jini Guru
>>>>>> > m: +27 (0) 83 442 7179
>>>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>>> Leslie
>>>>>> > Johannesburg, South Africa
>>>>>> > w: jini.guru e: paul@jini.guru
>>>>>> >
>>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>> intended recipient you may not disclose or distribute any of the
>>>>>> information contained within this message. In such case you must destroy
>>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>>> liability for any errors, omissions, information and viruses contained in
>>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>>> information contained within this message not related to Jini Guru official
>>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>>> Jini Guru.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > On Thu, Dec 12, 2019 at 1:17 AM Paul Carter-Brown
>>>>>> <pa...@jini.guru> wrote:
>>>>>> > Hi Jon,
>>>>>> >
>>>>>> > I took a look at the source and worked out that I could add my own
>>>>>> filters using openejb.additional.exclude.
>>>>>> >
>>>>>> > I set it to openejb.additional.exclude=com,net,io,org and the JVM
>>>>>> heap now sits at around 150MB whereas it would never go below 450MB
>>>>>> previously!!!
>>>>>> >
>>>>>> > Our EJB's and code is all in jars prefixed with guru.jini and hence
>>>>>> they are not filtered out and the system works 100%.
>>>>>> >
>>>>>> > My sense is that there should be an easier way to specify what jars
>>>>>> and/or packages to process for annotations. I have come across the
>>>>>> following settings but can't really work out what fits where:
>>>>>> >
>>>>>> > openejb.additional.exclude
>>>>>> > openejb.additional.include
>>>>>> > openejb.deployments.classpath.filter.descriptors
>>>>>> > openejb.deployments.classpath.filter.systemapps
>>>>>> > openejb.deployments.classpath.include
>>>>>> > openejb.deployments.classpath.exclude
>>>>>> > openejb.deployments.package.include
>>>>>> > openejb.deployments.package.exclude
>>>>>> >
>>>>>> >
>>>>>> > P.S. Perhaps also add these to the standard exclusion list as they
>>>>>> are common and yet won't need to be processed for annotations I guess?
>>>>>> > com.amazonaws
>>>>>> > com.fasterxml
>>>>>> > com.google
>>>>>> > com.hazelcast
>>>>>> > io.grpc
>>>>>> > io.netty
>>>>>> > org.docx4j
>>>>>> > org.mongodb
>>>>>> > org.rocksdb
>>>>>> > org.asynchttpclient
>>>>>> > org.apache
>>>>>> > org.aspectj
>>>>>> >
>>>>>> > But then maybe some are too broad and I don't really understand
>>>>>> what the annotation index/cache is really needed for at runtime? Can it not
>>>>>> be lazy loaded or discarded if unused? Just thinking out loud here :-(
>>>>>> Seems like the cache uses a significant amount of heap when considering the
>>>>>> drive towards tiny micro services.
>>>>>> >
>>>>>> > Paul Carter-Brown
>>>>>> > Director
>>>>>> > Jini Guru
>>>>>> > m: +27 (0) 83 442 7179
>>>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>>> Leslie
>>>>>> > Johannesburg, South Africa
>>>>>> > w: jini.guru e: paul@jini.guru
>>>>>> >
>>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>> intended recipient you may not disclose or distribute any of the
>>>>>> information contained within this message. In such case you must destroy
>>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>>> liability for any errors, omissions, information and viruses contained in
>>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>>> information contained within this message not related to Jini Guru official
>>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>>> Jini Guru.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > On Wed, Dec 11, 2019 at 7:18 PM Paul Carter-Brown
>>>>>> <pa...@jini.guru> wrote:
>>>>>> > Hi Jon,
>>>>>> >
>>>>>> > Unfortunately the snapshot behaves exactly the same way
>>>>>> >
>>>>>> > Paul Carter-Brown
>>>>>> > Director
>>>>>> > Jini Guru
>>>>>> > m: +27 (0) 83 442 7179
>>>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>>> Leslie
>>>>>> > Johannesburg, South Africa
>>>>>> > w: jini.guru e: paul@jini.guru
>>>>>> >
>>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>>> privileged, confidential and/or personal information. If you are not the
>>>>>> intended recipient you may not disclose or distribute any of the
>>>>>> information contained within this message. In such case you must destroy
>>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>>> liability for any errors, omissions, information and viruses contained in
>>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>>> information contained within this message not related to Jini Guru official
>>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>>> Jini Guru.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > On Wed, Dec 11, 2019 at 3:39 PM Jonathan Gallimore <
>>>>>> jonathan.gallimore@gmail.com> wrote:
>>>>>> > Hi Paul
>>>>>> >
>>>>>> > Would you mind trying your application with a recent snapshot?
>>>>>> >
>>>>>> https://repository.apache.org/content/repositories/snapshots/org/apache/tomee/apache-tomee/8.0.1-SNAPSHOT/
>>>>>> .
>>>>>> > I recently updated the exclude list.
>>>>>> >
>>>>>> > Many thanks
>>>>>> >
>>>>>> > Jon
>>>>>> >
>>>>>> > On Wed, Dec 11, 2019 at 12:31 PM Paul Carter-Brown
>>>>>> > <pa...@jini.guru> wrote:
>>>>>> >
>>>>>> > > Hi,
>>>>>> > >
>>>>>> > > I've been trying to lower the memory footprint of an EAR deployed
>>>>>> to TomEE
>>>>>> > > 8.0.0.
>>>>>> > >
>>>>>> > > Here is a screenshot of the heap histogram. The total Old Gen is
>>>>>> about
>>>>>> > > 450MB (after forcing multiple GC's). If I boot TomEE without my
>>>>>> EAR then
>>>>>> > > the old gen is about 6MB.
>>>>>> > >
>>>>>> > > [image: Screenshot from 2019-12-11 12-53-12.png]
>>>>>> > >
>>>>>> > > The entire ear is 140MB zip, most of which is in the ears /lib
>>>>>> directory
>>>>>> > > which contains libs such as Kafka, hazelcast, apache POI, Google
>>>>>> cloud
>>>>>> > > APIs, AWS client APIs etc etc. In total our code has about 100
>>>>>> actual
>>>>>> > > EJB's. If i remove files from the lib folder in the ear then I
>>>>>> can see that
>>>>>> > > the memory used by the annotation finder is lowered.
>>>>>> > >
>>>>>> > > Is there any way I can tell TomEE that it need not bother scanning
>>>>>> > > everything in the /lib folder of my EAR for annotations and
>>>>>> fulling up the
>>>>>> > > heap. I already
>>>>>> > > set openejb.deployments.classpath.include=.*jg-arch-core-impl.*
>>>>>> to scan
>>>>>> > > only the one jar in /lib which does have EJB's in it and it seems
>>>>>> to obey
>>>>>> > > this property but it doesn't seem to mean that annotation
>>>>>> processing is
>>>>>> > > skipped for all these other jars in /lib
>>>>>> > >
>>>>>> > > Thanks!
>>>>>> > >
>>>>>> > > Paul Carter-Brown
>>>>>> > > Director
>>>>>> > > Jini Guru
>>>>>> > > m: +27 (0) 83 442 7179 <+27834427179>
>>>>>> > > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>>> Leslie
>>>>>> > > Johannesburg, South Africa
>>>>>> > > w: jini.guru e: paul@jini.guru
>>>>>> > >
>>>>>> > > Disclaimer: This message and/or attachment(s) may contain
>>>>>> > > privileged, confidential and/or personal information. If you are
>>>>>> not the
>>>>>> > > intended recipient you may not disclose or distribute any of
>>>>>> > > the information contained within this message. In such case you
>>>>>> must
>>>>>> > > destroy this message and inform the sender of the error. Jini
>>>>>> Guru may not
>>>>>> > > accept liability for any errors, omissions, information and
>>>>>> viruses
>>>>>> > > contained in the transmission of this message. Any opinions,
>>>>>> conclusions
>>>>>> > > and other information contained within this message not related
>>>>>> to Jini
>>>>>> > > Guru official business is deemed to be that of the individual
>>>>>> only and is
>>>>>> > > not endorsed by Jini Guru.
>>>>>> > >
>>>>>> > >
>>>>>>
>>>>>>
Re: Old Gen Full of Annotation Finder Index
Posted by Jonathan Gallimore <jo...@gmail.com>.
Hi Paul
I've pushed a new snapshot which (hopefully) should address this. The CI
build is running on it now. Would you mind giving it a try? If it looks
good, I'll finish rolling the release. Fingers crossed it looks ok for you.
Jon
On Mon, Dec 16, 2019 at 2:48 PM Jonathan Gallimore <
jonathan.gallimore@gmail.com> wrote:
> Further update - I think my changes work with respect to this issue, but I
> do have a bunch of unit test failures which I need to look into. For info,
> here's the diff:
>
> diff --git
> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
> index a6e2bdc4f5..c0d1a8c98f 100644
> ---
> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
> +++
> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
> @@ -69,10 +69,7 @@ import org.apache.openejb.cdi.ThreadSingletonService;
> import org.apache.openejb.classloader.ClassLoaderConfigurer;
> import org.apache.openejb.classloader.CompositeClassLoaderConfigurer;
> import org.apache.openejb.component.ClassLoaderEnricher;
> -import org.apache.openejb.config.ConfigurationFactory;
> -import org.apache.openejb.config.NewLoaderLogic;
> -import org.apache.openejb.config.QuickJarsTxtParser;
> -import org.apache.openejb.config.TldScanner;
> +import org.apache.openejb.config.*;
> import org.apache.openejb.core.ConnectorReference;
> import org.apache.openejb.core.CoreContainerSystem;
> import org.apache.openejb.core.CoreUserTransaction;
> @@ -829,6 +826,14 @@ public class Assembler extends AssemblerTool
> implements org.apache.openejb.spi.A
>
> final AppContext appContext = new
> AppContext(appInfo.appId, SystemInstance.get(), classLoader,
> globalJndiContext, appJndiContext, appInfo.standaloneModule);
> appContext.getProperties().putAll(appInfo.properties);
> +
> + final Set<Object> keys =
> appContext.getProperties().keySet();
> + for (final Object key : keys) {
> + if (! Module.class.isInstance(key)) {
> + appContext.getProperties().remove(key);
> + }
> + }
> +
> appContext.getInjections().addAll(injections);
> appContext.getBindings().putAll(globalBindings);
> appContext.getBindings().putAll(appBindings);
> diff --git
> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
> index feff954d40..aed0fda941 100644
> ---
> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
> +++
> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
> @@ -384,12 +384,7 @@ public class JndiEncBuilder {
> reference = new LinkRef(jndiName);
>
> } else if (BeanManager.class.equals(type)) {
> - reference = new LazyObjectReference<BeanManager>(new
> Callable<BeanManager>() {
> - @Override
> - public BeanManager call() throws Exception {
> - return new
> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
> - }
> - });
> + reference = new LazyObjectReference<>(new
> BeanManagerLazyReference());
>
> } else if (UserTransaction.class.equals(type)) {
> reference = new
> IntraVmJndiReference("comp/UserTransaction");
> @@ -684,4 +679,11 @@ public class JndiEncBuilder {
> }
> }
> }
> +
> + public static class BeanManagerLazyReference implements
> Callable<BeanManager> {
> + @Override
> + public BeanManager call() throws Exception {
> + return new
> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
> + }
> + }
> }
> diff --git
> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
> index 2cf387e112..285007925f 100644
> ---
> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
> +++
> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
> @@ -59,6 +59,7 @@ public class TempClassLoader extends URLClassLoader {
> private final ClassLoader system;
> private final boolean embedded;
> private final boolean parentURLClassLoader;
> + private final StackTraceElement[] created;
>
> // 80% of class files are smaller then 6k
> private final ByteArrayOutputStream bout = new
> ByteArrayOutputStream(6 * 1024);
> @@ -69,6 +70,7 @@ public class TempClassLoader extends URLClassLoader {
> this.system = ClassLoader.getSystemClassLoader();
> this.embedded = this.getClass().getClassLoader() == this.system;
> this.parentURLClassLoader =
> URLClassLoader.class.isInstance(parent);
> + this.created = new Throwable().getStackTrace();
> }
>
> /*
>
> In essence, this does 2 things:
>
> 1. Defines the Callable for the LazyObjectReference for BeanManager as a
> static class. This stops that object from having a reference to
> JndiEncBuilder and everything that has attached to it.
> 2. Strips off the EjbModule / WebModule from the SuperProperties on
> AppContext. This is set when deploying through the apps folder and lives
> forever. It has the TempClassLoader attached, and therefore all the
> AnnotationFinder stuff.
>
> I suspect (2) is too aggressive and is causing the test failures, but will
> confirm.
>
> Jon
>
> On Mon, Dec 16, 2019 at 11:27 AM Jonathan Gallimore <
> jonathan.gallimore@gmail.com> wrote:
>
>> Thanks for sending those over. I'm digging through this - I *think* I
>> have pinned down a specific reference that stops that classloader from
>> being released. The behaviour for a war/ear deployed in apps/ seems
>> different to deploying in webapps/ - I'm assuming that's what you're doing.
>> If you can confirm, that would help. I'll test out a patch with some bigger
>> .ear files today.
>>
>> Thanks
>>
>> Jon
>>
>> On Thu, Dec 12, 2019 at 9:08 PM Jonathan Gallimore <
>> jonathan.gallimore@gmail.com> wrote:
>>
>>> The mailing list seems to be stripping off your images - can you resend
>>> and include my email address as well? I'd be keen to dig into this a little
>>> more.
>>>
>>> Thanks
>>>
>>> Jon
>>>
>>> On Thu, Dec 12, 2019 at 9:00 PM Paul Carter-Brown
>>> <pa...@jini.guru> wrote:
>>>
>>>> Hi,
>>>>
>>>> The code does seem to use org.apache.openejb.core.TempClassLoader. I
>>>> can see one instance in the heap for every war in my ear. + 1.
>>>>
>>>> The TempClassLoader however still has 100's of strong references to it.
>>>> E.g:
>>>>
>>>> [image: image.png]
>>>>
>>>> [image: image.png]
>>>>
>>>>
>>>> Paul Carter-Brown
>>>> Director
>>>> Jini Guru
>>>> m: +27 (0) 83 442 7179 <+27834427179>
>>>> a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
>>>> Johannesburg, South Africa
>>>> w: jini.guru e: paul@jini.guru
>>>>
>>>> Disclaimer: This message and/or attachment(s) may contain
>>>> privileged, confidential and/or personal information. If you are not the
>>>> intended recipient you may not disclose or distribute any of
>>>> the information contained within this message. In such case you must
>>>> destroy this message and inform the sender of the error. Jini Guru may not
>>>> accept liability for any errors, omissions, information and viruses
>>>> contained in the transmission of this message. Any opinions, conclusions
>>>> and other information contained within this message not related to Jini
>>>> Guru official business is deemed to be that of the individual only and is
>>>> not endorsed by Jini Guru.
>>>>
>>>>
>>>>
>>>> On Thu, Dec 12, 2019 at 9:29 AM Mark Struberg <st...@yahoo.de.invalid>
>>>> wrote:
>>>>
>>>>> Hi!
>>>>>
>>>>> All this is just a workaround imo.
>>>>>
>>>>> Afair we (Romain) introduced a temporary throw-away ClassLoader for
>>>>> scanning. That means after doing all the class scans we only keep the
>>>>> extracted information but do not keep the Classes in memory. Doesn't this
>>>>> work anymore?
>>>>>
>>>>> LieGrue,
>>>>> strub
>>>>>
>>>>>
>>>>> > Am 12.12.2019 um 00:45 schrieb Paul Carter-Brown
>>>>> <pa...@jini.guru>:
>>>>> >
>>>>> > FYI. Graph of the change in heap size on a prod instance after this
>>>>> change:
>>>>> >
>>>>> >
>>>>> > Paul Carter-Brown
>>>>> > Director
>>>>> > Jini Guru
>>>>> > m: +27 (0) 83 442 7179
>>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>> Leslie
>>>>> > Johannesburg, South Africa
>>>>> > w: jini.guru e: paul@jini.guru
>>>>> >
>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>> privileged, confidential and/or personal information. If you are not the
>>>>> intended recipient you may not disclose or distribute any of the
>>>>> information contained within this message. In such case you must destroy
>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>> liability for any errors, omissions, information and viruses contained in
>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>> information contained within this message not related to Jini Guru official
>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>> Jini Guru.
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Thu, Dec 12, 2019 at 1:17 AM Paul Carter-Brown
>>>>> <pa...@jini.guru> wrote:
>>>>> > Hi Jon,
>>>>> >
>>>>> > I took a look at the source and worked out that I could add my own
>>>>> filters using openejb.additional.exclude.
>>>>> >
>>>>> > I set it to openejb.additional.exclude=com,net,io,org and the JVM
>>>>> heap now sits at around 150MB whereas it would never go below 450MB
>>>>> previously!!!
>>>>> >
>>>>> > Our EJB's and code is all in jars prefixed with guru.jini and hence
>>>>> they are not filtered out and the system works 100%.
>>>>> >
>>>>> > My sense is that there should be an easier way to specify what jars
>>>>> and/or packages to process for annotations. I have come across the
>>>>> following settings but can't really work out what fits where:
>>>>> >
>>>>> > openejb.additional.exclude
>>>>> > openejb.additional.include
>>>>> > openejb.deployments.classpath.filter.descriptors
>>>>> > openejb.deployments.classpath.filter.systemapps
>>>>> > openejb.deployments.classpath.include
>>>>> > openejb.deployments.classpath.exclude
>>>>> > openejb.deployments.package.include
>>>>> > openejb.deployments.package.exclude
>>>>> >
>>>>> >
>>>>> > P.S. Perhaps also add these to the standard exclusion list as they
>>>>> are common and yet won't need to be processed for annotations I guess?
>>>>> > com.amazonaws
>>>>> > com.fasterxml
>>>>> > com.google
>>>>> > com.hazelcast
>>>>> > io.grpc
>>>>> > io.netty
>>>>> > org.docx4j
>>>>> > org.mongodb
>>>>> > org.rocksdb
>>>>> > org.asynchttpclient
>>>>> > org.apache
>>>>> > org.aspectj
>>>>> >
>>>>> > But then maybe some are too broad and I don't really understand what
>>>>> the annotation index/cache is really needed for at runtime? Can it not be
>>>>> lazy loaded or discarded if unused? Just thinking out loud here :-( Seems
>>>>> like the cache uses a significant amount of heap when considering the drive
>>>>> towards tiny micro services.
>>>>> >
>>>>> > Paul Carter-Brown
>>>>> > Director
>>>>> > Jini Guru
>>>>> > m: +27 (0) 83 442 7179
>>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>> Leslie
>>>>> > Johannesburg, South Africa
>>>>> > w: jini.guru e: paul@jini.guru
>>>>> >
>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>> privileged, confidential and/or personal information. If you are not the
>>>>> intended recipient you may not disclose or distribute any of the
>>>>> information contained within this message. In such case you must destroy
>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>> liability for any errors, omissions, information and viruses contained in
>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>> information contained within this message not related to Jini Guru official
>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>> Jini Guru.
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Wed, Dec 11, 2019 at 7:18 PM Paul Carter-Brown
>>>>> <pa...@jini.guru> wrote:
>>>>> > Hi Jon,
>>>>> >
>>>>> > Unfortunately the snapshot behaves exactly the same way
>>>>> >
>>>>> > Paul Carter-Brown
>>>>> > Director
>>>>> > Jini Guru
>>>>> > m: +27 (0) 83 442 7179
>>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>> Leslie
>>>>> > Johannesburg, South Africa
>>>>> > w: jini.guru e: paul@jini.guru
>>>>> >
>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>> privileged, confidential and/or personal information. If you are not the
>>>>> intended recipient you may not disclose or distribute any of the
>>>>> information contained within this message. In such case you must destroy
>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>> liability for any errors, omissions, information and viruses contained in
>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>> information contained within this message not related to Jini Guru official
>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>> Jini Guru.
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Wed, Dec 11, 2019 at 3:39 PM Jonathan Gallimore <
>>>>> jonathan.gallimore@gmail.com> wrote:
>>>>> > Hi Paul
>>>>> >
>>>>> > Would you mind trying your application with a recent snapshot?
>>>>> >
>>>>> https://repository.apache.org/content/repositories/snapshots/org/apache/tomee/apache-tomee/8.0.1-SNAPSHOT/
>>>>> .
>>>>> > I recently updated the exclude list.
>>>>> >
>>>>> > Many thanks
>>>>> >
>>>>> > Jon
>>>>> >
>>>>> > On Wed, Dec 11, 2019 at 12:31 PM Paul Carter-Brown
>>>>> > <pa...@jini.guru> wrote:
>>>>> >
>>>>> > > Hi,
>>>>> > >
>>>>> > > I've been trying to lower the memory footprint of an EAR deployed
>>>>> to TomEE
>>>>> > > 8.0.0.
>>>>> > >
>>>>> > > Here is a screenshot of the heap histogram. The total Old Gen is
>>>>> about
>>>>> > > 450MB (after forcing multiple GC's). If I boot TomEE without my
>>>>> EAR then
>>>>> > > the old gen is about 6MB.
>>>>> > >
>>>>> > > [image: Screenshot from 2019-12-11 12-53-12.png]
>>>>> > >
>>>>> > > The entire ear is 140MB zip, most of which is in the ears /lib
>>>>> directory
>>>>> > > which contains libs such as Kafka, hazelcast, apache POI, Google
>>>>> cloud
>>>>> > > APIs, AWS client APIs etc etc. In total our code has about 100
>>>>> actual
>>>>> > > EJB's. If i remove files from the lib folder in the ear then I can
>>>>> see that
>>>>> > > the memory used by the annotation finder is lowered.
>>>>> > >
>>>>> > > Is there any way I can tell TomEE that it need not bother scanning
>>>>> > > everything in the /lib folder of my EAR for annotations and
>>>>> fulling up the
>>>>> > > heap. I already
>>>>> > > set openejb.deployments.classpath.include=.*jg-arch-core-impl.* to
>>>>> scan
>>>>> > > only the one jar in /lib which does have EJB's in it and it seems
>>>>> to obey
>>>>> > > this property but it doesn't seem to mean that annotation
>>>>> processing is
>>>>> > > skipped for all these other jars in /lib
>>>>> > >
>>>>> > > Thanks!
>>>>> > >
>>>>> > > Paul Carter-Brown
>>>>> > > Director
>>>>> > > Jini Guru
>>>>> > > m: +27 (0) 83 442 7179 <+27834427179>
>>>>> > > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>> Leslie
>>>>> > > Johannesburg, South Africa
>>>>> > > w: jini.guru e: paul@jini.guru
>>>>> > >
>>>>> > > Disclaimer: This message and/or attachment(s) may contain
>>>>> > > privileged, confidential and/or personal information. If you are
>>>>> not the
>>>>> > > intended recipient you may not disclose or distribute any of
>>>>> > > the information contained within this message. In such case you
>>>>> must
>>>>> > > destroy this message and inform the sender of the error. Jini Guru
>>>>> may not
>>>>> > > accept liability for any errors, omissions, information and viruses
>>>>> > > contained in the transmission of this message. Any opinions,
>>>>> conclusions
>>>>> > > and other information contained within this message not related to
>>>>> Jini
>>>>> > > Guru official business is deemed to be that of the individual only
>>>>> and is
>>>>> > > not endorsed by Jini Guru.
>>>>> > >
>>>>> > >
>>>>>
>>>>>
Re: Old Gen Full of Annotation Finder Index
Posted by Paul Carter-Brown <pa...@jini.guru>.
Hi Jon
Excellent stuff. Yes it's an ear with about 10 skinny wars inside and all
shared libs in /libs in the ear . Each war is a microservice but we deploy
as an ear in environments where it's too much of a hassle to split the
services up and deploy individually.
Thanks so much for your effort on this!
On Mon, 16 Dec 2019, 16:48 Jonathan Gallimore, <jo...@gmail.com>
wrote:
> Further update - I think my changes work with respect to this issue, but I
> do have a bunch of unit test failures which I need to look into. For info,
> here's the diff:
>
> diff --git
>
> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
>
> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
> index a6e2bdc4f5..c0d1a8c98f 100644
> ---
>
> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
> +++
>
> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
> @@ -69,10 +69,7 @@ import org.apache.openejb.cdi.ThreadSingletonService;
> import org.apache.openejb.classloader.ClassLoaderConfigurer;
> import org.apache.openejb.classloader.CompositeClassLoaderConfigurer;
> import org.apache.openejb.component.ClassLoaderEnricher;
> -import org.apache.openejb.config.ConfigurationFactory;
> -import org.apache.openejb.config.NewLoaderLogic;
> -import org.apache.openejb.config.QuickJarsTxtParser;
> -import org.apache.openejb.config.TldScanner;
> +import org.apache.openejb.config.*;
> import org.apache.openejb.core.ConnectorReference;
> import org.apache.openejb.core.CoreContainerSystem;
> import org.apache.openejb.core.CoreUserTransaction;
> @@ -829,6 +826,14 @@ public class Assembler extends AssemblerTool
> implements org.apache.openejb.spi.A
>
> final AppContext appContext = new
> AppContext(appInfo.appId, SystemInstance.get(), classLoader,
> globalJndiContext, appJndiContext, appInfo.standaloneModule);
> appContext.getProperties().putAll(appInfo.properties);
> +
> + final Set<Object> keys =
> appContext.getProperties().keySet();
> + for (final Object key : keys) {
> + if (! Module.class.isInstance(key)) {
> + appContext.getProperties().remove(key);
> + }
> + }
> +
> appContext.getInjections().addAll(injections);
> appContext.getBindings().putAll(globalBindings);
> appContext.getBindings().putAll(appBindings);
> diff --git
>
> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
>
> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
> index feff954d40..aed0fda941 100644
> ---
>
> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
> +++
>
> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
> @@ -384,12 +384,7 @@ public class JndiEncBuilder {
> reference = new LinkRef(jndiName);
>
> } else if (BeanManager.class.equals(type)) {
> - reference = new LazyObjectReference<BeanManager>(new
> Callable<BeanManager>() {
> - @Override
> - public BeanManager call() throws Exception {
> - return new
>
> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
> - }
> - });
> + reference = new LazyObjectReference<>(new
> BeanManagerLazyReference());
>
> } else if (UserTransaction.class.equals(type)) {
> reference = new
> IntraVmJndiReference("comp/UserTransaction");
> @@ -684,4 +679,11 @@ public class JndiEncBuilder {
> }
> }
> }
> +
> + public static class BeanManagerLazyReference implements
> Callable<BeanManager> {
> + @Override
> + public BeanManager call() throws Exception {
> + return new
>
> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
> + }
> + }
> }
> diff --git
>
> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
>
> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
> index 2cf387e112..285007925f 100644
> ---
>
> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
> +++
>
> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
> @@ -59,6 +59,7 @@ public class TempClassLoader extends URLClassLoader {
> private final ClassLoader system;
> private final boolean embedded;
> private final boolean parentURLClassLoader;
> + private final StackTraceElement[] created;
>
> // 80% of class files are smaller then 6k
> private final ByteArrayOutputStream bout = new ByteArrayOutputStream(6
> * 1024);
> @@ -69,6 +70,7 @@ public class TempClassLoader extends URLClassLoader {
> this.system = ClassLoader.getSystemClassLoader();
> this.embedded = this.getClass().getClassLoader() == this.system;
> this.parentURLClassLoader =
> URLClassLoader.class.isInstance(parent);
> + this.created = new Throwable().getStackTrace();
> }
>
> /*
>
> In essence, this does 2 things:
>
> 1. Defines the Callable for the LazyObjectReference for BeanManager as a
> static class. This stops that object from having a reference to
> JndiEncBuilder and everything that has attached to it.
> 2. Strips off the EjbModule / WebModule from the SuperProperties on
> AppContext. This is set when deploying through the apps folder and lives
> forever. It has the TempClassLoader attached, and therefore all the
> AnnotationFinder stuff.
>
> I suspect (2) is too aggressive and is causing the test failures, but will
> confirm.
>
> Jon
>
> On Mon, Dec 16, 2019 at 11:27 AM Jonathan Gallimore <
> jonathan.gallimore@gmail.com> wrote:
>
> > Thanks for sending those over. I'm digging through this - I *think* I
> have
> > pinned down a specific reference that stops that classloader from being
> > released. The behaviour for a war/ear deployed in apps/ seems different
> to
> > deploying in webapps/ - I'm assuming that's what you're doing. If you can
> > confirm, that would help. I'll test out a patch with some bigger .ear
> files
> > today.
> >
> > Thanks
> >
> > Jon
> >
> > On Thu, Dec 12, 2019 at 9:08 PM Jonathan Gallimore <
> > jonathan.gallimore@gmail.com> wrote:
> >
> >> The mailing list seems to be stripping off your images - can you resend
> >> and include my email address as well? I'd be keen to dig into this a
> little
> >> more.
> >>
> >> Thanks
> >>
> >> Jon
> >>
> >> On Thu, Dec 12, 2019 at 9:00 PM Paul Carter-Brown
> >> <pa...@jini.guru> wrote:
> >>
> >>> Hi,
> >>>
> >>> The code does seem to use org.apache.openejb.core.TempClassLoader. I
> can
> >>> see one instance in the heap for every war in my ear. + 1.
> >>>
> >>> The TempClassLoader however still has 100's of strong references to it.
> >>> E.g:
> >>>
> >>> [image: image.png]
> >>>
> >>> [image: image.png]
> >>>
> >>>
> >>> Paul Carter-Brown
> >>> Director
> >>> Jini Guru
> >>> m: +27 (0) 83 442 7179 <+27834427179>
> >>> a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
> >>> Johannesburg, South Africa
> >>> w: jini.guru e: paul@jini.guru
> >>>
> >>> Disclaimer: This message and/or attachment(s) may contain
> >>> privileged, confidential and/or personal information. If you are not
> the
> >>> intended recipient you may not disclose or distribute any of
> >>> the information contained within this message. In such case you must
> >>> destroy this message and inform the sender of the error. Jini Guru may
> not
> >>> accept liability for any errors, omissions, information and viruses
> >>> contained in the transmission of this message. Any opinions,
> conclusions
> >>> and other information contained within this message not related to Jini
> >>> Guru official business is deemed to be that of the individual only and
> is
> >>> not endorsed by Jini Guru.
> >>>
> >>>
> >>>
> >>> On Thu, Dec 12, 2019 at 9:29 AM Mark Struberg
> <st...@yahoo.de.invalid>
> >>> wrote:
> >>>
> >>>> Hi!
> >>>>
> >>>> All this is just a workaround imo.
> >>>>
> >>>> Afair we (Romain) introduced a temporary throw-away ClassLoader for
> >>>> scanning. That means after doing all the class scans we only keep the
> >>>> extracted information but do not keep the Classes in memory. Doesn't
> this
> >>>> work anymore?
> >>>>
> >>>> LieGrue,
> >>>> strub
> >>>>
> >>>>
> >>>> > Am 12.12.2019 um 00:45 schrieb Paul Carter-Brown
> >>>> <pa...@jini.guru>:
> >>>> >
> >>>> > FYI. Graph of the change in heap size on a prod instance after this
> >>>> change:
> >>>> >
> >>>> >
> >>>> > Paul Carter-Brown
> >>>> > Director
> >>>> > Jini Guru
> >>>> > m: +27 (0) 83 442 7179
> >>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
> >>>> Leslie
> >>>> > Johannesburg, South Africa
> >>>> > w: jini.guru e: paul@jini.guru
> >>>> >
> >>>> > Disclaimer: This message and/or attachment(s) may contain
> privileged,
> >>>> confidential and/or personal information. If you are not the intended
> >>>> recipient you may not disclose or distribute any of the information
> >>>> contained within this message. In such case you must destroy this
> message
> >>>> and inform the sender of the error. Jini Guru may not accept
> liability for
> >>>> any errors, omissions, information and viruses contained in the
> >>>> transmission of this message. Any opinions, conclusions and other
> >>>> information contained within this message not related to Jini Guru
> official
> >>>> business is deemed to be that of the individual only and is not
> endorsed by
> >>>> Jini Guru.
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > On Thu, Dec 12, 2019 at 1:17 AM Paul Carter-Brown
> >>>> <pa...@jini.guru> wrote:
> >>>> > Hi Jon,
> >>>> >
> >>>> > I took a look at the source and worked out that I could add my own
> >>>> filters using openejb.additional.exclude.
> >>>> >
> >>>> > I set it to openejb.additional.exclude=com,net,io,org and the JVM
> >>>> heap now sits at around 150MB whereas it would never go below 450MB
> >>>> previously!!!
> >>>> >
> >>>> > Our EJB's and code is all in jars prefixed with guru.jini and hence
> >>>> they are not filtered out and the system works 100%.
> >>>> >
> >>>> > My sense is that there should be an easier way to specify what jars
> >>>> and/or packages to process for annotations. I have come across the
> >>>> following settings but can't really work out what fits where:
> >>>> >
> >>>> > openejb.additional.exclude
> >>>> > openejb.additional.include
> >>>> > openejb.deployments.classpath.filter.descriptors
> >>>> > openejb.deployments.classpath.filter.systemapps
> >>>> > openejb.deployments.classpath.include
> >>>> > openejb.deployments.classpath.exclude
> >>>> > openejb.deployments.package.include
> >>>> > openejb.deployments.package.exclude
> >>>> >
> >>>> >
> >>>> > P.S. Perhaps also add these to the standard exclusion list as they
> >>>> are common and yet won't need to be processed for annotations I guess?
> >>>> > com.amazonaws
> >>>> > com.fasterxml
> >>>> > com.google
> >>>> > com.hazelcast
> >>>> > io.grpc
> >>>> > io.netty
> >>>> > org.docx4j
> >>>> > org.mongodb
> >>>> > org.rocksdb
> >>>> > org.asynchttpclient
> >>>> > org.apache
> >>>> > org.aspectj
> >>>> >
> >>>> > But then maybe some are too broad and I don't really understand what
> >>>> the annotation index/cache is really needed for at runtime? Can it
> not be
> >>>> lazy loaded or discarded if unused? Just thinking out loud here :-(
> Seems
> >>>> like the cache uses a significant amount of heap when considering the
> drive
> >>>> towards tiny micro services.
> >>>> >
> >>>> > Paul Carter-Brown
> >>>> > Director
> >>>> > Jini Guru
> >>>> > m: +27 (0) 83 442 7179
> >>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
> >>>> Leslie
> >>>> > Johannesburg, South Africa
> >>>> > w: jini.guru e: paul@jini.guru
> >>>> >
> >>>> > Disclaimer: This message and/or attachment(s) may contain
> privileged,
> >>>> confidential and/or personal information. If you are not the intended
> >>>> recipient you may not disclose or distribute any of the information
> >>>> contained within this message. In such case you must destroy this
> message
> >>>> and inform the sender of the error. Jini Guru may not accept
> liability for
> >>>> any errors, omissions, information and viruses contained in the
> >>>> transmission of this message. Any opinions, conclusions and other
> >>>> information contained within this message not related to Jini Guru
> official
> >>>> business is deemed to be that of the individual only and is not
> endorsed by
> >>>> Jini Guru.
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > On Wed, Dec 11, 2019 at 7:18 PM Paul Carter-Brown
> >>>> <pa...@jini.guru> wrote:
> >>>> > Hi Jon,
> >>>> >
> >>>> > Unfortunately the snapshot behaves exactly the same way
> >>>> >
> >>>> > Paul Carter-Brown
> >>>> > Director
> >>>> > Jini Guru
> >>>> > m: +27 (0) 83 442 7179
> >>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
> >>>> Leslie
> >>>> > Johannesburg, South Africa
> >>>> > w: jini.guru e: paul@jini.guru
> >>>> >
> >>>> > Disclaimer: This message and/or attachment(s) may contain
> privileged,
> >>>> confidential and/or personal information. If you are not the intended
> >>>> recipient you may not disclose or distribute any of the information
> >>>> contained within this message. In such case you must destroy this
> message
> >>>> and inform the sender of the error. Jini Guru may not accept
> liability for
> >>>> any errors, omissions, information and viruses contained in the
> >>>> transmission of this message. Any opinions, conclusions and other
> >>>> information contained within this message not related to Jini Guru
> official
> >>>> business is deemed to be that of the individual only and is not
> endorsed by
> >>>> Jini Guru.
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > On Wed, Dec 11, 2019 at 3:39 PM Jonathan Gallimore <
> >>>> jonathan.gallimore@gmail.com> wrote:
> >>>> > Hi Paul
> >>>> >
> >>>> > Would you mind trying your application with a recent snapshot?
> >>>> >
> >>>>
> https://repository.apache.org/content/repositories/snapshots/org/apache/tomee/apache-tomee/8.0.1-SNAPSHOT/
> >>>> .
> >>>> > I recently updated the exclude list.
> >>>> >
> >>>> > Many thanks
> >>>> >
> >>>> > Jon
> >>>> >
> >>>> > On Wed, Dec 11, 2019 at 12:31 PM Paul Carter-Brown
> >>>> > <pa...@jini.guru> wrote:
> >>>> >
> >>>> > > Hi,
> >>>> > >
> >>>> > > I've been trying to lower the memory footprint of an EAR deployed
> >>>> to TomEE
> >>>> > > 8.0.0.
> >>>> > >
> >>>> > > Here is a screenshot of the heap histogram. The total Old Gen is
> >>>> about
> >>>> > > 450MB (after forcing multiple GC's). If I boot TomEE without my
> EAR
> >>>> then
> >>>> > > the old gen is about 6MB.
> >>>> > >
> >>>> > > [image: Screenshot from 2019-12-11 12-53-12.png]
> >>>> > >
> >>>> > > The entire ear is 140MB zip, most of which is in the ears /lib
> >>>> directory
> >>>> > > which contains libs such as Kafka, hazelcast, apache POI, Google
> >>>> cloud
> >>>> > > APIs, AWS client APIs etc etc. In total our code has about 100
> >>>> actual
> >>>> > > EJB's. If i remove files from the lib folder in the ear then I can
> >>>> see that
> >>>> > > the memory used by the annotation finder is lowered.
> >>>> > >
> >>>> > > Is there any way I can tell TomEE that it need not bother scanning
> >>>> > > everything in the /lib folder of my EAR for annotations and
> fulling
> >>>> up the
> >>>> > > heap. I already
> >>>> > > set openejb.deployments.classpath.include=.*jg-arch-core-impl.* to
> >>>> scan
> >>>> > > only the one jar in /lib which does have EJB's in it and it seems
> >>>> to obey
> >>>> > > this property but it doesn't seem to mean that annotation
> >>>> processing is
> >>>> > > skipped for all these other jars in /lib
> >>>> > >
> >>>> > > Thanks!
> >>>> > >
> >>>> > > Paul Carter-Brown
> >>>> > > Director
> >>>> > > Jini Guru
> >>>> > > m: +27 (0) 83 442 7179 <+27834427179>
> >>>> > > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
> >>>> Leslie
> >>>> > > Johannesburg, South Africa
> >>>> > > w: jini.guru e: paul@jini.guru
> >>>> > >
> >>>> > > Disclaimer: This message and/or attachment(s) may contain
> >>>> > > privileged, confidential and/or personal information. If you are
> >>>> not the
> >>>> > > intended recipient you may not disclose or distribute any of
> >>>> > > the information contained within this message. In such case you
> must
> >>>> > > destroy this message and inform the sender of the error. Jini Guru
> >>>> may not
> >>>> > > accept liability for any errors, omissions, information and
> viruses
> >>>> > > contained in the transmission of this message. Any opinions,
> >>>> conclusions
> >>>> > > and other information contained within this message not related to
> >>>> Jini
> >>>> > > Guru official business is deemed to be that of the individual only
> >>>> and is
> >>>> > > not endorsed by Jini Guru.
> >>>> > >
> >>>> > >
> >>>>
> >>>>
>
Re: Old Gen Full of Annotation Finder Index
Posted by Jonathan Gallimore <jo...@gmail.com>.
Hi Paul
I've pushed a new snapshot which (hopefully) should address this. The CI
build is running on it now. Would you mind giving it a try? If it looks
good, I'll finish rolling the release. Fingers crossed it looks ok for you.
Jon
On Mon, Dec 16, 2019 at 2:48 PM Jonathan Gallimore <
jonathan.gallimore@gmail.com> wrote:
> Further update - I think my changes work with respect to this issue, but I
> do have a bunch of unit test failures which I need to look into. For info,
> here's the diff:
>
> diff --git
> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
> index a6e2bdc4f5..c0d1a8c98f 100644
> ---
> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
> +++
> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
> @@ -69,10 +69,7 @@ import org.apache.openejb.cdi.ThreadSingletonService;
> import org.apache.openejb.classloader.ClassLoaderConfigurer;
> import org.apache.openejb.classloader.CompositeClassLoaderConfigurer;
> import org.apache.openejb.component.ClassLoaderEnricher;
> -import org.apache.openejb.config.ConfigurationFactory;
> -import org.apache.openejb.config.NewLoaderLogic;
> -import org.apache.openejb.config.QuickJarsTxtParser;
> -import org.apache.openejb.config.TldScanner;
> +import org.apache.openejb.config.*;
> import org.apache.openejb.core.ConnectorReference;
> import org.apache.openejb.core.CoreContainerSystem;
> import org.apache.openejb.core.CoreUserTransaction;
> @@ -829,6 +826,14 @@ public class Assembler extends AssemblerTool
> implements org.apache.openejb.spi.A
>
> final AppContext appContext = new
> AppContext(appInfo.appId, SystemInstance.get(), classLoader,
> globalJndiContext, appJndiContext, appInfo.standaloneModule);
> appContext.getProperties().putAll(appInfo.properties);
> +
> + final Set<Object> keys =
> appContext.getProperties().keySet();
> + for (final Object key : keys) {
> + if (! Module.class.isInstance(key)) {
> + appContext.getProperties().remove(key);
> + }
> + }
> +
> appContext.getInjections().addAll(injections);
> appContext.getBindings().putAll(globalBindings);
> appContext.getBindings().putAll(appBindings);
> diff --git
> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
> index feff954d40..aed0fda941 100644
> ---
> a/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
> +++
> b/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/JndiEncBuilder.java
> @@ -384,12 +384,7 @@ public class JndiEncBuilder {
> reference = new LinkRef(jndiName);
>
> } else if (BeanManager.class.equals(type)) {
> - reference = new LazyObjectReference<BeanManager>(new
> Callable<BeanManager>() {
> - @Override
> - public BeanManager call() throws Exception {
> - return new
> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
> - }
> - });
> + reference = new LazyObjectReference<>(new
> BeanManagerLazyReference());
>
> } else if (UserTransaction.class.equals(type)) {
> reference = new
> IntraVmJndiReference("comp/UserTransaction");
> @@ -684,4 +679,11 @@ public class JndiEncBuilder {
> }
> }
> }
> +
> + public static class BeanManagerLazyReference implements
> Callable<BeanManager> {
> + @Override
> + public BeanManager call() throws Exception {
> + return new
> InjectableBeanManager(WebBeansContext.currentInstance().getBeanManagerImpl());
> + }
> + }
> }
> diff --git
> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
> index 2cf387e112..285007925f 100644
> ---
> a/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
> +++
> b/container/openejb-core/src/main/java/org/apache/openejb/core/TempClassLoader.java
> @@ -59,6 +59,7 @@ public class TempClassLoader extends URLClassLoader {
> private final ClassLoader system;
> private final boolean embedded;
> private final boolean parentURLClassLoader;
> + private final StackTraceElement[] created;
>
> // 80% of class files are smaller then 6k
> private final ByteArrayOutputStream bout = new
> ByteArrayOutputStream(6 * 1024);
> @@ -69,6 +70,7 @@ public class TempClassLoader extends URLClassLoader {
> this.system = ClassLoader.getSystemClassLoader();
> this.embedded = this.getClass().getClassLoader() == this.system;
> this.parentURLClassLoader =
> URLClassLoader.class.isInstance(parent);
> + this.created = new Throwable().getStackTrace();
> }
>
> /*
>
> In essence, this does 2 things:
>
> 1. Defines the Callable for the LazyObjectReference for BeanManager as a
> static class. This stops that object from having a reference to
> JndiEncBuilder and everything that has attached to it.
> 2. Strips off the EjbModule / WebModule from the SuperProperties on
> AppContext. This is set when deploying through the apps folder and lives
> forever. It has the TempClassLoader attached, and therefore all the
> AnnotationFinder stuff.
>
> I suspect (2) is too aggressive and is causing the test failures, but will
> confirm.
>
> Jon
>
> On Mon, Dec 16, 2019 at 11:27 AM Jonathan Gallimore <
> jonathan.gallimore@gmail.com> wrote:
>
>> Thanks for sending those over. I'm digging through this - I *think* I
>> have pinned down a specific reference that stops that classloader from
>> being released. The behaviour for a war/ear deployed in apps/ seems
>> different to deploying in webapps/ - I'm assuming that's what you're doing.
>> If you can confirm, that would help. I'll test out a patch with some bigger
>> .ear files today.
>>
>> Thanks
>>
>> Jon
>>
>> On Thu, Dec 12, 2019 at 9:08 PM Jonathan Gallimore <
>> jonathan.gallimore@gmail.com> wrote:
>>
>>> The mailing list seems to be stripping off your images - can you resend
>>> and include my email address as well? I'd be keen to dig into this a little
>>> more.
>>>
>>> Thanks
>>>
>>> Jon
>>>
>>> On Thu, Dec 12, 2019 at 9:00 PM Paul Carter-Brown
>>> <pa...@jini.guru> wrote:
>>>
>>>> Hi,
>>>>
>>>> The code does seem to use org.apache.openejb.core.TempClassLoader. I
>>>> can see one instance in the heap for every war in my ear. + 1.
>>>>
>>>> The TempClassLoader however still has 100's of strong references to it.
>>>> E.g:
>>>>
>>>> [image: image.png]
>>>>
>>>> [image: image.png]
>>>>
>>>>
>>>> Paul Carter-Brown
>>>> Director
>>>> Jini Guru
>>>> m: +27 (0) 83 442 7179 <+27834427179>
>>>> a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and Leslie
>>>> Johannesburg, South Africa
>>>> w: jini.guru e: paul@jini.guru
>>>>
>>>> Disclaimer: This message and/or attachment(s) may contain
>>>> privileged, confidential and/or personal information. If you are not the
>>>> intended recipient you may not disclose or distribute any of
>>>> the information contained within this message. In such case you must
>>>> destroy this message and inform the sender of the error. Jini Guru may not
>>>> accept liability for any errors, omissions, information and viruses
>>>> contained in the transmission of this message. Any opinions, conclusions
>>>> and other information contained within this message not related to Jini
>>>> Guru official business is deemed to be that of the individual only and is
>>>> not endorsed by Jini Guru.
>>>>
>>>>
>>>>
>>>> On Thu, Dec 12, 2019 at 9:29 AM Mark Struberg <st...@yahoo.de.invalid>
>>>> wrote:
>>>>
>>>>> Hi!
>>>>>
>>>>> All this is just a workaround imo.
>>>>>
>>>>> Afair we (Romain) introduced a temporary throw-away ClassLoader for
>>>>> scanning. That means after doing all the class scans we only keep the
>>>>> extracted information but do not keep the Classes in memory. Doesn't this
>>>>> work anymore?
>>>>>
>>>>> LieGrue,
>>>>> strub
>>>>>
>>>>>
>>>>> > Am 12.12.2019 um 00:45 schrieb Paul Carter-Brown
>>>>> <pa...@jini.guru>:
>>>>> >
>>>>> > FYI. Graph of the change in heap size on a prod instance after this
>>>>> change:
>>>>> >
>>>>> >
>>>>> > Paul Carter-Brown
>>>>> > Director
>>>>> > Jini Guru
>>>>> > m: +27 (0) 83 442 7179
>>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>> Leslie
>>>>> > Johannesburg, South Africa
>>>>> > w: jini.guru e: paul@jini.guru
>>>>> >
>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>> privileged, confidential and/or personal information. If you are not the
>>>>> intended recipient you may not disclose or distribute any of the
>>>>> information contained within this message. In such case you must destroy
>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>> liability for any errors, omissions, information and viruses contained in
>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>> information contained within this message not related to Jini Guru official
>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>> Jini Guru.
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Thu, Dec 12, 2019 at 1:17 AM Paul Carter-Brown
>>>>> <pa...@jini.guru> wrote:
>>>>> > Hi Jon,
>>>>> >
>>>>> > I took a look at the source and worked out that I could add my own
>>>>> filters using openejb.additional.exclude.
>>>>> >
>>>>> > I set it to openejb.additional.exclude=com,net,io,org and the JVM
>>>>> heap now sits at around 150MB whereas it would never go below 450MB
>>>>> previously!!!
>>>>> >
>>>>> > Our EJB's and code is all in jars prefixed with guru.jini and hence
>>>>> they are not filtered out and the system works 100%.
>>>>> >
>>>>> > My sense is that there should be an easier way to specify what jars
>>>>> and/or packages to process for annotations. I have come across the
>>>>> following settings but can't really work out what fits where:
>>>>> >
>>>>> > openejb.additional.exclude
>>>>> > openejb.additional.include
>>>>> > openejb.deployments.classpath.filter.descriptors
>>>>> > openejb.deployments.classpath.filter.systemapps
>>>>> > openejb.deployments.classpath.include
>>>>> > openejb.deployments.classpath.exclude
>>>>> > openejb.deployments.package.include
>>>>> > openejb.deployments.package.exclude
>>>>> >
>>>>> >
>>>>> > P.S. Perhaps also add these to the standard exclusion list as they
>>>>> are common and yet won't need to be processed for annotations I guess?
>>>>> > com.amazonaws
>>>>> > com.fasterxml
>>>>> > com.google
>>>>> > com.hazelcast
>>>>> > io.grpc
>>>>> > io.netty
>>>>> > org.docx4j
>>>>> > org.mongodb
>>>>> > org.rocksdb
>>>>> > org.asynchttpclient
>>>>> > org.apache
>>>>> > org.aspectj
>>>>> >
>>>>> > But then maybe some are too broad and I don't really understand what
>>>>> the annotation index/cache is really needed for at runtime? Can it not be
>>>>> lazy loaded or discarded if unused? Just thinking out loud here :-( Seems
>>>>> like the cache uses a significant amount of heap when considering the drive
>>>>> towards tiny micro services.
>>>>> >
>>>>> > Paul Carter-Brown
>>>>> > Director
>>>>> > Jini Guru
>>>>> > m: +27 (0) 83 442 7179
>>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>> Leslie
>>>>> > Johannesburg, South Africa
>>>>> > w: jini.guru e: paul@jini.guru
>>>>> >
>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>> privileged, confidential and/or personal information. If you are not the
>>>>> intended recipient you may not disclose or distribute any of the
>>>>> information contained within this message. In such case you must destroy
>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>> liability for any errors, omissions, information and viruses contained in
>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>> information contained within this message not related to Jini Guru official
>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>> Jini Guru.
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Wed, Dec 11, 2019 at 7:18 PM Paul Carter-Brown
>>>>> <pa...@jini.guru> wrote:
>>>>> > Hi Jon,
>>>>> >
>>>>> > Unfortunately the snapshot behaves exactly the same way
>>>>> >
>>>>> > Paul Carter-Brown
>>>>> > Director
>>>>> > Jini Guru
>>>>> > m: +27 (0) 83 442 7179
>>>>> > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>> Leslie
>>>>> > Johannesburg, South Africa
>>>>> > w: jini.guru e: paul@jini.guru
>>>>> >
>>>>> > Disclaimer: This message and/or attachment(s) may contain
>>>>> privileged, confidential and/or personal information. If you are not the
>>>>> intended recipient you may not disclose or distribute any of the
>>>>> information contained within this message. In such case you must destroy
>>>>> this message and inform the sender of the error. Jini Guru may not accept
>>>>> liability for any errors, omissions, information and viruses contained in
>>>>> the transmission of this message. Any opinions, conclusions and other
>>>>> information contained within this message not related to Jini Guru official
>>>>> business is deemed to be that of the individual only and is not endorsed by
>>>>> Jini Guru.
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Wed, Dec 11, 2019 at 3:39 PM Jonathan Gallimore <
>>>>> jonathan.gallimore@gmail.com> wrote:
>>>>> > Hi Paul
>>>>> >
>>>>> > Would you mind trying your application with a recent snapshot?
>>>>> >
>>>>> https://repository.apache.org/content/repositories/snapshots/org/apache/tomee/apache-tomee/8.0.1-SNAPSHOT/
>>>>> .
>>>>> > I recently updated the exclude list.
>>>>> >
>>>>> > Many thanks
>>>>> >
>>>>> > Jon
>>>>> >
>>>>> > On Wed, Dec 11, 2019 at 12:31 PM Paul Carter-Brown
>>>>> > <pa...@jini.guru> wrote:
>>>>> >
>>>>> > > Hi,
>>>>> > >
>>>>> > > I've been trying to lower the memory footprint of an EAR deployed
>>>>> to TomEE
>>>>> > > 8.0.0.
>>>>> > >
>>>>> > > Here is a screenshot of the heap histogram. The total Old Gen is
>>>>> about
>>>>> > > 450MB (after forcing multiple GC's). If I boot TomEE without my
>>>>> EAR then
>>>>> > > the old gen is about 6MB.
>>>>> > >
>>>>> > > [image: Screenshot from 2019-12-11 12-53-12.png]
>>>>> > >
>>>>> > > The entire ear is 140MB zip, most of which is in the ears /lib
>>>>> directory
>>>>> > > which contains libs such as Kafka, hazelcast, apache POI, Google
>>>>> cloud
>>>>> > > APIs, AWS client APIs etc etc. In total our code has about 100
>>>>> actual
>>>>> > > EJB's. If i remove files from the lib folder in the ear then I can
>>>>> see that
>>>>> > > the memory used by the annotation finder is lowered.
>>>>> > >
>>>>> > > Is there any way I can tell TomEE that it need not bother scanning
>>>>> > > everything in the /lib folder of my EAR for annotations and
>>>>> fulling up the
>>>>> > > heap. I already
>>>>> > > set openejb.deployments.classpath.include=.*jg-arch-core-impl.* to
>>>>> scan
>>>>> > > only the one jar in /lib which does have EJB's in it and it seems
>>>>> to obey
>>>>> > > this property but it doesn't seem to mean that annotation
>>>>> processing is
>>>>> > > skipped for all these other jars in /lib
>>>>> > >
>>>>> > > Thanks!
>>>>> > >
>>>>> > > Paul Carter-Brown
>>>>> > > Director
>>>>> > > Jini Guru
>>>>> > > m: +27 (0) 83 442 7179 <+27834427179>
>>>>> > > a: 1st Floor, Golf House, Design Quarter, Cnr. William Nicol and
>>>>> Leslie
>>>>> > > Johannesburg, South Africa
>>>>> > > w: jini.guru e: paul@jini.guru
>>>>> > >
>>>>> > > Disclaimer: This message and/or attachment(s) may contain
>>>>> > > privileged, confidential and/or personal information. If you are
>>>>> not the
>>>>> > > intended recipient you may not disclose or distribute any of
>>>>> > > the information contained within this message. In such case you
>>>>> must
>>>>> > > destroy this message and inform the sender of the error. Jini Guru
>>>>> may not
>>>>> > > accept liability for any errors, omissions, information and viruses
>>>>> > > contained in the transmission of this message. Any opinions,
>>>>> conclusions
>>>>> > > and other information contained within this message not related to
>>>>> Jini
>>>>> > > Guru official business is deemed to be that of the individual only
>>>>> and is
>>>>> > > not endorsed by Jini Guru.
>>>>> > >
>>>>> > >
>>>>>
>>>>>