You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomee.apache.org by "David A. Eberhart" <DE...@nycourts.gov> on 2023/02/01 15:27:26 UTC
CXF JAXRSClientFactoryBean, Johnzon JSON-B provider, and a CDI memory leak on TomEE 8.x
Good morning,
It appears that using JAXRSClientFactoryBean to create proxy-beans for JAX-RS webservices from within a TomEE 8.x container triggers a rather severe memory leak via Johnzon's JSON-B provider's CDI integration in TomEE 8.x (I did not test 9.x).
Specifically, if JAXRSClientFactoryBean is used while TomEEJsonbProvider (or JsonbJaxrsProvider) is on the OpenEJB JAX-RS JSON providers list (either the hard-coded default, or via 'openejb.jaxrs.jsonProviders'), then each call to JAXRSClientFactoryBean.create() will cause a new JohnzonJsonb object to be created and tracked by JohnzonCdiExtension. For example, the following code is sufficient to trigger the issue if placed within a servlet class' doGet():
JAXRSClientFactoryBean bean = new JAXRSClientFactoryBean();
bean.setServiceClass(IJaxrsWebservice.class);
bean.setAddress(http://127.0.0.1:8080/reproducer);
final IJaxrsWebservice proxy = bean.create(IJaxrsWebservice.class);
I have a sample project that includes this code and a mock JAX-RS webservice, reproducing the issue; a copy of this is available in this GitHub repo: https://github.com/deberhar/TomeeJohnzonBugReproducer. If I repeatedly call the servlet, I can exhaust a 1GB application heap in ~4-6k GETs to the servlet -- the server reports OutOfMemoryErrors, and a heap dump shows 924MB retained by the JohnzonCdiExtension. If it would be helpful, I can provide a copy of this project (what's the best way to do so?).
[Text Description automatically generated]
I believe the underlying cause is described in this Johnzon bug report: https://issues.apache.org/jira/browse/JOHNZON-161. However, it seems the Johnzon developers believe their CDI integration is behaving as expected, and that it is the responsibility of the integrating container to cache and re-use Jsonb instances, writing, in part:
> Hi skay, first of all it is a bug in your application so best is to fix your coding style. Now you have a workaround: set johnzon.skip-cdi=true or johnzon.cdi.activated=false in the jsonbconfig properties or set property johnzon.factory to new SimpleJohnzonAdapterFactory().
> Side note: in terms of performance, recreating Jsonb instances looses all caching and redo all the reflection so should be slow so in any case I recommend you to change the pattern you have (it is true for jackson too btw)
It appears the Meecrowave developers made changes in their source code to prevent this leak via Johnzon's JsonbJaxrsProvider's CDI integration when used with CXF; see https://issues.apache.org/jira/browse/MEECROWAVE-104 or https://github.com/apache/openwebbeans-meecrowave/commit/eafd5b6eda81471c4abd61297a2455b2bca0a013.
For our organization's use of TomEE, we've been able to hack around this issue by subclassing TomEEJsonbProvider to set johnzon.cdi.activated=false, and then overriding the default provider list via openejb.jaxrs.jsonProviders. However, this is likely not a proper solution.
What's the best way forward to addressing this properly?
Thank you,
David Eberhart
NYS Unified Court System
Office of Court Administration
Division of Technology and Court Research
RE: CXF JAXRSClientFactoryBean, Johnzon JSON-B provider, and a CDI memory leak on TomEE 8.x
Posted by "David A. Eberhart" <DE...@nycourts.gov>.
Good morning David,
Unfortunately, I don't believe I have the depth-of-knowledge of how TomEE's CDI is interacting with CXF and Johnzon to evaluate what the proper solution should be, in this case. In our use case, turning off Johnzon's CDI integration makes the memory leak go away, however, I'm concerned about possible side-effects of doing so -- our code is not injecting any of the JSON-B objects, and we're not using any custom adapters, so it's possible the change causes some regression there. Also, in the JOHNZON-161 bug report I linked in my original e-mail, a Johnzon maintainer expressed concern about disabling CDI integration could still lead to memory leaks in some circumstances:
> Just a side note for future readers: it is not "closed automatically", just let to the garbage collector (so some advanced usages of the underlying mapper - addCloseable - or other features can still leak). It is unlikely but can happen and I don't want readers assume there is any magic there.
Would someone with more expertise in this subject area be able to take a look, please? If it turns out that the proper solution *is* to turn off Johnzon's CDI integration for the JAX-RS provider, I can send you a one-liner PR for that?
Thank you,
David
-----Original Message-----
From: David Blevins <da...@gmail.com>
Sent: Wednesday, February 8, 2023 4:03 PM
To: dev@tomee.apache.org
Subject: Re: CXF JAXRSClientFactoryBean, Johnzon JSON-B provider, and a CDI memory leak on TomEE 8.x
Hey David,
Looks like you've done some very great investigation and you're pretty deep into the code! The best path forward would be to get a PR to the main TomEE branch and once that's looking good some help backporting it to TomEE 8 and 9.
Does that sound like something you'd be able to help with? If, so here's the source in Github:
- https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Ftomee&data=05%7C01%7CDEBERHAR%40nycourts.gov%7Ce70bdb3db1dd4a79b89308db0a17e04a%7C3456fe92cbd1406db5a35364bec0a833%7C0%7C0%7C638114869898134445%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ha2A7c4j%2BmsCMRXm0sIU%2B6bqu2tlT%2F4lXtmiyGDz8rA%3D&reserved=0
- https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Ftomee%2Fpulls&data=05%7C01%7CDEBERHAR%40nycourts.gov%7Ce70bdb3db1dd4a79b89308db0a17e04a%7C3456fe92cbd1406db5a35364bec0a833%7C0%7C0%7C638114869898134445%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2F4RaJOEnP%2FXbDI13OLfz%2F%2FH8U0WGxC5cOgZ%2FDdjTnhE%3D&reserved=0
If you're not familiar with git and making PRs, we're very happy to help you through it - we're always willing to help new people get into the project!
-David
> On Feb 1, 2023, at 7:27 AM, David A. Eberhart <DE...@nycourts.gov> wrote:
>
> Good morning,
> It appears that using JAXRSClientFactoryBean to create proxy-beans for JAX-RS webservices from within a TomEE 8.x container triggers a rather severe memory leak via Johnzon's JSON-B provider's CDI integration in TomEE 8.x (I did not test 9.x).
>
> Specifically, if JAXRSClientFactoryBean is used while TomEEJsonbProvider (or JsonbJaxrsProvider) is on the OpenEJB JAX-RS JSON providers list (either the hard-coded default, or via 'openejb.jaxrs.jsonProviders'), then each call to JAXRSClientFactoryBean.create() will cause a new JohnzonJsonb object to be created and tracked by JohnzonCdiExtension. For example, the following code is sufficient to trigger the issue if placed within a servlet class' doGet():
> JAXRSClientFactoryBean bean = new JAXRSClientFactoryBean();
> bean.setServiceClass(IJaxrsWebservice.class);
> bean.setAddress(https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1%3A8080%2Freproducer&data=05%7C01%7CDEBERHAR%40nycourts.gov%7Ce70bdb3db1dd4a79b89308db0a17e04a%7C3456fe92cbd1406db5a35364bec0a833%7C0%7C0%7C638114869898134445%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=u%2Bz8eJ5gnvpFJZZYetZGoLLFlUT%2FKCQ2iSlznG6Q0jM%3D&reserved=0 <https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2F127.0.0.1%3A8080%2Freproducer&data=05%7C01%7CDEBERHAR%40nycourts.gov%7Ce70bdb3db1dd4a79b89308db0a17e04a%7C3456fe92cbd1406db5a35364bec0a833%7C0%7C0%7C638114869898134445%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=u%2Bz8eJ5gnvpFJZZYetZGoLLFlUT%2FKCQ2iSlznG6Q0jM%3D&reserved=0>);
>
> final IJaxrsWebservice proxy = bean.create(IJaxrsWebservice.class);
>
> I have a sample project that includes this code and a mock JAX-RS webservice, reproducing the issue; a copy of this is available in this GitHub repo: https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdeberhar%2FTomeeJohnzonBugReproducer&data=05%7C01%7CDEBERHAR%40nycourts.gov%7Ce70bdb3db1dd4a79b89308db0a17e04a%7C3456fe92cbd1406db5a35364bec0a833%7C0%7C0%7C638114869898134445%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pUJTBr97KqEVZ5x05fhX%2BqPQrS141Rwv0OEu2Fm9%2FvQ%3D&reserved=0 <https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdeberhar%2FTomeeJohnzonBugReproducer&data=05%7C01%7CDEBERHAR%40nycourts.gov%7Ce70bdb3db1dd4a79b89308db0a17e04a%7C3456fe92cbd1406db5a35364bec0a833%7C0%7C0%7C638114869898134445%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pUJTBr97KqEVZ5x05fhX%2BqPQrS141Rwv0OEu2Fm9%2FvQ%3D&reserved=0>. If I repeatedly call the servlet, I can exhaust a 1GB application heap in ~4-6k GETs to the servlet -- the server reports OutOfMemoryErrors, and a heap dump shows 924MB retained by the JohnzonCdiExtension. If it would be helpful, I can provide a copy of this project (what's the best way to do so?).
>
>
>
> I believe the underlying cause is described in this Johnzon bug report:https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FJOHNZON-161&data=05%7C01%7CDEBERHAR%40nycourts.gov%7Ce70bdb3db1dd4a79b89308db0a17e04a%7C3456fe92cbd1406db5a35364bec0a833%7C0%7C0%7C638114869898134445%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=cctDfaMZIa41hj%2Fkbp2uSO7XSQQiROAK1kZrNV40lAg%3D&reserved=0 <https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FJOHNZON-161&data=05%7C01%7CDEBERHAR%40nycourts.gov%7Ce70bdb3db1dd4a79b89308db0a17e04a%7C3456fe92cbd1406db5a35364bec0a833%7C0%7C0%7C638114869898134445%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=cctDfaMZIa41hj%2Fkbp2uSO7XSQQiROAK1kZrNV40lAg%3D&reserved=0>. However, it seems the Johnzon developers believe their CDI integration is behaving as expected, and that it is the responsibility of the integrating container to cache and re-use Jsonb instances, writing, in part:
> > Hi skay, first of all it is a bug in your application so best is to fix your coding style. Now you have a workaround: set johnzon.skip-cdi=true or johnzon.cdi.activated=false in the jsonbconfig properties or set property johnzon.factory to new SimpleJohnzonAdapterFactory().
> > Side note: in terms of performance, recreating Jsonb instances looses all caching and redo all the reflection so should be slow so in any case I recommend you to change the pattern you have (it is true for jackson too btw)
>
> It appears the Meecrowave developers made changes in their source code to prevent this leak via Johnzon's JsonbJaxrsProvider's CDI integration when used with CXF; seehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FMEECROWAVE-104&data=05%7C01%7CDEBERHAR%40nycourts.gov%7Ce70bdb3db1dd4a79b89308db0a17e04a%7C3456fe92cbd1406db5a35364bec0a833%7C0%7C0%7C638114869898134445%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ZTIVakT5zOcu%2FeoagRQdKQf12m4t8cvrl%2FKcrn0EFEM%3D&reserved=0 <https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FMEECROWAVE-104&data=05%7C01%7CDEBERHAR%40nycourts.gov%7Ce70bdb3db1dd4a79b89308db0a17e04a%7C3456fe92cbd1406db5a35364bec0a833%7C0%7C0%7C638114869898134445%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ZTIVakT5zOcu%2FeoagRQdKQf12m4t8cvrl%2FKcrn0EFEM%3D&reserved=0> or https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fopenwebbeans-meecrowave%2Fcommit%2Feafd5b6eda81471c4abd61297a2455b2bca0a013&data=05%7C01%7CDEBERHAR%40nycourts.gov%7Ce70bdb3db1dd4a79b89308db0a17e04a%7C3456fe92cbd1406db5a35364bec0a833%7C0%7C0%7C638114869898134445%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=DLYd4ZKnWYZorixGA0%2FnG%2BH8tqE9zvE6tn16sjDmBd8%3D&reserved=0 <https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fopenwebbeans-meecrowave%2Fcommit%2Feafd5b6eda81471c4abd61297a2455b2bca0a013&data=05%7C01%7CDEBERHAR%40nycourts.gov%7Ce70bdb3db1dd4a79b89308db0a17e04a%7C3456fe92cbd1406db5a35364bec0a833%7C0%7C0%7C638114869898134445%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=DLYd4ZKnWYZorixGA0%2FnG%2BH8tqE9zvE6tn16sjDmBd8%3D&reserved=0>.
> For our organization's use of TomEE, we've been able to hack around this issue by subclassing TomEEJsonbProvider to set johnzon.cdi.activated=false, and then overriding the default provider list via openejb.jaxrs.jsonProviders. However, this is likely not a proper solution.
>
> What's the best way forward to addressing this properly?
>
>
>
> Thank you,
> David Eberhart
> NYS Unified Court System
> Office of Court Administration
> Division of Technology and Court Research
Please be CAREFUL when clicking links or opening attachments from external senders.
Re: CXF JAXRSClientFactoryBean, Johnzon JSON-B provider, and a CDI memory leak on TomEE 8.x
Posted by David Blevins <da...@gmail.com>.
Hey David,
Looks like you’ve done some very great investigation and you’re pretty deep into the code! The best path forward would be to get a PR to the main TomEE branch and once that’s looking good some help backporting it to TomEE 8 and 9.
Does that sound like something you’d be able to help with? If, so here’s the source in Github:
- https://github.com/apache/tomee
- https://github.com/apache/tomee/pulls
If you’re not familiar with git and making PRs, we’re very happy to help you through it — we’re always willing to help new people get into the project!
-David
> On Feb 1, 2023, at 7:27 AM, David A. Eberhart <DE...@nycourts.gov> wrote:
>
> Good morning,
> It appears that using JAXRSClientFactoryBean to create proxy-beans for JAX-RS webservices from within a TomEE 8.x container triggers a rather severe memory leak via Johnzon's JSON-B provider's CDI integration in TomEE 8.x (I did not test 9.x).
>
> Specifically, if JAXRSClientFactoryBean is used while TomEEJsonbProvider (or JsonbJaxrsProvider) is on the OpenEJB JAX-RS JSON providers list (either the hard-coded default, or via 'openejb.jaxrs.jsonProviders'), then each call to JAXRSClientFactoryBean.create() will cause a new JohnzonJsonb object to be created and tracked by JohnzonCdiExtension. For example, the following code is sufficient to trigger the issue if placed within a servlet class' doGet():
> JAXRSClientFactoryBean bean = new JAXRSClientFactoryBean();
> bean.setServiceClass(IJaxrsWebservice.class);
> bean.setAddress(http://127.0.0.1:8080/reproducer <http://127.0.0.1:8080/reproducer>);
>
> final IJaxrsWebservice proxy = bean.create(IJaxrsWebservice.class);
>
> I have a sample project that includes this code and a mock JAX-RS webservice, reproducing the issue; a copy of this is available in this GitHub repo: https://github.com/deberhar/TomeeJohnzonBugReproducer <https://github.com/deberhar/TomeeJohnzonBugReproducer>. If I repeatedly call the servlet, I can exhaust a 1GB application heap in ~4-6k GETs to the servlet -- the server reports OutOfMemoryErrors, and a heap dump shows 924MB retained by the JohnzonCdiExtension. If it would be helpful, I can provide a copy of this project (what's the best way to do so?).
>
>
>
> I believe the underlying cause is described in this Johnzon bug report:https://issues.apache.org/jira/browse/JOHNZON-161 <https://issues.apache.org/jira/browse/JOHNZON-161>. However, it seems the Johnzon developers believe their CDI integration is behaving as expected, and that it is the responsibility of the integrating container to cache and re-use Jsonb instances, writing, in part:
> > Hi skay, first of all it is a bug in your application so best is to fix your coding style. Now you have a workaround: set johnzon.skip-cdi=true or johnzon.cdi.activated=false in the jsonbconfig properties or set property johnzon.factory to new SimpleJohnzonAdapterFactory().
> > Side note: in terms of performance, recreating Jsonb instances looses all caching and redo all the reflection so should be slow so in any case I recommend you to change the pattern you have (it is true for jackson too btw)
>
> It appears the Meecrowave developers made changes in their source code to prevent this leak via Johnzon's JsonbJaxrsProvider's CDI integration when used with CXF; seehttps://issues.apache.org/jira/browse/MEECROWAVE-104 <https://issues.apache.org/jira/browse/MEECROWAVE-104> or https://github.com/apache/openwebbeans-meecrowave/commit/eafd5b6eda81471c4abd61297a2455b2bca0a013 <https://github.com/apache/openwebbeans-meecrowave/commit/eafd5b6eda81471c4abd61297a2455b2bca0a013>.
> For our organization's use of TomEE, we've been able to hack around this issue by subclassing TomEEJsonbProvider to set johnzon.cdi.activated=false, and then overriding the default provider list via openejb.jaxrs.jsonProviders. However, this is likely not a proper solution.
>
> What's the best way forward to addressing this properly?
>
>
>
> Thank you,
> David Eberhart
> NYS Unified Court System
> Office of Court Administration
> Division of Technology and Court Research