You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Franck Malka <fm...@modulo.co.il> on 2025/02/19 11:34:36 UTC
ActiveMq 2.14.0 Native Memory Leak
Hi, all
I have a case with an Artemis ActiveMq cluster which process a very large
number of messages per second.
Each few weeks of operation, JVM memory is still cleaning properly but
native memory is not getting cleared.
Did anyone encounter this case?
Does it sounds more like a configuration issue? Or like something upgrading
to latest version would address?
Franck.
Re: ActiveMq 2.14.0 Native Memory Leak
Posted by Clebert Suconic <cl...@gmail.com>.
There was a lot of work in testing after 2.20. I highly recommend resina a
recent version às Anton said.
Clebert Suconic
On Thu, Feb 20, 2025 at 12:17 PM Justin Bertram <jb...@apache.org> wrote:
> I agree with Anton here 100%.
>
> Netty uses direct memory for performance reasons. If you disable it there
> will certainly be performance trade-offs. Whether that really impacts you
> depends on your use-case.
>
> Also, keep in mind that if Netty doesn't use direct memory it will use the
> JVM's heap instead. You may need to increase the heap size to account for
> this. Only testing will show.
>
>
> Justin
>
> On Thu, Feb 20, 2025 at 3:46 AM Anton Roskvist <an...@apache.org> wrote:
>
> > Sorry, but to my knowledge that is almost impossible to say for sure, as
> > it's highly dependent on your specific use case and setup. As before
> > though, if you can safely try it out, it might help in narrowing down the
> > source of the issue.
> >
> > I'd really like to reiterate that version upgrades to any potentially
> > involved parts (broker, JVM, OS, etc) should ideally be considered as a
> > first step.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> > For additional commands, e-mail: users-help@activemq.apache.org
> > For further information, visit: https://activemq.apache.org/contact
> >
> >
> >
>
Re: ActiveMq 2.14.0 Native Memory Leak
Posted by Justin Bertram <jb...@apache.org>.
I agree with Anton here 100%.
Netty uses direct memory for performance reasons. If you disable it there
will certainly be performance trade-offs. Whether that really impacts you
depends on your use-case.
Also, keep in mind that if Netty doesn't use direct memory it will use the
JVM's heap instead. You may need to increase the heap size to account for
this. Only testing will show.
Justin
On Thu, Feb 20, 2025 at 3:46 AM Anton Roskvist <an...@apache.org> wrote:
> Sorry, but to my knowledge that is almost impossible to say for sure, as
> it's highly dependent on your specific use case and setup. As before
> though, if you can safely try it out, it might help in narrowing down the
> source of the issue.
>
> I'd really like to reiterate that version upgrades to any potentially
> involved parts (broker, JVM, OS, etc) should ideally be considered as a
> first step.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> For additional commands, e-mail: users-help@activemq.apache.org
> For further information, visit: https://activemq.apache.org/contact
>
>
>
Re: ActiveMq 2.14.0 Native Memory Leak
Posted by Domenico Francesco Bruscino <br...@gmail.com>.
Send a blank email to users-unsubscribe@activemq.apache.org to unsubscribe
from ActiveMQ Users mailing list, for further details see
https://activemq.apache.org/contact
On Thu, 20 Feb 2025 at 11:15, 23F3000346 NASHEETA FARZANA <
23f3000346@ds.study.iitm.ac.in> wrote:
> Please unsubscribe me from this mailing list
>
> On Thu, 20 Feb, 2025, 3:15 pm Anton Roskvist, <an...@apache.org> wrote:
>
> > Sorry, but to my knowledge that is almost impossible to say for sure, as
> > it's highly dependent on your specific use case and setup. As before
> > though, if you can safely try it out, it might help in narrowing down the
> > source of the issue.
> >
> > I'd really like to reiterate that version upgrades to any potentially
> > involved parts (broker, JVM, OS, etc) should ideally be considered as a
> > first step.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> > For additional commands, e-mail: users-help@activemq.apache.org
> > For further information, visit: https://activemq.apache.org/contact
> >
> >
> >
>
Re: ActiveMq 2.14.0 Native Memory Leak
Posted by 23F3000346 NASHEETA FARZANA <23...@ds.study.iitm.ac.in>.
Please unsubscribe me from this mailing list
On Thu, 20 Feb, 2025, 3:15 pm Anton Roskvist, <an...@apache.org> wrote:
> Sorry, but to my knowledge that is almost impossible to say for sure, as
> it's highly dependent on your specific use case and setup. As before
> though, if you can safely try it out, it might help in narrowing down the
> source of the issue.
>
> I'd really like to reiterate that version upgrades to any potentially
> involved parts (broker, JVM, OS, etc) should ideally be considered as a
> first step.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> For additional commands, e-mail: users-help@activemq.apache.org
> For further information, visit: https://activemq.apache.org/contact
>
>
>
Re: ActiveMq 2.14.0 Native Memory Leak
Posted by Anton Roskvist <an...@apache.org>.
Sorry, but to my knowledge that is almost impossible to say for sure, as it's highly dependent on your specific use case and setup. As before though, if you can safely try it out, it might help in narrowing down the source of the issue.
I'd really like to reiterate that version upgrades to any potentially involved parts (broker, JVM, OS, etc) should ideally be considered as a first step.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
For additional commands, e-mail: users-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
Re: ActiveMq 2.14.0 Native Memory Leak
Posted by Franck Malka <fm...@modulo.co.il>.
Ok,
In case the problem is direct memory
Do you think disable netty direct memory would have a serious impact on
performance?
E.g.
-Dio.netty.noPreferDirect=true -Dio.netty.maxDirectMemory=0
-Dio.netty.allocator.type=unpooled
On Thu, Feb 20, 2025 at 10:47 AM Anton Roskvist <an...@apache.org> wrote:
> Hi,
>
> I think the point would be that if possible, the idea would be to try the
> same conditions against a more recent version of the broker. All else being
> equal, this would rule out any and all issues that might already have been
> addressed in the roughly 3000 commits since 2.14.0. Same could apply to
> your JVM.
>
> If it's still an issue, then at least the scope has been narrowed.
>
> As an alternative if you are unable to upgrade, "Native memory tracking"
> or "Flight recording" can be helpful in tracking these types of issues
> down. Refer to the documentation of your JDK on how to enable these (though
> be aware that they incur some overhead while running).
>
> Switching to NIO is possible but it could very well affect your
> performance in a negative way... though depending on your options it could
> serve as yet another way of narrowing the scope of this issue.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> For additional commands, e-mail: users-help@activemq.apache.org
> For further information, visit: https://activemq.apache.org/contact
>
>
>
Re: ActiveMq 2.14.0 Native Memory Leak
Posted by Anton Roskvist <an...@apache.org>.
Hi,
I think the point would be that if possible, the idea would be to try the same conditions against a more recent version of the broker. All else being equal, this would rule out any and all issues that might already have been addressed in the roughly 3000 commits since 2.14.0. Same could apply to your JVM.
If it's still an issue, then at least the scope has been narrowed.
As an alternative if you are unable to upgrade, "Native memory tracking" or "Flight recording" can be helpful in tracking these types of issues down. Refer to the documentation of your JDK on how to enable these (though be aware that they incur some overhead while running).
Switching to NIO is possible but it could very well affect your performance in a negative way... though depending on your options it could serve as yet another way of narrowing the scope of this issue.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
For additional commands, e-mail: users-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
Re: ActiveMq 2.14.0 Native Memory Leak
Posted by Justin Bertram <jb...@apache.org>.
In absolute terms AIO is faster than NIO. That's the whole reason we
created the activemq-artemis-native integration layer.
However, whether this difference actually impacts your deployment depends
on whether disk writes are a bottleneck. Therefore, it's really impossible
for us to answer this question in a holistic way.
I do believe Netty's direct memory usage is configurable, but I don't know
the details.
Justin
On Thu, Feb 20, 2025 at 12:45 AM Franck Malka <fm...@modulo.co.il> wrote:
> Hi,
>
> Yes, I use ASYNCIO and libAIO.
> I have seen performance is better than NIO.
> You think I can achieve the same performance with NIO?
> Do you think I can disable netty direct memory?
>
> -Franck.
>
> On Thu, Feb 20, 2025 at 8:05 AM Justin Bertram <jb...@apache.org>
> wrote:
>
> > It's really impossible to say whether or not this is some kind of memory
> > leak which might be fixed in a later release. However, I do know that
> > 2.14.0 was released over 4 years ago now and there's been almost 3,000
> > commits between then and 2.39.0 (i.e. the current release).
> >
> > Regarding the specifics of your problem...Do you really mean "native"
> > memory or are you talking about "direct" memory?
> >
> > The only thing which I think would be using native memory is the
> > artemis-native JNI library [1] which serves as an interface between the
> > broker and libAIO on Linux. Are you using this?
> >
> > On the other hand, Netty is a big user of direct memory. Artemis 2.14.0
> > shipped 4.1.48.Final and 2.39.0 shipped 4.1.115.Final so there's a fairly
> > big difference there.
> >
> > Generally speaking we encourage folks to stay on the latest release
> > possible so when they have issues we can rule out things that have
> already
> > been fixed. Are you able to at least try the latest release to see if it
> > resolves the issue you're seeing (assuming it is a legitimate problem)?
> If
> > not, could you provide any kind of test-case which reproduces the
> problem?
> >
> >
> > Justin
> >
> > [1] https://github.com/apache/activemq-artemis-native
> >
> > On Wed, Feb 19, 2025 at 5:35 AM Franck Malka <fm...@modulo.co.il>
> wrote:
> >
> > > Hi, all
> > >
> > > I have a case with an Artemis ActiveMq cluster which process a very
> large
> > > number of messages per second.
> > >
> > > Each few weeks of operation, JVM memory is still cleaning properly but
> > > native memory is not getting cleared.
> > >
> > > Did anyone encounter this case?
> > > Does it sounds more like a configuration issue? Or like something
> > upgrading
> > > to latest version would address?
> > >
> > > Franck.
> > >
> >
>
Re: ActiveMq 2.14.0 Native Memory Leak
Posted by Franck Malka <fm...@modulo.co.il>.
Hi,
Yes, I use ASYNCIO and libAIO.
I have seen performance is better than NIO.
You think I can achieve the same performance with NIO?
Do you think I can disable netty direct memory?
-Franck.
On Thu, Feb 20, 2025 at 8:05 AM Justin Bertram <jb...@apache.org> wrote:
> It's really impossible to say whether or not this is some kind of memory
> leak which might be fixed in a later release. However, I do know that
> 2.14.0 was released over 4 years ago now and there's been almost 3,000
> commits between then and 2.39.0 (i.e. the current release).
>
> Regarding the specifics of your problem...Do you really mean "native"
> memory or are you talking about "direct" memory?
>
> The only thing which I think would be using native memory is the
> artemis-native JNI library [1] which serves as an interface between the
> broker and libAIO on Linux. Are you using this?
>
> On the other hand, Netty is a big user of direct memory. Artemis 2.14.0
> shipped 4.1.48.Final and 2.39.0 shipped 4.1.115.Final so there's a fairly
> big difference there.
>
> Generally speaking we encourage folks to stay on the latest release
> possible so when they have issues we can rule out things that have already
> been fixed. Are you able to at least try the latest release to see if it
> resolves the issue you're seeing (assuming it is a legitimate problem)? If
> not, could you provide any kind of test-case which reproduces the problem?
>
>
> Justin
>
> [1] https://github.com/apache/activemq-artemis-native
>
> On Wed, Feb 19, 2025 at 5:35 AM Franck Malka <fm...@modulo.co.il> wrote:
>
> > Hi, all
> >
> > I have a case with an Artemis ActiveMq cluster which process a very large
> > number of messages per second.
> >
> > Each few weeks of operation, JVM memory is still cleaning properly but
> > native memory is not getting cleared.
> >
> > Did anyone encounter this case?
> > Does it sounds more like a configuration issue? Or like something
> upgrading
> > to latest version would address?
> >
> > Franck.
> >
>
Re: ActiveMq 2.14.0 Native Memory Leak
Posted by Justin Bertram <jb...@apache.org>.
It's really impossible to say whether or not this is some kind of memory
leak which might be fixed in a later release. However, I do know that
2.14.0 was released over 4 years ago now and there's been almost 3,000
commits between then and 2.39.0 (i.e. the current release).
Regarding the specifics of your problem...Do you really mean "native"
memory or are you talking about "direct" memory?
The only thing which I think would be using native memory is the
artemis-native JNI library [1] which serves as an interface between the
broker and libAIO on Linux. Are you using this?
On the other hand, Netty is a big user of direct memory. Artemis 2.14.0
shipped 4.1.48.Final and 2.39.0 shipped 4.1.115.Final so there's a fairly
big difference there.
Generally speaking we encourage folks to stay on the latest release
possible so when they have issues we can rule out things that have already
been fixed. Are you able to at least try the latest release to see if it
resolves the issue you're seeing (assuming it is a legitimate problem)? If
not, could you provide any kind of test-case which reproduces the problem?
Justin
[1] https://github.com/apache/activemq-artemis-native
On Wed, Feb 19, 2025 at 5:35 AM Franck Malka <fm...@modulo.co.il> wrote:
> Hi, all
>
> I have a case with an Artemis ActiveMq cluster which process a very large
> number of messages per second.
>
> Each few weeks of operation, JVM memory is still cleaning properly but
> native memory is not getting cleared.
>
> Did anyone encounter this case?
> Does it sounds more like a configuration issue? Or like something upgrading
> to latest version would address?
>
> Franck.
>