You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Ben Warrick <ho...@gmail.com> on 2022/06/03 19:54:57 UTC
Optimizing for many short connections
I'm running Artemis 2.14, and one of the systems publishing to it creates a
lot of separate tasks -- from its perspective. I can see on Artemis that
the Total connection count vs Total messages acknowledged is pretty close,
so it's apparent that the client system is making use of the anti-pattern
of creating a new connection for each message it publishes. I don't have
access to the client system implementation, but is there anything I can do
on the Artemis end to make it more efficient? Latency with this system is
becoming an issue. Also every 30 minutes or so there is a spike in CPU
usage (~50%) which I assume is garbage collection. During that spike
message publishing slows down a lot.
Thanks,
Ben
Re: Optimizing for many short connections
Posted by Clebert Suconic <cl...@gmail.com>.
the image.png you sent could not be delivered by the infra for some reason.
It's rare an occasion where I affirm I am 100% sure about something. I
usually always keep a margin for the changing my opinion, or just being
wrong.. But on this case I am 10000% sure you will hit the bug I fixed.
The issue is on creating new sessions. I added some instructions on how you
could reproduce the issue on my last comment:
https://issues.apache.org/jira/browse/ARTEMIS-3848
Also, as part of my day job (I work for Red Hat), I have a JIRA where I
also describe the issue:
https://issues.redhat.com/browse/ENTMQBR-6683
I would just go straight to the upgrade to be honest.. you could do some
acceptance test as it's usually standard before you accept any new software
into your production system.
On Tue, Jun 7, 2022 at 5:47 PM Ben Warrick <ho...@gmail.com>
wrote:
> @Clebert: Thanks for finding this. Here is some more evidence for you.
> This is only one monkey on my back right now, but I do plan on upgrading.
> I'll get you some feedback for sure. If I can manage, I want to try
> reproducing this in a lower environment.
>
> [image: image.png]
>
> On Tue, Jun 7, 2022 at 5:07 PM Clebert Suconic <cl...@gmail.com>
> wrote:
>
>> @Ben Warrick: BTW When I said "you are likely" hitting the bug, I was
>> just using polite words...
>>
>>
>> I am 100% sure you are hitting that condition by creating one session
>> per message. You should definitely look to upgrade or cherry-pick the
>> fix to your own branch.
>>
>> On Tue, Jun 7, 2022 at 3:14 PM Clebert Suconic
>> <cl...@gmail.com> wrote:
>> >
>> > The release candidate :
>> >
>> >
>> > https://dist.apache.org/repos/dist/dev/activemq/activemq-artemis/2.23.0
>> >
>> > I would appreciate your feedback here or even better on the dev list at
>> the voting thread.
>> >
>> > On Tue, Jun 7, 2022 at 12:52 PM Clebert Suconic <
>> clebert.suconic@gmail.com> wrote:
>> >>
>> >> right, but since I'm just releasing it would be great to hear his
>> >> feedback on top of the release candidate itself.
>> >>
>> >> On Tue, Jun 7, 2022 at 12:38 PM Justin Bertram <jb...@apache.org>
>> wrote:
>> >> >
>> >> > A snapshot [1] is always a possibility as well.
>> >> >
>> >> >
>> >> > Justin
>> >> >
>> >> > [1]
>> >> >
>> https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-artemis/2.23.0-SNAPSHOT/
>> >> >
>> >> > On Tue, Jun 7, 2022 at 11:34 AM Clebert Suconic <
>> clebert.suconic@gmail.com>
>> >> > wrote:
>> >> >
>> >> > > I am preparing artemis 2.23.0 at the moment...
>> >> > >
>> >> > > The way a release works in apache, you send it for vote and it
>> will be
>> >> > > officially released after 72 hours of a passing vote. but you
>> should
>> >> > > have a candidate release today that you can try it out. Based on
>> the
>> >> > > current diffs it should be fairly safe for you to upgrade.
>> >> > >
>> >> > > 2.14.0 would require JDK 1.8 minimal. while artemis 2.23 will
>> require
>> >> > > JDK 11 minimal.
>> >> > >
>> >> > >
>> >> > > I will send you a link to the Candidate release as soon as it's
>> ready
>> >> > > on this thread.
>> >> > >
>> >> > > On Mon, Jun 6, 2022 at 10:25 PM Ben Warrick <
>> horseatingweeds@gmail.com>
>> >> > > wrote:
>> >> > > >
>> >> > > > Hi Clebert,
>> >> > > >
>> >> > > > Yeah, I can use whatever I want after I prove it out in lower
>> >> > > environments.
>> >> > > >
>> >> > > > Thanks,
>> >> > > >
>> >> > > > Ben
>> >> > > >
>> >> > > > On Mon, Jun 6, 2022 at 10:06 PM Clebert Suconic <
>> >> > > clebert.suconic@gmail.com>
>> >> > > > wrote:
>> >> > > >
>> >> > > > > Can you use jdk 11?
>> >> > > > >
>> >> > > > > On Mon, Jun 6, 2022 at 8:32 PM Clebert Suconic <
>> >> > > clebert.suconic@gmail.com>
>> >> > > > > wrote:
>> >> > > > >
>> >> > > > > > You are likely hitting the bug I fixed at:
>> >> > > > > >
>> >> > > > > > https://github.com/apache/activemq-artemis/pull/4098
>> >> > > > > >
>> >> > > > > > I introduced a Thread Local Leakage. I thought by setting
>> >> > > > > > threadLocal.set(null) I would be clearing it... and in fact
>> I just
>> >> > > > > > updated the threadLocal variable with null.
>> >> > > > > >
>> >> > > > > > That ThreadLocal is created for every new session.
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > I was going to start a release today but I couldn't start it
>> today as
>> >> > > > > > I'm a bit sick today. I should be able to start a release
>> tomorrow
>> >> > > and
>> >> > > > > > then you could try it out.
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > your other option would be to cherry-pick the change and
>> build the
>> >> > > > > > version yourself so you can run it in production.
>> >> > > > > >
>> >> > > > > > On Fri, Jun 3, 2022 at 4:27 PM Francesco Nigro <
>> nigro.fra@gmail.com>
>> >> > > > > > wrote:
>> >> > > > > > >
>> >> > > > > > > Some more info about it here
>> >> > > https://netty.io/wiki/tcp-fast-open.html
>> >> > > > > > >
>> >> > > > > > > Given that Netty can makes use of it and Artemis can as
>> well: if
>> >> > > it's
>> >> > > > > not
>> >> > > > > > > present in the doc it means we need to expose it in the
>> acceptor
>> >> > > > > > > configuration (for epoll) and...we love contributions :)
>> >> > > > > > > But I suggest first to give it a shot first in a test env
>> to see
>> >> > > if it
>> >> > > > > > > solves your problem!
>> >> > > > > > >
>> >> > > > > > > Il ven 3 giu 2022, 22:21 Ben Warrick <
>> horseatingweeds@gmail.com>
>> >> > > ha
>> >> > > > > > scritto:
>> >> > > > > > >
>> >> > > > > > > > Hi Francesco,
>> >> > > > > > > >
>> >> > > > > > > > I don't find anything in the Artemis docs about
>> TCP_FAST_OPEN. It
>> >> > > > > must
>> >> > > > > > not
>> >> > > > > > > > be an exposed configuration on Artemis? Where do I find
>> more on
>> >> > > this?
>> >> > > > > > > >
>> >> > > > > > > > Thanks,
>> >> > > > > > > >
>> >> > > > > > > > Ben
>> >> > > > > > > >
>> >> > > > > > > > On Fri, Jun 3, 2022 at 4:08 PM Francesco Nigro <
>> >> > > nigro.fra@gmail.com>
>> >> > > > > > > > wrote:
>> >> > > > > > > >
>> >> > > > > > > > > Are coming, not commenting: typo introduced by the the
>> phone,
>> >> > > > > sorry!
>> >> > > > > > > > >
>> >> > > > > > > > > Il ven 3 giu 2022, 22:07 Francesco Nigro <
>> nigro.fra@gmail.com>
>> >> > > ha
>> >> > > > > > > > scritto:
>> >> > > > > > > > >
>> >> > > > > > > > > > Hi Ben,
>> >> > > > > > > > > >
>> >> > > > > > > > > > You can use TCP_FAST_OPEN on Artemis acceptors in
>> case
>> >> > > > > connections
>> >> > > > > > are
>> >> > > > > > > > > > commenting from the came clients (I don't remember
>> if is an
>> >> > > > > exposes
>> >> > > > > > > > Netty
>> >> > > > > > > > > > configuration) or...submit a PR to let Artemis
>> separate
>> >> > > > > (multiple)
>> >> > > > > > > > > threads
>> >> > > > > > > > > > to accept connections but, as you said, this is just
>> an
>> >> > > > > > anti-pattern
>> >> > > > > > > > and
>> >> > > > > > > > > in
>> >> > > > > > > > > > order to reduce the client code you can think about
>> using a
>> >> > > > > > transparent
>> >> > > > > > > > > > connection pooling mechanism, instead.
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > > > Il ven 3 giu 2022, 21:55 Ben Warrick <
>> >> > > horseatingweeds@gmail.com>
>> >> > > > > > ha
>> >> > > > > > > > > > scritto:
>> >> > > > > > > > > >
>> >> > > > > > > > > >> I'm running Artemis 2.14, and one of the systems
>> publishing
>> >> > > to
>> >> > > > > it
>> >> > > > > > > > > creates
>> >> > > > > > > > > >> a
>> >> > > > > > > > > >> lot of separate tasks -- from its perspective. I
>> can see on
>> >> > > > > > Artemis
>> >> > > > > > > > that
>> >> > > > > > > > > >> the Total connection count vs Total messages
>> acknowledged is
>> >> > > > > > pretty
>> >> > > > > > > > > close,
>> >> > > > > > > > > >> so it's apparent that the client system is making
>> use of the
>> >> > > > > > > > > anti-pattern
>> >> > > > > > > > > >> of creating a new connection for each message it
>> publishes.
>> >> > > I
>> >> > > > > > don't
>> >> > > > > > > > have
>> >> > > > > > > > > >> access to the client system implementation, but is
>> there
>> >> > > > > anything
>> >> > > > > > I
>> >> > > > > > > > can
>> >> > > > > > > > > do
>> >> > > > > > > > > >> on the Artemis end to make it more efficient?
>> Latency with
>> >> > > this
>> >> > > > > > system
>> >> > > > > > > > > is
>> >> > > > > > > > > >> becoming an issue. Also every 30 minutes or so
>> there is a
>> >> > > spike
>> >> > > > > > in CPU
>> >> > > > > > > > > >> usage (~50%) which I assume is garbage collection.
>> During
>> >> > > that
>> >> > > > > > spike
>> >> > > > > > > > > >> message publishing slows down a lot.
>> >> > > > > > > > > >>
>> >> > > > > > > > > >> Thanks,
>> >> > > > > > > > > >>
>> >> > > > > > > > > >> Ben
>> >> > > > > > > > > >>
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > >
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > --
>> >> > > > > > Clebert Suconic
>> >> > > > > >
>> >> > > > > --
>> >> > > > > Clebert Suconic
>> >> > > > >
>> >> > >
>> >> > >
>> >> > >
>> >> > > --
>> >> > > Clebert Suconic
>> >> > >
>> >> > >
>> >>
>> >>
>> >>
>> >> --
>> >> Clebert Suconic
>> >
>> > --
>> > Clebert Suconic
>>
>>
>>
>> --
>> Clebert Suconic
>>
>
--
Clebert Suconic
Re: Optimizing for many short connections
Posted by Ben Warrick <ho...@gmail.com>.
@Clebert: Thanks for finding this. Here is some more evidence for you. This
is only one monkey on my back right now, but I do plan on upgrading. I'll
get you some feedback for sure. If I can manage, I want to try reproducing
this in a lower environment.
[image: image.png]
On Tue, Jun 7, 2022 at 5:07 PM Clebert Suconic <cl...@gmail.com>
wrote:
> @Ben Warrick: BTW When I said "you are likely" hitting the bug, I was
> just using polite words...
>
>
> I am 100% sure you are hitting that condition by creating one session
> per message. You should definitely look to upgrade or cherry-pick the
> fix to your own branch.
>
> On Tue, Jun 7, 2022 at 3:14 PM Clebert Suconic
> <cl...@gmail.com> wrote:
> >
> > The release candidate :
> >
> >
> > https://dist.apache.org/repos/dist/dev/activemq/activemq-artemis/2.23.0
> >
> > I would appreciate your feedback here or even better on the dev list at
> the voting thread.
> >
> > On Tue, Jun 7, 2022 at 12:52 PM Clebert Suconic <
> clebert.suconic@gmail.com> wrote:
> >>
> >> right, but since I'm just releasing it would be great to hear his
> >> feedback on top of the release candidate itself.
> >>
> >> On Tue, Jun 7, 2022 at 12:38 PM Justin Bertram <jb...@apache.org>
> wrote:
> >> >
> >> > A snapshot [1] is always a possibility as well.
> >> >
> >> >
> >> > Justin
> >> >
> >> > [1]
> >> >
> https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-artemis/2.23.0-SNAPSHOT/
> >> >
> >> > On Tue, Jun 7, 2022 at 11:34 AM Clebert Suconic <
> clebert.suconic@gmail.com>
> >> > wrote:
> >> >
> >> > > I am preparing artemis 2.23.0 at the moment...
> >> > >
> >> > > The way a release works in apache, you send it for vote and it will
> be
> >> > > officially released after 72 hours of a passing vote. but you should
> >> > > have a candidate release today that you can try it out. Based on the
> >> > > current diffs it should be fairly safe for you to upgrade.
> >> > >
> >> > > 2.14.0 would require JDK 1.8 minimal. while artemis 2.23 will
> require
> >> > > JDK 11 minimal.
> >> > >
> >> > >
> >> > > I will send you a link to the Candidate release as soon as it's
> ready
> >> > > on this thread.
> >> > >
> >> > > On Mon, Jun 6, 2022 at 10:25 PM Ben Warrick <
> horseatingweeds@gmail.com>
> >> > > wrote:
> >> > > >
> >> > > > Hi Clebert,
> >> > > >
> >> > > > Yeah, I can use whatever I want after I prove it out in lower
> >> > > environments.
> >> > > >
> >> > > > Thanks,
> >> > > >
> >> > > > Ben
> >> > > >
> >> > > > On Mon, Jun 6, 2022 at 10:06 PM Clebert Suconic <
> >> > > clebert.suconic@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > > Can you use jdk 11?
> >> > > > >
> >> > > > > On Mon, Jun 6, 2022 at 8:32 PM Clebert Suconic <
> >> > > clebert.suconic@gmail.com>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > You are likely hitting the bug I fixed at:
> >> > > > > >
> >> > > > > > https://github.com/apache/activemq-artemis/pull/4098
> >> > > > > >
> >> > > > > > I introduced a Thread Local Leakage. I thought by setting
> >> > > > > > threadLocal.set(null) I would be clearing it... and in fact I
> just
> >> > > > > > updated the threadLocal variable with null.
> >> > > > > >
> >> > > > > > That ThreadLocal is created for every new session.
> >> > > > > >
> >> > > > > >
> >> > > > > > I was going to start a release today but I couldn't start it
> today as
> >> > > > > > I'm a bit sick today. I should be able to start a release
> tomorrow
> >> > > and
> >> > > > > > then you could try it out.
> >> > > > > >
> >> > > > > >
> >> > > > > > your other option would be to cherry-pick the change and
> build the
> >> > > > > > version yourself so you can run it in production.
> >> > > > > >
> >> > > > > > On Fri, Jun 3, 2022 at 4:27 PM Francesco Nigro <
> nigro.fra@gmail.com>
> >> > > > > > wrote:
> >> > > > > > >
> >> > > > > > > Some more info about it here
> >> > > https://netty.io/wiki/tcp-fast-open.html
> >> > > > > > >
> >> > > > > > > Given that Netty can makes use of it and Artemis can as
> well: if
> >> > > it's
> >> > > > > not
> >> > > > > > > present in the doc it means we need to expose it in the
> acceptor
> >> > > > > > > configuration (for epoll) and...we love contributions :)
> >> > > > > > > But I suggest first to give it a shot first in a test env
> to see
> >> > > if it
> >> > > > > > > solves your problem!
> >> > > > > > >
> >> > > > > > > Il ven 3 giu 2022, 22:21 Ben Warrick <
> horseatingweeds@gmail.com>
> >> > > ha
> >> > > > > > scritto:
> >> > > > > > >
> >> > > > > > > > Hi Francesco,
> >> > > > > > > >
> >> > > > > > > > I don't find anything in the Artemis docs about
> TCP_FAST_OPEN. It
> >> > > > > must
> >> > > > > > not
> >> > > > > > > > be an exposed configuration on Artemis? Where do I find
> more on
> >> > > this?
> >> > > > > > > >
> >> > > > > > > > Thanks,
> >> > > > > > > >
> >> > > > > > > > Ben
> >> > > > > > > >
> >> > > > > > > > On Fri, Jun 3, 2022 at 4:08 PM Francesco Nigro <
> >> > > nigro.fra@gmail.com>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Are coming, not commenting: typo introduced by the the
> phone,
> >> > > > > sorry!
> >> > > > > > > > >
> >> > > > > > > > > Il ven 3 giu 2022, 22:07 Francesco Nigro <
> nigro.fra@gmail.com>
> >> > > ha
> >> > > > > > > > scritto:
> >> > > > > > > > >
> >> > > > > > > > > > Hi Ben,
> >> > > > > > > > > >
> >> > > > > > > > > > You can use TCP_FAST_OPEN on Artemis acceptors in case
> >> > > > > connections
> >> > > > > > are
> >> > > > > > > > > > commenting from the came clients (I don't remember if
> is an
> >> > > > > exposes
> >> > > > > > > > Netty
> >> > > > > > > > > > configuration) or...submit a PR to let Artemis
> separate
> >> > > > > (multiple)
> >> > > > > > > > > threads
> >> > > > > > > > > > to accept connections but, as you said, this is just
> an
> >> > > > > > anti-pattern
> >> > > > > > > > and
> >> > > > > > > > > in
> >> > > > > > > > > > order to reduce the client code you can think about
> using a
> >> > > > > > transparent
> >> > > > > > > > > > connection pooling mechanism, instead.
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > Il ven 3 giu 2022, 21:55 Ben Warrick <
> >> > > horseatingweeds@gmail.com>
> >> > > > > > ha
> >> > > > > > > > > > scritto:
> >> > > > > > > > > >
> >> > > > > > > > > >> I'm running Artemis 2.14, and one of the systems
> publishing
> >> > > to
> >> > > > > it
> >> > > > > > > > > creates
> >> > > > > > > > > >> a
> >> > > > > > > > > >> lot of separate tasks -- from its perspective. I can
> see on
> >> > > > > > Artemis
> >> > > > > > > > that
> >> > > > > > > > > >> the Total connection count vs Total messages
> acknowledged is
> >> > > > > > pretty
> >> > > > > > > > > close,
> >> > > > > > > > > >> so it's apparent that the client system is making
> use of the
> >> > > > > > > > > anti-pattern
> >> > > > > > > > > >> of creating a new connection for each message it
> publishes.
> >> > > I
> >> > > > > > don't
> >> > > > > > > > have
> >> > > > > > > > > >> access to the client system implementation, but is
> there
> >> > > > > anything
> >> > > > > > I
> >> > > > > > > > can
> >> > > > > > > > > do
> >> > > > > > > > > >> on the Artemis end to make it more efficient?
> Latency with
> >> > > this
> >> > > > > > system
> >> > > > > > > > > is
> >> > > > > > > > > >> becoming an issue. Also every 30 minutes or so there
> is a
> >> > > spike
> >> > > > > > in CPU
> >> > > > > > > > > >> usage (~50%) which I assume is garbage collection.
> During
> >> > > that
> >> > > > > > spike
> >> > > > > > > > > >> message publishing slows down a lot.
> >> > > > > > > > > >>
> >> > > > > > > > > >> Thanks,
> >> > > > > > > > > >>
> >> > > > > > > > > >> Ben
> >> > > > > > > > > >>
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > --
> >> > > > > > Clebert Suconic
> >> > > > > >
> >> > > > > --
> >> > > > > Clebert Suconic
> >> > > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > Clebert Suconic
> >> > >
> >> > >
> >>
> >>
> >>
> >> --
> >> Clebert Suconic
> >
> > --
> > Clebert Suconic
>
>
>
> --
> Clebert Suconic
>
Re: Optimizing for many short connections
Posted by Clebert Suconic <cl...@gmail.com>.
@Ben Warrick: BTW When I said "you are likely" hitting the bug, I was
just using polite words...
I am 100% sure you are hitting that condition by creating one session
per message. You should definitely look to upgrade or cherry-pick the
fix to your own branch.
On Tue, Jun 7, 2022 at 3:14 PM Clebert Suconic
<cl...@gmail.com> wrote:
>
> The release candidate :
>
>
> https://dist.apache.org/repos/dist/dev/activemq/activemq-artemis/2.23.0
>
> I would appreciate your feedback here or even better on the dev list at the voting thread.
>
> On Tue, Jun 7, 2022 at 12:52 PM Clebert Suconic <cl...@gmail.com> wrote:
>>
>> right, but since I'm just releasing it would be great to hear his
>> feedback on top of the release candidate itself.
>>
>> On Tue, Jun 7, 2022 at 12:38 PM Justin Bertram <jb...@apache.org> wrote:
>> >
>> > A snapshot [1] is always a possibility as well.
>> >
>> >
>> > Justin
>> >
>> > [1]
>> > https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-artemis/2.23.0-SNAPSHOT/
>> >
>> > On Tue, Jun 7, 2022 at 11:34 AM Clebert Suconic <cl...@gmail.com>
>> > wrote:
>> >
>> > > I am preparing artemis 2.23.0 at the moment...
>> > >
>> > > The way a release works in apache, you send it for vote and it will be
>> > > officially released after 72 hours of a passing vote. but you should
>> > > have a candidate release today that you can try it out. Based on the
>> > > current diffs it should be fairly safe for you to upgrade.
>> > >
>> > > 2.14.0 would require JDK 1.8 minimal. while artemis 2.23 will require
>> > > JDK 11 minimal.
>> > >
>> > >
>> > > I will send you a link to the Candidate release as soon as it's ready
>> > > on this thread.
>> > >
>> > > On Mon, Jun 6, 2022 at 10:25 PM Ben Warrick <ho...@gmail.com>
>> > > wrote:
>> > > >
>> > > > Hi Clebert,
>> > > >
>> > > > Yeah, I can use whatever I want after I prove it out in lower
>> > > environments.
>> > > >
>> > > > Thanks,
>> > > >
>> > > > Ben
>> > > >
>> > > > On Mon, Jun 6, 2022 at 10:06 PM Clebert Suconic <
>> > > clebert.suconic@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Can you use jdk 11?
>> > > > >
>> > > > > On Mon, Jun 6, 2022 at 8:32 PM Clebert Suconic <
>> > > clebert.suconic@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > You are likely hitting the bug I fixed at:
>> > > > > >
>> > > > > > https://github.com/apache/activemq-artemis/pull/4098
>> > > > > >
>> > > > > > I introduced a Thread Local Leakage. I thought by setting
>> > > > > > threadLocal.set(null) I would be clearing it... and in fact I just
>> > > > > > updated the threadLocal variable with null.
>> > > > > >
>> > > > > > That ThreadLocal is created for every new session.
>> > > > > >
>> > > > > >
>> > > > > > I was going to start a release today but I couldn't start it today as
>> > > > > > I'm a bit sick today. I should be able to start a release tomorrow
>> > > and
>> > > > > > then you could try it out.
>> > > > > >
>> > > > > >
>> > > > > > your other option would be to cherry-pick the change and build the
>> > > > > > version yourself so you can run it in production.
>> > > > > >
>> > > > > > On Fri, Jun 3, 2022 at 4:27 PM Francesco Nigro <ni...@gmail.com>
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > Some more info about it here
>> > > https://netty.io/wiki/tcp-fast-open.html
>> > > > > > >
>> > > > > > > Given that Netty can makes use of it and Artemis can as well: if
>> > > it's
>> > > > > not
>> > > > > > > present in the doc it means we need to expose it in the acceptor
>> > > > > > > configuration (for epoll) and...we love contributions :)
>> > > > > > > But I suggest first to give it a shot first in a test env to see
>> > > if it
>> > > > > > > solves your problem!
>> > > > > > >
>> > > > > > > Il ven 3 giu 2022, 22:21 Ben Warrick <ho...@gmail.com>
>> > > ha
>> > > > > > scritto:
>> > > > > > >
>> > > > > > > > Hi Francesco,
>> > > > > > > >
>> > > > > > > > I don't find anything in the Artemis docs about TCP_FAST_OPEN. It
>> > > > > must
>> > > > > > not
>> > > > > > > > be an exposed configuration on Artemis? Where do I find more on
>> > > this?
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > >
>> > > > > > > > Ben
>> > > > > > > >
>> > > > > > > > On Fri, Jun 3, 2022 at 4:08 PM Francesco Nigro <
>> > > nigro.fra@gmail.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Are coming, not commenting: typo introduced by the the phone,
>> > > > > sorry!
>> > > > > > > > >
>> > > > > > > > > Il ven 3 giu 2022, 22:07 Francesco Nigro <ni...@gmail.com>
>> > > ha
>> > > > > > > > scritto:
>> > > > > > > > >
>> > > > > > > > > > Hi Ben,
>> > > > > > > > > >
>> > > > > > > > > > You can use TCP_FAST_OPEN on Artemis acceptors in case
>> > > > > connections
>> > > > > > are
>> > > > > > > > > > commenting from the came clients (I don't remember if is an
>> > > > > exposes
>> > > > > > > > Netty
>> > > > > > > > > > configuration) or...submit a PR to let Artemis separate
>> > > > > (multiple)
>> > > > > > > > > threads
>> > > > > > > > > > to accept connections but, as you said, this is just an
>> > > > > > anti-pattern
>> > > > > > > > and
>> > > > > > > > > in
>> > > > > > > > > > order to reduce the client code you can think about using a
>> > > > > > transparent
>> > > > > > > > > > connection pooling mechanism, instead.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Il ven 3 giu 2022, 21:55 Ben Warrick <
>> > > horseatingweeds@gmail.com>
>> > > > > > ha
>> > > > > > > > > > scritto:
>> > > > > > > > > >
>> > > > > > > > > >> I'm running Artemis 2.14, and one of the systems publishing
>> > > to
>> > > > > it
>> > > > > > > > > creates
>> > > > > > > > > >> a
>> > > > > > > > > >> lot of separate tasks -- from its perspective. I can see on
>> > > > > > Artemis
>> > > > > > > > that
>> > > > > > > > > >> the Total connection count vs Total messages acknowledged is
>> > > > > > pretty
>> > > > > > > > > close,
>> > > > > > > > > >> so it's apparent that the client system is making use of the
>> > > > > > > > > anti-pattern
>> > > > > > > > > >> of creating a new connection for each message it publishes.
>> > > I
>> > > > > > don't
>> > > > > > > > have
>> > > > > > > > > >> access to the client system implementation, but is there
>> > > > > anything
>> > > > > > I
>> > > > > > > > can
>> > > > > > > > > do
>> > > > > > > > > >> on the Artemis end to make it more efficient? Latency with
>> > > this
>> > > > > > system
>> > > > > > > > > is
>> > > > > > > > > >> becoming an issue. Also every 30 minutes or so there is a
>> > > spike
>> > > > > > in CPU
>> > > > > > > > > >> usage (~50%) which I assume is garbage collection. During
>> > > that
>> > > > > > spike
>> > > > > > > > > >> message publishing slows down a lot.
>> > > > > > > > > >>
>> > > > > > > > > >> Thanks,
>> > > > > > > > > >>
>> > > > > > > > > >> Ben
>> > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > Clebert Suconic
>> > > > > >
>> > > > > --
>> > > > > Clebert Suconic
>> > > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Clebert Suconic
>> > >
>> > >
>>
>>
>>
>> --
>> Clebert Suconic
>
> --
> Clebert Suconic
--
Clebert Suconic
Re: Optimizing for many short connections
Posted by Clebert Suconic <cl...@gmail.com>.
The release candidate :
https://dist.apache.org/repos/dist/dev/activemq/activemq-artemis/2.23.0
I would appreciate your feedback here or even better on the dev list at the
voting thread.
On Tue, Jun 7, 2022 at 12:52 PM Clebert Suconic <cl...@gmail.com>
wrote:
> right, but since I'm just releasing it would be great to hear his
> feedback on top of the release candidate itself.
>
> On Tue, Jun 7, 2022 at 12:38 PM Justin Bertram <jb...@apache.org>
> wrote:
> >
> > A snapshot [1] is always a possibility as well.
> >
> >
> > Justin
> >
> > [1]
> >
> https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-artemis/2.23.0-SNAPSHOT/
> >
> > On Tue, Jun 7, 2022 at 11:34 AM Clebert Suconic <
> clebert.suconic@gmail.com>
> > wrote:
> >
> > > I am preparing artemis 2.23.0 at the moment...
> > >
> > > The way a release works in apache, you send it for vote and it will be
> > > officially released after 72 hours of a passing vote. but you should
> > > have a candidate release today that you can try it out. Based on the
> > > current diffs it should be fairly safe for you to upgrade.
> > >
> > > 2.14.0 would require JDK 1.8 minimal. while artemis 2.23 will require
> > > JDK 11 minimal.
> > >
> > >
> > > I will send you a link to the Candidate release as soon as it's ready
> > > on this thread.
> > >
> > > On Mon, Jun 6, 2022 at 10:25 PM Ben Warrick <horseatingweeds@gmail.com
> >
> > > wrote:
> > > >
> > > > Hi Clebert,
> > > >
> > > > Yeah, I can use whatever I want after I prove it out in lower
> > > environments.
> > > >
> > > > Thanks,
> > > >
> > > > Ben
> > > >
> > > > On Mon, Jun 6, 2022 at 10:06 PM Clebert Suconic <
> > > clebert.suconic@gmail.com>
> > > > wrote:
> > > >
> > > > > Can you use jdk 11?
> > > > >
> > > > > On Mon, Jun 6, 2022 at 8:32 PM Clebert Suconic <
> > > clebert.suconic@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > You are likely hitting the bug I fixed at:
> > > > > >
> > > > > > https://github.com/apache/activemq-artemis/pull/4098
> > > > > >
> > > > > > I introduced a Thread Local Leakage. I thought by setting
> > > > > > threadLocal.set(null) I would be clearing it... and in fact I
> just
> > > > > > updated the threadLocal variable with null.
> > > > > >
> > > > > > That ThreadLocal is created for every new session.
> > > > > >
> > > > > >
> > > > > > I was going to start a release today but I couldn't start it
> today as
> > > > > > I'm a bit sick today. I should be able to start a release
> tomorrow
> > > and
> > > > > > then you could try it out.
> > > > > >
> > > > > >
> > > > > > your other option would be to cherry-pick the change and build
> the
> > > > > > version yourself so you can run it in production.
> > > > > >
> > > > > > On Fri, Jun 3, 2022 at 4:27 PM Francesco Nigro <
> nigro.fra@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > Some more info about it here
> > > https://netty.io/wiki/tcp-fast-open.html
> > > > > > >
> > > > > > > Given that Netty can makes use of it and Artemis can as well:
> if
> > > it's
> > > > > not
> > > > > > > present in the doc it means we need to expose it in the
> acceptor
> > > > > > > configuration (for epoll) and...we love contributions :)
> > > > > > > But I suggest first to give it a shot first in a test env to
> see
> > > if it
> > > > > > > solves your problem!
> > > > > > >
> > > > > > > Il ven 3 giu 2022, 22:21 Ben Warrick <
> horseatingweeds@gmail.com>
> > > ha
> > > > > > scritto:
> > > > > > >
> > > > > > > > Hi Francesco,
> > > > > > > >
> > > > > > > > I don't find anything in the Artemis docs about
> TCP_FAST_OPEN. It
> > > > > must
> > > > > > not
> > > > > > > > be an exposed configuration on Artemis? Where do I find more
> on
> > > this?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Ben
> > > > > > > >
> > > > > > > > On Fri, Jun 3, 2022 at 4:08 PM Francesco Nigro <
> > > nigro.fra@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Are coming, not commenting: typo introduced by the the
> phone,
> > > > > sorry!
> > > > > > > > >
> > > > > > > > > Il ven 3 giu 2022, 22:07 Francesco Nigro <
> nigro.fra@gmail.com>
> > > ha
> > > > > > > > scritto:
> > > > > > > > >
> > > > > > > > > > Hi Ben,
> > > > > > > > > >
> > > > > > > > > > You can use TCP_FAST_OPEN on Artemis acceptors in case
> > > > > connections
> > > > > > are
> > > > > > > > > > commenting from the came clients (I don't remember if is
> an
> > > > > exposes
> > > > > > > > Netty
> > > > > > > > > > configuration) or...submit a PR to let Artemis separate
> > > > > (multiple)
> > > > > > > > > threads
> > > > > > > > > > to accept connections but, as you said, this is just an
> > > > > > anti-pattern
> > > > > > > > and
> > > > > > > > > in
> > > > > > > > > > order to reduce the client code you can think about
> using a
> > > > > > transparent
> > > > > > > > > > connection pooling mechanism, instead.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Il ven 3 giu 2022, 21:55 Ben Warrick <
> > > horseatingweeds@gmail.com>
> > > > > > ha
> > > > > > > > > > scritto:
> > > > > > > > > >
> > > > > > > > > >> I'm running Artemis 2.14, and one of the systems
> publishing
> > > to
> > > > > it
> > > > > > > > > creates
> > > > > > > > > >> a
> > > > > > > > > >> lot of separate tasks -- from its perspective. I can
> see on
> > > > > > Artemis
> > > > > > > > that
> > > > > > > > > >> the Total connection count vs Total messages
> acknowledged is
> > > > > > pretty
> > > > > > > > > close,
> > > > > > > > > >> so it's apparent that the client system is making use
> of the
> > > > > > > > > anti-pattern
> > > > > > > > > >> of creating a new connection for each message it
> publishes.
> > > I
> > > > > > don't
> > > > > > > > have
> > > > > > > > > >> access to the client system implementation, but is there
> > > > > anything
> > > > > > I
> > > > > > > > can
> > > > > > > > > do
> > > > > > > > > >> on the Artemis end to make it more efficient? Latency
> with
> > > this
> > > > > > system
> > > > > > > > > is
> > > > > > > > > >> becoming an issue. Also every 30 minutes or so there is
> a
> > > spike
> > > > > > in CPU
> > > > > > > > > >> usage (~50%) which I assume is garbage collection.
> During
> > > that
> > > > > > spike
> > > > > > > > > >> message publishing slows down a lot.
> > > > > > > > > >>
> > > > > > > > > >> Thanks,
> > > > > > > > > >>
> > > > > > > > > >> Ben
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Clebert Suconic
> > > > > >
> > > > > --
> > > > > Clebert Suconic
> > > > >
> > >
> > >
> > >
> > > --
> > > Clebert Suconic
> > >
> > >
>
>
>
> --
> Clebert Suconic
>
--
Clebert Suconic
Re: Optimizing for many short connections
Posted by Clebert Suconic <cl...@gmail.com>.
right, but since I'm just releasing it would be great to hear his
feedback on top of the release candidate itself.
On Tue, Jun 7, 2022 at 12:38 PM Justin Bertram <jb...@apache.org> wrote:
>
> A snapshot [1] is always a possibility as well.
>
>
> Justin
>
> [1]
> https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-artemis/2.23.0-SNAPSHOT/
>
> On Tue, Jun 7, 2022 at 11:34 AM Clebert Suconic <cl...@gmail.com>
> wrote:
>
> > I am preparing artemis 2.23.0 at the moment...
> >
> > The way a release works in apache, you send it for vote and it will be
> > officially released after 72 hours of a passing vote. but you should
> > have a candidate release today that you can try it out. Based on the
> > current diffs it should be fairly safe for you to upgrade.
> >
> > 2.14.0 would require JDK 1.8 minimal. while artemis 2.23 will require
> > JDK 11 minimal.
> >
> >
> > I will send you a link to the Candidate release as soon as it's ready
> > on this thread.
> >
> > On Mon, Jun 6, 2022 at 10:25 PM Ben Warrick <ho...@gmail.com>
> > wrote:
> > >
> > > Hi Clebert,
> > >
> > > Yeah, I can use whatever I want after I prove it out in lower
> > environments.
> > >
> > > Thanks,
> > >
> > > Ben
> > >
> > > On Mon, Jun 6, 2022 at 10:06 PM Clebert Suconic <
> > clebert.suconic@gmail.com>
> > > wrote:
> > >
> > > > Can you use jdk 11?
> > > >
> > > > On Mon, Jun 6, 2022 at 8:32 PM Clebert Suconic <
> > clebert.suconic@gmail.com>
> > > > wrote:
> > > >
> > > > > You are likely hitting the bug I fixed at:
> > > > >
> > > > > https://github.com/apache/activemq-artemis/pull/4098
> > > > >
> > > > > I introduced a Thread Local Leakage. I thought by setting
> > > > > threadLocal.set(null) I would be clearing it... and in fact I just
> > > > > updated the threadLocal variable with null.
> > > > >
> > > > > That ThreadLocal is created for every new session.
> > > > >
> > > > >
> > > > > I was going to start a release today but I couldn't start it today as
> > > > > I'm a bit sick today. I should be able to start a release tomorrow
> > and
> > > > > then you could try it out.
> > > > >
> > > > >
> > > > > your other option would be to cherry-pick the change and build the
> > > > > version yourself so you can run it in production.
> > > > >
> > > > > On Fri, Jun 3, 2022 at 4:27 PM Francesco Nigro <ni...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > Some more info about it here
> > https://netty.io/wiki/tcp-fast-open.html
> > > > > >
> > > > > > Given that Netty can makes use of it and Artemis can as well: if
> > it's
> > > > not
> > > > > > present in the doc it means we need to expose it in the acceptor
> > > > > > configuration (for epoll) and...we love contributions :)
> > > > > > But I suggest first to give it a shot first in a test env to see
> > if it
> > > > > > solves your problem!
> > > > > >
> > > > > > Il ven 3 giu 2022, 22:21 Ben Warrick <ho...@gmail.com>
> > ha
> > > > > scritto:
> > > > > >
> > > > > > > Hi Francesco,
> > > > > > >
> > > > > > > I don't find anything in the Artemis docs about TCP_FAST_OPEN. It
> > > > must
> > > > > not
> > > > > > > be an exposed configuration on Artemis? Where do I find more on
> > this?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Ben
> > > > > > >
> > > > > > > On Fri, Jun 3, 2022 at 4:08 PM Francesco Nigro <
> > nigro.fra@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Are coming, not commenting: typo introduced by the the phone,
> > > > sorry!
> > > > > > > >
> > > > > > > > Il ven 3 giu 2022, 22:07 Francesco Nigro <ni...@gmail.com>
> > ha
> > > > > > > scritto:
> > > > > > > >
> > > > > > > > > Hi Ben,
> > > > > > > > >
> > > > > > > > > You can use TCP_FAST_OPEN on Artemis acceptors in case
> > > > connections
> > > > > are
> > > > > > > > > commenting from the came clients (I don't remember if is an
> > > > exposes
> > > > > > > Netty
> > > > > > > > > configuration) or...submit a PR to let Artemis separate
> > > > (multiple)
> > > > > > > > threads
> > > > > > > > > to accept connections but, as you said, this is just an
> > > > > anti-pattern
> > > > > > > and
> > > > > > > > in
> > > > > > > > > order to reduce the client code you can think about using a
> > > > > transparent
> > > > > > > > > connection pooling mechanism, instead.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Il ven 3 giu 2022, 21:55 Ben Warrick <
> > horseatingweeds@gmail.com>
> > > > > ha
> > > > > > > > > scritto:
> > > > > > > > >
> > > > > > > > >> I'm running Artemis 2.14, and one of the systems publishing
> > to
> > > > it
> > > > > > > > creates
> > > > > > > > >> a
> > > > > > > > >> lot of separate tasks -- from its perspective. I can see on
> > > > > Artemis
> > > > > > > that
> > > > > > > > >> the Total connection count vs Total messages acknowledged is
> > > > > pretty
> > > > > > > > close,
> > > > > > > > >> so it's apparent that the client system is making use of the
> > > > > > > > anti-pattern
> > > > > > > > >> of creating a new connection for each message it publishes.
> > I
> > > > > don't
> > > > > > > have
> > > > > > > > >> access to the client system implementation, but is there
> > > > anything
> > > > > I
> > > > > > > can
> > > > > > > > do
> > > > > > > > >> on the Artemis end to make it more efficient? Latency with
> > this
> > > > > system
> > > > > > > > is
> > > > > > > > >> becoming an issue. Also every 30 minutes or so there is a
> > spike
> > > > > in CPU
> > > > > > > > >> usage (~50%) which I assume is garbage collection. During
> > that
> > > > > spike
> > > > > > > > >> message publishing slows down a lot.
> > > > > > > > >>
> > > > > > > > >> Thanks,
> > > > > > > > >>
> > > > > > > > >> Ben
> > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Clebert Suconic
> > > > >
> > > > --
> > > > Clebert Suconic
> > > >
> >
> >
> >
> > --
> > Clebert Suconic
> >
> >
--
Clebert Suconic
Re: Optimizing for many short connections
Posted by Justin Bertram <jb...@apache.org>.
A snapshot [1] is always a possibility as well.
Justin
[1]
https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-artemis/2.23.0-SNAPSHOT/
On Tue, Jun 7, 2022 at 11:34 AM Clebert Suconic <cl...@gmail.com>
wrote:
> I am preparing artemis 2.23.0 at the moment...
>
> The way a release works in apache, you send it for vote and it will be
> officially released after 72 hours of a passing vote. but you should
> have a candidate release today that you can try it out. Based on the
> current diffs it should be fairly safe for you to upgrade.
>
> 2.14.0 would require JDK 1.8 minimal. while artemis 2.23 will require
> JDK 11 minimal.
>
>
> I will send you a link to the Candidate release as soon as it's ready
> on this thread.
>
> On Mon, Jun 6, 2022 at 10:25 PM Ben Warrick <ho...@gmail.com>
> wrote:
> >
> > Hi Clebert,
> >
> > Yeah, I can use whatever I want after I prove it out in lower
> environments.
> >
> > Thanks,
> >
> > Ben
> >
> > On Mon, Jun 6, 2022 at 10:06 PM Clebert Suconic <
> clebert.suconic@gmail.com>
> > wrote:
> >
> > > Can you use jdk 11?
> > >
> > > On Mon, Jun 6, 2022 at 8:32 PM Clebert Suconic <
> clebert.suconic@gmail.com>
> > > wrote:
> > >
> > > > You are likely hitting the bug I fixed at:
> > > >
> > > > https://github.com/apache/activemq-artemis/pull/4098
> > > >
> > > > I introduced a Thread Local Leakage. I thought by setting
> > > > threadLocal.set(null) I would be clearing it... and in fact I just
> > > > updated the threadLocal variable with null.
> > > >
> > > > That ThreadLocal is created for every new session.
> > > >
> > > >
> > > > I was going to start a release today but I couldn't start it today as
> > > > I'm a bit sick today. I should be able to start a release tomorrow
> and
> > > > then you could try it out.
> > > >
> > > >
> > > > your other option would be to cherry-pick the change and build the
> > > > version yourself so you can run it in production.
> > > >
> > > > On Fri, Jun 3, 2022 at 4:27 PM Francesco Nigro <ni...@gmail.com>
> > > > wrote:
> > > > >
> > > > > Some more info about it here
> https://netty.io/wiki/tcp-fast-open.html
> > > > >
> > > > > Given that Netty can makes use of it and Artemis can as well: if
> it's
> > > not
> > > > > present in the doc it means we need to expose it in the acceptor
> > > > > configuration (for epoll) and...we love contributions :)
> > > > > But I suggest first to give it a shot first in a test env to see
> if it
> > > > > solves your problem!
> > > > >
> > > > > Il ven 3 giu 2022, 22:21 Ben Warrick <ho...@gmail.com>
> ha
> > > > scritto:
> > > > >
> > > > > > Hi Francesco,
> > > > > >
> > > > > > I don't find anything in the Artemis docs about TCP_FAST_OPEN. It
> > > must
> > > > not
> > > > > > be an exposed configuration on Artemis? Where do I find more on
> this?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Ben
> > > > > >
> > > > > > On Fri, Jun 3, 2022 at 4:08 PM Francesco Nigro <
> nigro.fra@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Are coming, not commenting: typo introduced by the the phone,
> > > sorry!
> > > > > > >
> > > > > > > Il ven 3 giu 2022, 22:07 Francesco Nigro <ni...@gmail.com>
> ha
> > > > > > scritto:
> > > > > > >
> > > > > > > > Hi Ben,
> > > > > > > >
> > > > > > > > You can use TCP_FAST_OPEN on Artemis acceptors in case
> > > connections
> > > > are
> > > > > > > > commenting from the came clients (I don't remember if is an
> > > exposes
> > > > > > Netty
> > > > > > > > configuration) or...submit a PR to let Artemis separate
> > > (multiple)
> > > > > > > threads
> > > > > > > > to accept connections but, as you said, this is just an
> > > > anti-pattern
> > > > > > and
> > > > > > > in
> > > > > > > > order to reduce the client code you can think about using a
> > > > transparent
> > > > > > > > connection pooling mechanism, instead.
> > > > > > > >
> > > > > > > >
> > > > > > > > Il ven 3 giu 2022, 21:55 Ben Warrick <
> horseatingweeds@gmail.com>
> > > > ha
> > > > > > > > scritto:
> > > > > > > >
> > > > > > > >> I'm running Artemis 2.14, and one of the systems publishing
> to
> > > it
> > > > > > > creates
> > > > > > > >> a
> > > > > > > >> lot of separate tasks -- from its perspective. I can see on
> > > > Artemis
> > > > > > that
> > > > > > > >> the Total connection count vs Total messages acknowledged is
> > > > pretty
> > > > > > > close,
> > > > > > > >> so it's apparent that the client system is making use of the
> > > > > > > anti-pattern
> > > > > > > >> of creating a new connection for each message it publishes.
> I
> > > > don't
> > > > > > have
> > > > > > > >> access to the client system implementation, but is there
> > > anything
> > > > I
> > > > > > can
> > > > > > > do
> > > > > > > >> on the Artemis end to make it more efficient? Latency with
> this
> > > > system
> > > > > > > is
> > > > > > > >> becoming an issue. Also every 30 minutes or so there is a
> spike
> > > > in CPU
> > > > > > > >> usage (~50%) which I assume is garbage collection. During
> that
> > > > spike
> > > > > > > >> message publishing slows down a lot.
> > > > > > > >>
> > > > > > > >> Thanks,
> > > > > > > >>
> > > > > > > >> Ben
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Clebert Suconic
> > > >
> > > --
> > > Clebert Suconic
> > >
>
>
>
> --
> Clebert Suconic
>
>
Re: Optimizing for many short connections
Posted by Clebert Suconic <cl...@gmail.com>.
I am preparing artemis 2.23.0 at the moment...
The way a release works in apache, you send it for vote and it will be
officially released after 72 hours of a passing vote. but you should
have a candidate release today that you can try it out. Based on the
current diffs it should be fairly safe for you to upgrade.
2.14.0 would require JDK 1.8 minimal. while artemis 2.23 will require
JDK 11 minimal.
I will send you a link to the Candidate release as soon as it's ready
on this thread.
On Mon, Jun 6, 2022 at 10:25 PM Ben Warrick <ho...@gmail.com> wrote:
>
> Hi Clebert,
>
> Yeah, I can use whatever I want after I prove it out in lower environments.
>
> Thanks,
>
> Ben
>
> On Mon, Jun 6, 2022 at 10:06 PM Clebert Suconic <cl...@gmail.com>
> wrote:
>
> > Can you use jdk 11?
> >
> > On Mon, Jun 6, 2022 at 8:32 PM Clebert Suconic <cl...@gmail.com>
> > wrote:
> >
> > > You are likely hitting the bug I fixed at:
> > >
> > > https://github.com/apache/activemq-artemis/pull/4098
> > >
> > > I introduced a Thread Local Leakage. I thought by setting
> > > threadLocal.set(null) I would be clearing it... and in fact I just
> > > updated the threadLocal variable with null.
> > >
> > > That ThreadLocal is created for every new session.
> > >
> > >
> > > I was going to start a release today but I couldn't start it today as
> > > I'm a bit sick today. I should be able to start a release tomorrow and
> > > then you could try it out.
> > >
> > >
> > > your other option would be to cherry-pick the change and build the
> > > version yourself so you can run it in production.
> > >
> > > On Fri, Jun 3, 2022 at 4:27 PM Francesco Nigro <ni...@gmail.com>
> > > wrote:
> > > >
> > > > Some more info about it here https://netty.io/wiki/tcp-fast-open.html
> > > >
> > > > Given that Netty can makes use of it and Artemis can as well: if it's
> > not
> > > > present in the doc it means we need to expose it in the acceptor
> > > > configuration (for epoll) and...we love contributions :)
> > > > But I suggest first to give it a shot first in a test env to see if it
> > > > solves your problem!
> > > >
> > > > Il ven 3 giu 2022, 22:21 Ben Warrick <ho...@gmail.com> ha
> > > scritto:
> > > >
> > > > > Hi Francesco,
> > > > >
> > > > > I don't find anything in the Artemis docs about TCP_FAST_OPEN. It
> > must
> > > not
> > > > > be an exposed configuration on Artemis? Where do I find more on this?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Ben
> > > > >
> > > > > On Fri, Jun 3, 2022 at 4:08 PM Francesco Nigro <ni...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Are coming, not commenting: typo introduced by the the phone,
> > sorry!
> > > > > >
> > > > > > Il ven 3 giu 2022, 22:07 Francesco Nigro <ni...@gmail.com> ha
> > > > > scritto:
> > > > > >
> > > > > > > Hi Ben,
> > > > > > >
> > > > > > > You can use TCP_FAST_OPEN on Artemis acceptors in case
> > connections
> > > are
> > > > > > > commenting from the came clients (I don't remember if is an
> > exposes
> > > > > Netty
> > > > > > > configuration) or...submit a PR to let Artemis separate
> > (multiple)
> > > > > > threads
> > > > > > > to accept connections but, as you said, this is just an
> > > anti-pattern
> > > > > and
> > > > > > in
> > > > > > > order to reduce the client code you can think about using a
> > > transparent
> > > > > > > connection pooling mechanism, instead.
> > > > > > >
> > > > > > >
> > > > > > > Il ven 3 giu 2022, 21:55 Ben Warrick <ho...@gmail.com>
> > > ha
> > > > > > > scritto:
> > > > > > >
> > > > > > >> I'm running Artemis 2.14, and one of the systems publishing to
> > it
> > > > > > creates
> > > > > > >> a
> > > > > > >> lot of separate tasks -- from its perspective. I can see on
> > > Artemis
> > > > > that
> > > > > > >> the Total connection count vs Total messages acknowledged is
> > > pretty
> > > > > > close,
> > > > > > >> so it's apparent that the client system is making use of the
> > > > > > anti-pattern
> > > > > > >> of creating a new connection for each message it publishes. I
> > > don't
> > > > > have
> > > > > > >> access to the client system implementation, but is there
> > anything
> > > I
> > > > > can
> > > > > > do
> > > > > > >> on the Artemis end to make it more efficient? Latency with this
> > > system
> > > > > > is
> > > > > > >> becoming an issue. Also every 30 minutes or so there is a spike
> > > in CPU
> > > > > > >> usage (~50%) which I assume is garbage collection. During that
> > > spike
> > > > > > >> message publishing slows down a lot.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >>
> > > > > > >> Ben
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
> > >
> > >
> > > --
> > > Clebert Suconic
> > >
> > --
> > Clebert Suconic
> >
--
Clebert Suconic
Re: Optimizing for many short connections
Posted by Ben Warrick <ho...@gmail.com>.
Hi Clebert,
Yeah, I can use whatever I want after I prove it out in lower environments.
Thanks,
Ben
On Mon, Jun 6, 2022 at 10:06 PM Clebert Suconic <cl...@gmail.com>
wrote:
> Can you use jdk 11?
>
> On Mon, Jun 6, 2022 at 8:32 PM Clebert Suconic <cl...@gmail.com>
> wrote:
>
> > You are likely hitting the bug I fixed at:
> >
> > https://github.com/apache/activemq-artemis/pull/4098
> >
> > I introduced a Thread Local Leakage. I thought by setting
> > threadLocal.set(null) I would be clearing it... and in fact I just
> > updated the threadLocal variable with null.
> >
> > That ThreadLocal is created for every new session.
> >
> >
> > I was going to start a release today but I couldn't start it today as
> > I'm a bit sick today. I should be able to start a release tomorrow and
> > then you could try it out.
> >
> >
> > your other option would be to cherry-pick the change and build the
> > version yourself so you can run it in production.
> >
> > On Fri, Jun 3, 2022 at 4:27 PM Francesco Nigro <ni...@gmail.com>
> > wrote:
> > >
> > > Some more info about it here https://netty.io/wiki/tcp-fast-open.html
> > >
> > > Given that Netty can makes use of it and Artemis can as well: if it's
> not
> > > present in the doc it means we need to expose it in the acceptor
> > > configuration (for epoll) and...we love contributions :)
> > > But I suggest first to give it a shot first in a test env to see if it
> > > solves your problem!
> > >
> > > Il ven 3 giu 2022, 22:21 Ben Warrick <ho...@gmail.com> ha
> > scritto:
> > >
> > > > Hi Francesco,
> > > >
> > > > I don't find anything in the Artemis docs about TCP_FAST_OPEN. It
> must
> > not
> > > > be an exposed configuration on Artemis? Where do I find more on this?
> > > >
> > > > Thanks,
> > > >
> > > > Ben
> > > >
> > > > On Fri, Jun 3, 2022 at 4:08 PM Francesco Nigro <ni...@gmail.com>
> > > > wrote:
> > > >
> > > > > Are coming, not commenting: typo introduced by the the phone,
> sorry!
> > > > >
> > > > > Il ven 3 giu 2022, 22:07 Francesco Nigro <ni...@gmail.com> ha
> > > > scritto:
> > > > >
> > > > > > Hi Ben,
> > > > > >
> > > > > > You can use TCP_FAST_OPEN on Artemis acceptors in case
> connections
> > are
> > > > > > commenting from the came clients (I don't remember if is an
> exposes
> > > > Netty
> > > > > > configuration) or...submit a PR to let Artemis separate
> (multiple)
> > > > > threads
> > > > > > to accept connections but, as you said, this is just an
> > anti-pattern
> > > > and
> > > > > in
> > > > > > order to reduce the client code you can think about using a
> > transparent
> > > > > > connection pooling mechanism, instead.
> > > > > >
> > > > > >
> > > > > > Il ven 3 giu 2022, 21:55 Ben Warrick <ho...@gmail.com>
> > ha
> > > > > > scritto:
> > > > > >
> > > > > >> I'm running Artemis 2.14, and one of the systems publishing to
> it
> > > > > creates
> > > > > >> a
> > > > > >> lot of separate tasks -- from its perspective. I can see on
> > Artemis
> > > > that
> > > > > >> the Total connection count vs Total messages acknowledged is
> > pretty
> > > > > close,
> > > > > >> so it's apparent that the client system is making use of the
> > > > > anti-pattern
> > > > > >> of creating a new connection for each message it publishes. I
> > don't
> > > > have
> > > > > >> access to the client system implementation, but is there
> anything
> > I
> > > > can
> > > > > do
> > > > > >> on the Artemis end to make it more efficient? Latency with this
> > system
> > > > > is
> > > > > >> becoming an issue. Also every 30 minutes or so there is a spike
> > in CPU
> > > > > >> usage (~50%) which I assume is garbage collection. During that
> > spike
> > > > > >> message publishing slows down a lot.
> > > > > >>
> > > > > >> Thanks,
> > > > > >>
> > > > > >> Ben
> > > > > >>
> > > > > >
> > > > >
> > > >
> >
> >
> >
> > --
> > Clebert Suconic
> >
> --
> Clebert Suconic
>
Re: Optimizing for many short connections
Posted by Clebert Suconic <cl...@gmail.com>.
Can you use jdk 11?
On Mon, Jun 6, 2022 at 8:32 PM Clebert Suconic <cl...@gmail.com>
wrote:
> You are likely hitting the bug I fixed at:
>
> https://github.com/apache/activemq-artemis/pull/4098
>
> I introduced a Thread Local Leakage. I thought by setting
> threadLocal.set(null) I would be clearing it... and in fact I just
> updated the threadLocal variable with null.
>
> That ThreadLocal is created for every new session.
>
>
> I was going to start a release today but I couldn't start it today as
> I'm a bit sick today. I should be able to start a release tomorrow and
> then you could try it out.
>
>
> your other option would be to cherry-pick the change and build the
> version yourself so you can run it in production.
>
> On Fri, Jun 3, 2022 at 4:27 PM Francesco Nigro <ni...@gmail.com>
> wrote:
> >
> > Some more info about it here https://netty.io/wiki/tcp-fast-open.html
> >
> > Given that Netty can makes use of it and Artemis can as well: if it's not
> > present in the doc it means we need to expose it in the acceptor
> > configuration (for epoll) and...we love contributions :)
> > But I suggest first to give it a shot first in a test env to see if it
> > solves your problem!
> >
> > Il ven 3 giu 2022, 22:21 Ben Warrick <ho...@gmail.com> ha
> scritto:
> >
> > > Hi Francesco,
> > >
> > > I don't find anything in the Artemis docs about TCP_FAST_OPEN. It must
> not
> > > be an exposed configuration on Artemis? Where do I find more on this?
> > >
> > > Thanks,
> > >
> > > Ben
> > >
> > > On Fri, Jun 3, 2022 at 4:08 PM Francesco Nigro <ni...@gmail.com>
> > > wrote:
> > >
> > > > Are coming, not commenting: typo introduced by the the phone, sorry!
> > > >
> > > > Il ven 3 giu 2022, 22:07 Francesco Nigro <ni...@gmail.com> ha
> > > scritto:
> > > >
> > > > > Hi Ben,
> > > > >
> > > > > You can use TCP_FAST_OPEN on Artemis acceptors in case connections
> are
> > > > > commenting from the came clients (I don't remember if is an exposes
> > > Netty
> > > > > configuration) or...submit a PR to let Artemis separate (multiple)
> > > > threads
> > > > > to accept connections but, as you said, this is just an
> anti-pattern
> > > and
> > > > in
> > > > > order to reduce the client code you can think about using a
> transparent
> > > > > connection pooling mechanism, instead.
> > > > >
> > > > >
> > > > > Il ven 3 giu 2022, 21:55 Ben Warrick <ho...@gmail.com>
> ha
> > > > > scritto:
> > > > >
> > > > >> I'm running Artemis 2.14, and one of the systems publishing to it
> > > > creates
> > > > >> a
> > > > >> lot of separate tasks -- from its perspective. I can see on
> Artemis
> > > that
> > > > >> the Total connection count vs Total messages acknowledged is
> pretty
> > > > close,
> > > > >> so it's apparent that the client system is making use of the
> > > > anti-pattern
> > > > >> of creating a new connection for each message it publishes. I
> don't
> > > have
> > > > >> access to the client system implementation, but is there anything
> I
> > > can
> > > > do
> > > > >> on the Artemis end to make it more efficient? Latency with this
> system
> > > > is
> > > > >> becoming an issue. Also every 30 minutes or so there is a spike
> in CPU
> > > > >> usage (~50%) which I assume is garbage collection. During that
> spike
> > > > >> message publishing slows down a lot.
> > > > >>
> > > > >> Thanks,
> > > > >>
> > > > >> Ben
> > > > >>
> > > > >
> > > >
> > >
>
>
>
> --
> Clebert Suconic
>
--
Clebert Suconic
Re: Optimizing for many short connections
Posted by Clebert Suconic <cl...@gmail.com>.
You are likely hitting the bug I fixed at:
https://github.com/apache/activemq-artemis/pull/4098
I introduced a Thread Local Leakage. I thought by setting
threadLocal.set(null) I would be clearing it... and in fact I just
updated the threadLocal variable with null.
That ThreadLocal is created for every new session.
I was going to start a release today but I couldn't start it today as
I'm a bit sick today. I should be able to start a release tomorrow and
then you could try it out.
your other option would be to cherry-pick the change and build the
version yourself so you can run it in production.
On Fri, Jun 3, 2022 at 4:27 PM Francesco Nigro <ni...@gmail.com> wrote:
>
> Some more info about it here https://netty.io/wiki/tcp-fast-open.html
>
> Given that Netty can makes use of it and Artemis can as well: if it's not
> present in the doc it means we need to expose it in the acceptor
> configuration (for epoll) and...we love contributions :)
> But I suggest first to give it a shot first in a test env to see if it
> solves your problem!
>
> Il ven 3 giu 2022, 22:21 Ben Warrick <ho...@gmail.com> ha scritto:
>
> > Hi Francesco,
> >
> > I don't find anything in the Artemis docs about TCP_FAST_OPEN. It must not
> > be an exposed configuration on Artemis? Where do I find more on this?
> >
> > Thanks,
> >
> > Ben
> >
> > On Fri, Jun 3, 2022 at 4:08 PM Francesco Nigro <ni...@gmail.com>
> > wrote:
> >
> > > Are coming, not commenting: typo introduced by the the phone, sorry!
> > >
> > > Il ven 3 giu 2022, 22:07 Francesco Nigro <ni...@gmail.com> ha
> > scritto:
> > >
> > > > Hi Ben,
> > > >
> > > > You can use TCP_FAST_OPEN on Artemis acceptors in case connections are
> > > > commenting from the came clients (I don't remember if is an exposes
> > Netty
> > > > configuration) or...submit a PR to let Artemis separate (multiple)
> > > threads
> > > > to accept connections but, as you said, this is just an anti-pattern
> > and
> > > in
> > > > order to reduce the client code you can think about using a transparent
> > > > connection pooling mechanism, instead.
> > > >
> > > >
> > > > Il ven 3 giu 2022, 21:55 Ben Warrick <ho...@gmail.com> ha
> > > > scritto:
> > > >
> > > >> I'm running Artemis 2.14, and one of the systems publishing to it
> > > creates
> > > >> a
> > > >> lot of separate tasks -- from its perspective. I can see on Artemis
> > that
> > > >> the Total connection count vs Total messages acknowledged is pretty
> > > close,
> > > >> so it's apparent that the client system is making use of the
> > > anti-pattern
> > > >> of creating a new connection for each message it publishes. I don't
> > have
> > > >> access to the client system implementation, but is there anything I
> > can
> > > do
> > > >> on the Artemis end to make it more efficient? Latency with this system
> > > is
> > > >> becoming an issue. Also every 30 minutes or so there is a spike in CPU
> > > >> usage (~50%) which I assume is garbage collection. During that spike
> > > >> message publishing slows down a lot.
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Ben
> > > >>
> > > >
> > >
> >
--
Clebert Suconic
Re: Optimizing for many short connections
Posted by Francesco Nigro <ni...@gmail.com>.
Some more info about it here https://netty.io/wiki/tcp-fast-open.html
Given that Netty can makes use of it and Artemis can as well: if it's not
present in the doc it means we need to expose it in the acceptor
configuration (for epoll) and...we love contributions :)
But I suggest first to give it a shot first in a test env to see if it
solves your problem!
Il ven 3 giu 2022, 22:21 Ben Warrick <ho...@gmail.com> ha scritto:
> Hi Francesco,
>
> I don't find anything in the Artemis docs about TCP_FAST_OPEN. It must not
> be an exposed configuration on Artemis? Where do I find more on this?
>
> Thanks,
>
> Ben
>
> On Fri, Jun 3, 2022 at 4:08 PM Francesco Nigro <ni...@gmail.com>
> wrote:
>
> > Are coming, not commenting: typo introduced by the the phone, sorry!
> >
> > Il ven 3 giu 2022, 22:07 Francesco Nigro <ni...@gmail.com> ha
> scritto:
> >
> > > Hi Ben,
> > >
> > > You can use TCP_FAST_OPEN on Artemis acceptors in case connections are
> > > commenting from the came clients (I don't remember if is an exposes
> Netty
> > > configuration) or...submit a PR to let Artemis separate (multiple)
> > threads
> > > to accept connections but, as you said, this is just an anti-pattern
> and
> > in
> > > order to reduce the client code you can think about using a transparent
> > > connection pooling mechanism, instead.
> > >
> > >
> > > Il ven 3 giu 2022, 21:55 Ben Warrick <ho...@gmail.com> ha
> > > scritto:
> > >
> > >> I'm running Artemis 2.14, and one of the systems publishing to it
> > creates
> > >> a
> > >> lot of separate tasks -- from its perspective. I can see on Artemis
> that
> > >> the Total connection count vs Total messages acknowledged is pretty
> > close,
> > >> so it's apparent that the client system is making use of the
> > anti-pattern
> > >> of creating a new connection for each message it publishes. I don't
> have
> > >> access to the client system implementation, but is there anything I
> can
> > do
> > >> on the Artemis end to make it more efficient? Latency with this system
> > is
> > >> becoming an issue. Also every 30 minutes or so there is a spike in CPU
> > >> usage (~50%) which I assume is garbage collection. During that spike
> > >> message publishing slows down a lot.
> > >>
> > >> Thanks,
> > >>
> > >> Ben
> > >>
> > >
> >
>
Re: Optimizing for many short connections
Posted by Ben Warrick <ho...@gmail.com>.
Hi Francesco,
I don't find anything in the Artemis docs about TCP_FAST_OPEN. It must not
be an exposed configuration on Artemis? Where do I find more on this?
Thanks,
Ben
On Fri, Jun 3, 2022 at 4:08 PM Francesco Nigro <ni...@gmail.com> wrote:
> Are coming, not commenting: typo introduced by the the phone, sorry!
>
> Il ven 3 giu 2022, 22:07 Francesco Nigro <ni...@gmail.com> ha scritto:
>
> > Hi Ben,
> >
> > You can use TCP_FAST_OPEN on Artemis acceptors in case connections are
> > commenting from the came clients (I don't remember if is an exposes Netty
> > configuration) or...submit a PR to let Artemis separate (multiple)
> threads
> > to accept connections but, as you said, this is just an anti-pattern and
> in
> > order to reduce the client code you can think about using a transparent
> > connection pooling mechanism, instead.
> >
> >
> > Il ven 3 giu 2022, 21:55 Ben Warrick <ho...@gmail.com> ha
> > scritto:
> >
> >> I'm running Artemis 2.14, and one of the systems publishing to it
> creates
> >> a
> >> lot of separate tasks -- from its perspective. I can see on Artemis that
> >> the Total connection count vs Total messages acknowledged is pretty
> close,
> >> so it's apparent that the client system is making use of the
> anti-pattern
> >> of creating a new connection for each message it publishes. I don't have
> >> access to the client system implementation, but is there anything I can
> do
> >> on the Artemis end to make it more efficient? Latency with this system
> is
> >> becoming an issue. Also every 30 minutes or so there is a spike in CPU
> >> usage (~50%) which I assume is garbage collection. During that spike
> >> message publishing slows down a lot.
> >>
> >> Thanks,
> >>
> >> Ben
> >>
> >
>
Re: Optimizing for many short connections
Posted by Francesco Nigro <ni...@gmail.com>.
Are coming, not commenting: typo introduced by the the phone, sorry!
Il ven 3 giu 2022, 22:07 Francesco Nigro <ni...@gmail.com> ha scritto:
> Hi Ben,
>
> You can use TCP_FAST_OPEN on Artemis acceptors in case connections are
> commenting from the came clients (I don't remember if is an exposes Netty
> configuration) or...submit a PR to let Artemis separate (multiple) threads
> to accept connections but, as you said, this is just an anti-pattern and in
> order to reduce the client code you can think about using a transparent
> connection pooling mechanism, instead.
>
>
> Il ven 3 giu 2022, 21:55 Ben Warrick <ho...@gmail.com> ha
> scritto:
>
>> I'm running Artemis 2.14, and one of the systems publishing to it creates
>> a
>> lot of separate tasks -- from its perspective. I can see on Artemis that
>> the Total connection count vs Total messages acknowledged is pretty close,
>> so it's apparent that the client system is making use of the anti-pattern
>> of creating a new connection for each message it publishes. I don't have
>> access to the client system implementation, but is there anything I can do
>> on the Artemis end to make it more efficient? Latency with this system is
>> becoming an issue. Also every 30 minutes or so there is a spike in CPU
>> usage (~50%) which I assume is garbage collection. During that spike
>> message publishing slows down a lot.
>>
>> Thanks,
>>
>> Ben
>>
>
Re: Optimizing for many short connections
Posted by Francesco Nigro <ni...@gmail.com>.
Hi Ben,
You can use TCP_FAST_OPEN on Artemis acceptors in case connections are
commenting from the came clients (I don't remember if is an exposes Netty
configuration) or...submit a PR to let Artemis separate (multiple) threads
to accept connections but, as you said, this is just an anti-pattern and in
order to reduce the client code you can think about using a transparent
connection pooling mechanism, instead.
Il ven 3 giu 2022, 21:55 Ben Warrick <ho...@gmail.com> ha scritto:
> I'm running Artemis 2.14, and one of the systems publishing to it creates a
> lot of separate tasks -- from its perspective. I can see on Artemis that
> the Total connection count vs Total messages acknowledged is pretty close,
> so it's apparent that the client system is making use of the anti-pattern
> of creating a new connection for each message it publishes. I don't have
> access to the client system implementation, but is there anything I can do
> on the Artemis end to make it more efficient? Latency with this system is
> becoming an issue. Also every 30 minutes or so there is a spike in CPU
> usage (~50%) which I assume is garbage collection. During that spike
> message publishing slows down a lot.
>
> Thanks,
>
> Ben
>