You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Thorsten Meinl <th...@knime.com> on 2021/03/05 16:08:50 UTC
Message filters vs. topic hierachies
Hi,
We are currently building an application using ActiveMQ Artemis. The rough
setup is as follows: We have consumers that are interested in all messages and
other consumers that are only interested in a subset. So far we have been
using topic hierarchies. The first group of consumers listens on e.g. "jobs.*"
whereas the others listens on "jobs.A", "jobs.B", etc. All addresses, topics,
queues are auto-created. What we notice now is that even if all consumers are
gone there are still addresses "jobs.A", "jobs.B", ... remaining even though
they don't have anything attached to them any more. No consumers, no queues,
nothing. Therefore the first question: why is this and how can we prevent it?
An alternative approach would be to use just one topic "jobs" and filter
messages with message selectors (e.g. "id=A", "id=B", ...). This would
eliminate the issue above. The question is whether there is a performance or
functionality penalty involved with this approach.
Thanks,
Thorsten
--
Dr.-Ing. Thorsten Meinl
KNIME AG
Hardturmstrasse 66
8005 Zurich, Switzerland
Re: Message filters vs. topic hierachies
Posted by Justin Bertram <jb...@apache.org>.
This looks like a bug. Go ahead and create a Jira and attach your
reproducer. Thanks!
Justin
On Mon, Mar 8, 2021 at 4:30 AM Thorsten Meinl <th...@knime.com>
wrote:
> So I did some debugging and I believe I have found a bug in Artemis.
> I create two topics, "t.*" and "t.123" (all using JMS). I register two
> consumers on both "t.*" and "t.123" and send a message to "t.123". If the
> consumers close both topics are correctly deleted from the broker.
> However, if
> I do *not* register a consumer to the more specific topic "t.123" its
> address
> will stay in the broker even after the consumer for "t.*" closes. The
> address
> "t.*" is correctly deleted, though. Reproducer it attached.
> The problem seems to be that in the second case no
> "bindingRemovedTimestamp"
> is set in the corresponding AddressInfo and hence it stays -1 and the
> adsress
> is not eligible for auto-deletion.
> If you confirm that this is indeed a bug I can open a ticket.
>
> Thorsten
>
>
>
> Am Freitag, 5. März 2021, 17:56:37 CET schrieb Justin Bertram:
> > I don't know if there's a way to see why it wasn't deleted from the web
> > console. You could always attach a debugger to the broker and inspect it
> > that way. I believe the two main classes involved here would be
> >
> org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.AddressQueue
> > Reaper and org.apache.activemq.artemis.core.server.impl.QueueManagerImpl.
> >
> > If you can work up a simple way to reproduce this I can take a closer
> look.
> > To this end I often recommend users create a variation of one of the
> > examples we ship with the broker. We ship a "topic-hierarchies" examples
> > which would probably be a good starting point.
> >
> >
> > Justin
> >
> > On Fri, Mar 5, 2021 at 10:44 AM Thorsten Meinl <thorsten.meinl@knime.com
> >
> >
> > wrote:
> > > Hi,
> > >
> > > Am Freitag, 5. März 2021, 17:35:32 CET schrieb Justin Bertram:
> > > > First off, what version of ActiveMQ Artemis are you using?
> > >
> > > Sorry, forgot to mention: 2.16.0
> > >
> > > > Do you have auto-delete-addresses = true? If so, addresses *should*
> be
> > > > deleted automatically when they have no more bindings. The wildcard
> > >
> > > itself
> > >
> > > > is a binding so that's probably why the address isn't removed.
> > >
> > > Yes, auto-delete-addresses = true. During operation I can see the
> wilcard
> > > adress "jobs.\*" which *does* get auto-deleted after all consumers are
> > > gone.
> > > Also some other addresses (without wilcard consumers attached) get
> deleted
> > > but
> > > not the jobs.A, jobs.B, ... addresses.
> > > Is there a way to see why they would stay there? In the "Attributes"
> tab
> > > in
> > > the Artemis Console I can see "Address size", "All queue names" and
> > > "Binding
> > > names" and all three are 0 or empty.
> > >
> > > > I think using a single topic with selectors on the subscriptions is a
> > > > viable alternative. There shouldn't be any performance penalty and
> the
> > > > semantics should be exactly what you're looking for.
> > >
> > > OK, good to hear. The we may likely go that way.
> > >
> > > Thanks!
> > >
> > > Thorsten
> > >
> > > > On Fri, Mar 5, 2021 at 10:09 AM Thorsten Meinl <
> thorsten.meinl@knime.com
> > > >
> > > > wrote:
> > > > > Hi,
> > > > >
> > > > > We are currently building an application using ActiveMQ Artemis.
> The
> > >
> > > rough
> > >
> > > > > setup is as follows: We have consumers that are interested in all
> > >
> > > messages
> > >
> > > > > and
> > > > > other consumers that are only interested in a subset. So far we
> have
> > >
> > > been
> > >
> > > > > using topic hierarchies. The first group of consumers listens on
> e.g.
> > > > > "jobs.*"
> > > > > whereas the others listens on "jobs.A", "jobs.B", etc. All
> addresses,
> > > > > topics,
> > > > > queues are auto-created. What we notice now is that even if all
> > >
> > > consumers
> > >
> > > > > are
> > > > > gone there are still addresses "jobs.A", "jobs.B", ... remaining
> even
> > > > > though
> > > > > they don't have anything attached to them any more. No consumers,
> no
> > > > > queues,
> > > > > nothing. Therefore the first question: why is this and how can we
> > >
> > > prevent
> > >
> > > > > it?
> > > > >
> > > > > An alternative approach would be to use just one topic "jobs" and
> > >
> > > filter
> > >
> > > > > messages with message selectors (e.g. "id=A", "id=B", ...). This
> would
> > > > > eliminate the issue above. The question is whether there is a
> > >
> > > performance
> > >
> > > > > or
> > > > > functionality penalty involved with this approach.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Thorsten
> > > > >
> > > > > --
> > > > > Dr.-Ing. Thorsten Meinl
> > > > > KNIME AG
> > > > > Hardturmstrasse 66
> > > > > 8005 Zurich, Switzerland
> > >
> > > --
> > > Dr.-Ing. Thorsten Meinl
> > > KNIME AG
> > > Hardturmstrasse 66
> > > 8005 Zurich, Switzerland
>
>
> --
> Dr.-Ing. Thorsten Meinl
> KNIME AG
> Hardturmstrasse 66
> 8005 Zurich, Switzerland
Re: Message filters vs. topic hierachies
Posted by Thorsten Meinl <th...@knime.com>.
So I did some debugging and I believe I have found a bug in Artemis.
I create two topics, "t.*" and "t.123" (all using JMS). I register two
consumers on both "t.*" and "t.123" and send a message to "t.123". If the
consumers close both topics are correctly deleted from the broker. However, if
I do *not* register a consumer to the more specific topic "t.123" its address
will stay in the broker even after the consumer for "t.*" closes. The address
"t.*" is correctly deleted, though. Reproducer it attached.
The problem seems to be that in the second case no "bindingRemovedTimestamp"
is set in the corresponding AddressInfo and hence it stays -1 and the adsress
is not eligible for auto-deletion.
If you confirm that this is indeed a bug I can open a ticket.
Thorsten
Am Freitag, 5. M�rz 2021, 17:56:37 CET schrieb Justin Bertram:
> I don't know if there's a way to see why it wasn't deleted from the web
> console. You could always attach a debugger to the broker and inspect it
> that way. I believe the two main classes involved here would be
> org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.AddressQueue
> Reaper and org.apache.activemq.artemis.core.server.impl.QueueManagerImpl.
>
> If you can work up a simple way to reproduce this I can take a closer look.
> To this end I often recommend users create a variation of one of the
> examples we ship with the broker. We ship a "topic-hierarchies" examples
> which would probably be a good starting point.
>
>
> Justin
>
> On Fri, Mar 5, 2021 at 10:44 AM Thorsten Meinl <th...@knime.com>
>
> wrote:
> > Hi,
> >
> > Am Freitag, 5. M�rz 2021, 17:35:32 CET schrieb Justin Bertram:
> > > First off, what version of ActiveMQ Artemis are you using?
> >
> > Sorry, forgot to mention: 2.16.0
> >
> > > Do you have auto-delete-addresses = true? If so, addresses *should* be
> > > deleted automatically when they have no more bindings. The wildcard
> >
> > itself
> >
> > > is a binding so that's probably why the address isn't removed.
> >
> > Yes, auto-delete-addresses = true. During operation I can see the wilcard
> > adress "jobs.\*" which *does* get auto-deleted after all consumers are
> > gone.
> > Also some other addresses (without wilcard consumers attached) get deleted
> > but
> > not the jobs.A, jobs.B, ... addresses.
> > Is there a way to see why they would stay there? In the "Attributes" tab
> > in
> > the Artemis Console I can see "Address size", "All queue names" and
> > "Binding
> > names" and all three are 0 or empty.
> >
> > > I think using a single topic with selectors on the subscriptions is a
> > > viable alternative. There shouldn't be any performance penalty and the
> > > semantics should be exactly what you're looking for.
> >
> > OK, good to hear. The we may likely go that way.
> >
> > Thanks!
> >
> > Thorsten
> >
> > > On Fri, Mar 5, 2021 at 10:09 AM Thorsten Meinl <thorsten.meinl@knime.com
> > >
> > > wrote:
> > > > Hi,
> > > >
> > > > We are currently building an application using ActiveMQ Artemis. The
> >
> > rough
> >
> > > > setup is as follows: We have consumers that are interested in all
> >
> > messages
> >
> > > > and
> > > > other consumers that are only interested in a subset. So far we have
> >
> > been
> >
> > > > using topic hierarchies. The first group of consumers listens on e.g.
> > > > "jobs.*"
> > > > whereas the others listens on "jobs.A", "jobs.B", etc. All addresses,
> > > > topics,
> > > > queues are auto-created. What we notice now is that even if all
> >
> > consumers
> >
> > > > are
> > > > gone there are still addresses "jobs.A", "jobs.B", ... remaining even
> > > > though
> > > > they don't have anything attached to them any more. No consumers, no
> > > > queues,
> > > > nothing. Therefore the first question: why is this and how can we
> >
> > prevent
> >
> > > > it?
> > > >
> > > > An alternative approach would be to use just one topic "jobs" and
> >
> > filter
> >
> > > > messages with message selectors (e.g. "id=A", "id=B", ...). This would
> > > > eliminate the issue above. The question is whether there is a
> >
> > performance
> >
> > > > or
> > > > functionality penalty involved with this approach.
> > > >
> > > > Thanks,
> > > >
> > > > Thorsten
> > > >
> > > > --
> > > > Dr.-Ing. Thorsten Meinl
> > > > KNIME AG
> > > > Hardturmstrasse 66
> > > > 8005 Zurich, Switzerland
> >
> > --
> > Dr.-Ing. Thorsten Meinl
> > KNIME AG
> > Hardturmstrasse 66
> > 8005 Zurich, Switzerland
--
Dr.-Ing. Thorsten Meinl
KNIME AG
Hardturmstrasse 66
8005 Zurich, Switzerland
Re: Message filters vs. topic hierachies
Posted by Justin Bertram <jb...@apache.org>.
I don't know if there's a way to see why it wasn't deleted from the web
console. You could always attach a debugger to the broker and inspect it
that way. I believe the two main classes involved here would be
org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.AddressQueueReaper
and org.apache.activemq.artemis.core.server.impl.QueueManagerImpl.
If you can work up a simple way to reproduce this I can take a closer look.
To this end I often recommend users create a variation of one of the
examples we ship with the broker. We ship a "topic-hierarchies" examples
which would probably be a good starting point.
Justin
On Fri, Mar 5, 2021 at 10:44 AM Thorsten Meinl <th...@knime.com>
wrote:
> Hi,
>
> Am Freitag, 5. März 2021, 17:35:32 CET schrieb Justin Bertram:
> > First off, what version of ActiveMQ Artemis are you using?
> Sorry, forgot to mention: 2.16.0
>
> > Do you have auto-delete-addresses = true? If so, addresses *should* be
> > deleted automatically when they have no more bindings. The wildcard
> itself
> > is a binding so that's probably why the address isn't removed.
> Yes, auto-delete-addresses = true. During operation I can see the wilcard
> adress "jobs.\*" which *does* get auto-deleted after all consumers are
> gone.
> Also some other addresses (without wilcard consumers attached) get deleted
> but
> not the jobs.A, jobs.B, ... addresses.
> Is there a way to see why they would stay there? In the "Attributes" tab
> in
> the Artemis Console I can see "Address size", "All queue names" and
> "Binding
> names" and all three are 0 or empty.
>
> > I think using a single topic with selectors on the subscriptions is a
> > viable alternative. There shouldn't be any performance penalty and the
> > semantics should be exactly what you're looking for.
> OK, good to hear. The we may likely go that way.
>
> Thanks!
>
> Thorsten
>
>
>
> > On Fri, Mar 5, 2021 at 10:09 AM Thorsten Meinl <thorsten.meinl@knime.com
> >
> >
> > wrote:
> > > Hi,
> > >
> > > We are currently building an application using ActiveMQ Artemis. The
> rough
> > > setup is as follows: We have consumers that are interested in all
> messages
> > > and
> > > other consumers that are only interested in a subset. So far we have
> been
> > > using topic hierarchies. The first group of consumers listens on e.g.
> > > "jobs.*"
> > > whereas the others listens on "jobs.A", "jobs.B", etc. All addresses,
> > > topics,
> > > queues are auto-created. What we notice now is that even if all
> consumers
> > > are
> > > gone there are still addresses "jobs.A", "jobs.B", ... remaining even
> > > though
> > > they don't have anything attached to them any more. No consumers, no
> > > queues,
> > > nothing. Therefore the first question: why is this and how can we
> prevent
> > > it?
> > >
> > > An alternative approach would be to use just one topic "jobs" and
> filter
> > > messages with message selectors (e.g. "id=A", "id=B", ...). This would
> > > eliminate the issue above. The question is whether there is a
> performance
> > > or
> > > functionality penalty involved with this approach.
> > >
> > > Thanks,
> > >
> > > Thorsten
> > >
> > > --
> > > Dr.-Ing. Thorsten Meinl
> > > KNIME AG
> > > Hardturmstrasse 66
> > > 8005 Zurich, Switzerland
>
>
> --
> Dr.-Ing. Thorsten Meinl
> KNIME AG
> Hardturmstrasse 66
> 8005 Zurich, Switzerland
Re: Message filters vs. topic hierachies
Posted by Thorsten Meinl <th...@knime.com>.
Hi,
Am Freitag, 5. März 2021, 17:35:32 CET schrieb Justin Bertram:
> First off, what version of ActiveMQ Artemis are you using?
Sorry, forgot to mention: 2.16.0
> Do you have auto-delete-addresses = true? If so, addresses *should* be
> deleted automatically when they have no more bindings. The wildcard itself
> is a binding so that's probably why the address isn't removed.
Yes, auto-delete-addresses = true. During operation I can see the wilcard
adress "jobs.\*" which *does* get auto-deleted after all consumers are gone.
Also some other addresses (without wilcard consumers attached) get deleted but
not the jobs.A, jobs.B, ... addresses.
Is there a way to see why they would stay there? In the "Attributes" tab in
the Artemis Console I can see "Address size", "All queue names" and "Binding
names" and all three are 0 or empty.
> I think using a single topic with selectors on the subscriptions is a
> viable alternative. There shouldn't be any performance penalty and the
> semantics should be exactly what you're looking for.
OK, good to hear. The we may likely go that way.
Thanks!
Thorsten
> On Fri, Mar 5, 2021 at 10:09 AM Thorsten Meinl <th...@knime.com>
>
> wrote:
> > Hi,
> >
> > We are currently building an application using ActiveMQ Artemis. The rough
> > setup is as follows: We have consumers that are interested in all messages
> > and
> > other consumers that are only interested in a subset. So far we have been
> > using topic hierarchies. The first group of consumers listens on e.g.
> > "jobs.*"
> > whereas the others listens on "jobs.A", "jobs.B", etc. All addresses,
> > topics,
> > queues are auto-created. What we notice now is that even if all consumers
> > are
> > gone there are still addresses "jobs.A", "jobs.B", ... remaining even
> > though
> > they don't have anything attached to them any more. No consumers, no
> > queues,
> > nothing. Therefore the first question: why is this and how can we prevent
> > it?
> >
> > An alternative approach would be to use just one topic "jobs" and filter
> > messages with message selectors (e.g. "id=A", "id=B", ...). This would
> > eliminate the issue above. The question is whether there is a performance
> > or
> > functionality penalty involved with this approach.
> >
> > Thanks,
> >
> > Thorsten
> >
> > --
> > Dr.-Ing. Thorsten Meinl
> > KNIME AG
> > Hardturmstrasse 66
> > 8005 Zurich, Switzerland
--
Dr.-Ing. Thorsten Meinl
KNIME AG
Hardturmstrasse 66
8005 Zurich, Switzerland
Re: Message filters vs. topic hierachies
Posted by Justin Bertram <jb...@apache.org>.
First off, what version of ActiveMQ Artemis are you using?
Do you have auto-delete-addresses = true? If so, addresses *should* be
deleted automatically when they have no more bindings. The wildcard itself
is a binding so that's probably why the address isn't removed.
I think using a single topic with selectors on the subscriptions is a
viable alternative. There shouldn't be any performance penalty and the
semantics should be exactly what you're looking for.
Justin
On Fri, Mar 5, 2021 at 10:09 AM Thorsten Meinl <th...@knime.com>
wrote:
> Hi,
>
> We are currently building an application using ActiveMQ Artemis. The rough
> setup is as follows: We have consumers that are interested in all messages
> and
> other consumers that are only interested in a subset. So far we have been
> using topic hierarchies. The first group of consumers listens on e.g.
> "jobs.*"
> whereas the others listens on "jobs.A", "jobs.B", etc. All addresses,
> topics,
> queues are auto-created. What we notice now is that even if all consumers
> are
> gone there are still addresses "jobs.A", "jobs.B", ... remaining even
> though
> they don't have anything attached to them any more. No consumers, no
> queues,
> nothing. Therefore the first question: why is this and how can we prevent
> it?
>
> An alternative approach would be to use just one topic "jobs" and filter
> messages with message selectors (e.g. "id=A", "id=B", ...). This would
> eliminate the issue above. The question is whether there is a performance
> or
> functionality penalty involved with this approach.
>
> Thanks,
>
> Thorsten
>
> --
> Dr.-Ing. Thorsten Meinl
> KNIME AG
> Hardturmstrasse 66
> 8005 Zurich, Switzerland